Project 5 - Machine Learning - CS 188: Introduction to Artificial Intelligence, Fall 2020 Project 5: Machine Learning Version XXXXXXXXXXLast Updated: 11/15/2020. Due: Friday 12/04 at 11:59 pm In this...

Project 5 - Machine Learning - CS 188: Introduction to Artificial Intelligence, Fall 2020
Project 5: Machine Learning
Version XXXXXXXXXXLast Updated: 11/15/2020.
Due: Friday 12/04 at 11:59 pm
In this project you will build a neural network to classify digits, and more!
Introduction
This project will be an introduction to machine learning.
The code for this project contains the following files, available as a zip archive.
Files you'll edit:
models.py Perceptron and neural network models for a variety of
applications
Files you should read but NOT edit:
nn.py Neural network mini-li
ary
Files you will not edit:
autograder.py Project autograde
ackend.py Backend code for various machine learning tasks
data Datasets for digit classification and language identification
submission_autograder.py Submission autograder (generates tokens for submission)
Files to Edit and Submit: You will fill in portions of models.py during the assignment. Please do not
change the other files in this distribution.
Note: You only need to submit machinelearning.token , generated by running
submission_autograder.py . It contains the evaluation results from your local autograder, and a
copy of all your code. You do not need to submit any other files. See Submission for details.
Evaluation: Your code will be autograded for technical co
ectness. Please do not change the names of
any provided functions or classes within the code, or you will wreak havoc on the autograder. However,
the co
ectness of your implementation – not the autograder’s judgements – will be the final judge of
your score. If necessary, we will review and grade assignments individually to ensure that you receive due
credit for your work.
Academic Dishonesty: We will be checking your code against other submissions in the class for logical
Introduction
Installation
Question 1 (6 points):
Perceptron
Question 2 (6 points): Non-
linear Regression
Question 3 (6 points): Digit
Classification
Question 4 (7 points):
Language Identification
Submission
https:
inst.eecs.berkeley.edu/~cs188/fa20/assets/files/machinelearning.zip
https:
inst.eecs.berkeley.edu/~cs188/fa20/assets/files/machinelearning.zip
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#Submission
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#Submission
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#introduction
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#introduction
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#installation
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#installation
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-1-6-points-perceptron
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-1-6-points-perceptron
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-1-6-points-perceptron
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-2-6-points-non-linear-regression
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-2-6-points-non-linear-regression
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-2-6-points-non-linear-regression
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-3-6-points-digit-classification
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-3-6-points-digit-classification
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-3-6-points-digit-classification
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-4-7-points-language-identification
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-4-7-points-language-identification
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#question-4-7-points-language-identification
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#submission
https:
inst.eecs.berkeley.edu/~cs188/fa20/project5/#submission
edundancy. If you copy someone else’s code and submit it with minor changes, we will know. These cheat
detectors are quite hard to fool, so please don’t try. We trust you all to submit your own work only; please
don’t let us down. If you do, we will pursue the strongest consequences available to us.
Proper Dataset Use: Part of your score for this project will depend on how well the models you train
perform on the test set included with the autograder. We do not provide any APIs for you to access the
test set directly. Any attempts to bypass this separation or to use the testing data during training will be
considered cheating.
Getting Help: You are not alone! If you find yourself stuck on something, contact the course staff for help.
Office hours, section, and the discussion forum are there for your support; please use them. If you can’t
make our office hours, let us know and we will schedule more. We want these projects to be rewarding
and instructional, not frustrating and demoralizing. But, we don’t know when or how to help unless you
ask.
Discussion: Please be careful not to post spoilers.
Installation
For this project, you will need to install the following two li
aries:
• numpy, which provides support for large multi-dimensional a
ays - installation instructions
• matplotlib, a 2D plotting li
ary - installation instructions
If you have a conda environment, you can install both packages on the command line by running:
You will not be using these li
aries directly, but they are required in order to run the provided code and
autograder.
To test that everything has been installed, run:
If numpy and matplotlib are installed co
ectly, you should see a window pop up where a line
segment spins in a circle:
conda activate [your environment name]
pip install numpy
pip install matplotli
python autograder.py --check-dependencies
http:
www.numpy.org
http:
www.numpy.org
https:
docs.scipy.org/doc/numpy-1.13.0/use
install.html
https:
docs.scipy.org/doc/numpy-1.13.0/use
install.html
https:
matplotlib.org
https:
matplotlib.org
https:
matplotlib.org/users/installing.html
https:
matplotlib.org/users/installing.html
Provided Code (Part I)
For this project, you have been provided with a neural network mini-li
ary ( nn.py ) and a collection of
datasets ( backend.py ).
The li
ary in nn.py defines a collection of node objects. Each node represents a real number or a
matrix of real numbers. Operations on node objects are optimized to work faster than using Python’s
uilt-in types (such as lists).
Here are a few of the provided node types:
• nn.Constant represents a matrix (2D a
ay) of floating point numbers. It is typically used to
epresent input features or target outputs/labels. Instances of this type will be provided to you by
other functions in the API; you will not need to construct them directly.
• nn.Parameter represents a trainable parameter of a perceptron or neural network.
• nn.DotProduct computes a dot product between its inputs.
Additional provided functions:
• nn.as_scalar can extract a Python floating-point number from a node.
When training a perceptron or neural network, you will be passed a dataset object. You can retrieve
atches of training examples by calling dataset.iterate_once(batch_size) :
For example, let’s extract a batch of size 1 (i.e., a single training example) from the perceptron training
data:
The input features x and the co
ect label y are provided in the form of nn.Constant nodes. The
shape of x will be batch_size x num_features , and the shape of y is
atch_size x num_outputs . Here is an example of computing a dot product of x with itself, first as
a node and then as a Python number.
Question 1 (6 points): Perceptron
Before starting this part, be sure you have numpy and matplotlib installed!
In this part, you will implement a binary perceptron. Your task will be to complete the implementation of
the PerceptronModel class in models.py .
For the perceptron, the output labels will be either or , meaning that data points (x, y) from the
dataset will have y be a nn.Constant node that contains either or as its entries.
We have already initialized the perceptron weights self.w to be a parameter node.
The provided code will include a bias feature inside x when needed, so you will not need a separate
parameter for the bias.
Your tasks are to:
• Implement the run(self, x) method. This should compute the dot product of the stored
weight vector and the given input, returning an nn.DotProduct object.
• Implement get_prediction(self, x) , which should return if the dot product is non-
negative or otherwise. You should use nn.as_scalar to convert a scalar Node into a Python
for x, y in dataset.iterate_once(batch_size):
...
batch_size = 1
for x, y in dataset.iterate_once(batch_size):
... print(x)
... print(y)
...
eak
...
Constant shape=1x3 at 0x11a8856a0
Constant shape=1x1 at 0x11a89efd0
nn.DotProduct(x, x)
DotProduct shape=1x1 at 0x11a89edd8
nn.as_scalar(nn.DotProduct(x, x))
XXXXXXXXXX
1 −1
1 −1
1 × dimensions
1
−1
floating-point number.
• Write the train(self) method. This should repeatedly loop over the data set and make updates
on examples that are misclassified. Use the update method of the nn.Parameter class to
update the weights. When an entire pass over the data set is completed without making any
mistakes, 100% training accuracy has been achieved, and training can terminate.
In this project, the only way to change the value of a parameter is by calling
parameter.update(direction, multiplier) , which will perform the update to the weights:
The direction argument is a Node with the same shape as the parameter, and the multiplie
argument is a Python scalar.
To test your implementation, run the autograder:
Note: the autograder should take at most 20 seconds or so to run for a co
ect implementation. If the
autograder is taking forever to run, your code probably has a bug.
Neural Network Tips
In the remaining parts of the project, you will implement the following models:
• Q2: Regression
• Q3: Handwritten Digit Classification
• Q4: Language Identification
Building Neural Nets
Throughout the applications portion of the project, you’ll use the framework provided in nn.py to
create neural networks to solve a variety of machine learning problems. A simple neural network has
layers, where each layer performs a linear operation (just like perceptron). Layers are separated by a non-
linearity, which allows the network to approximate general functions. We’ll use the ReLU operation fo
our non-linearity, defined as . For example, a simple two-layer neural network fo
mapping an input row vector to an output vector would be given by the function:
where we have parameter matrices and and parameter vectors and to learn during
gradient descent. will be an matrix, where is the dimension of our input vectors , and is the
hidden layer size. will be a size vector. We are free to choose any value we want for the hidden size
(we will just need to make sure the dimensions of the other matrices and vectors agree so that we can
perform the operations). Using a larger hidden size will usually make the network more powerful (able to
fit more training data), but can make the network harder to train (since it adds more parameters to all the
matrices and vectors we need to learn), or can lead to overfitting on the training data. 
We can also create deeper networks by adding more layers, for example a three-layer net:
Note on Batching
For efficiency, you will be required to process whole batches of data at once rather than a single example
at a time. This means that instead of a single input row vector with size , you will be presented with a
atch of inputs represented as a matrix . We provide an example for linear regression to
demonstrate how a linear layer can be implemented in the batched setting.
Note on Randomness
The parameters of your neural network will be randomly initialized, and data in some tasks will be
presented in shuffled order. Due to this randomness, it’s possible that you will still occasionally fail some
tasks even with a strong architecture – this is the problem of local optima! This should happen very
arely, though – if when testing your code you fail