PROJECT#2 – Raisin
VERSION 3b updated 12-8, 2022
Collaborative Assignment (groups of 2-3)
Use the AIMA-PYTHON li
ary to explore the UCI raisin dataset
You will be building classifier models, analyzing results, comparing models and offering improvements.
GOAL: Explore AIMA Python modules for classification, UCI dataset, and metrics (AIMA & custom – NO sklearn)
REQUIREMENTS:
• Create classification models using AIMA learners (simple perceptron, ensemble, neural network);
• Make use of AIMA performance metrics (E
or ratio, accuracy_score, r2_score, grade_learner) & SciKit-Learn metrics
(confusion matrix & classification report). [minimum of 2]
• Useful modules will be found in learning4e, neural_networks, deep_learning4e
• Sample code can be found in learning.ipynb, test_learning, Learning_RR_modified_Perceptron, Learners… ;
• Code must be THOROUGHLY DOCUMENTED, with a complete General Header, and inline comments that will make
SEARCHING for tasks completed easy to locate. If the grader is unable to quickly locate supporting code for each task
elow, the task will get 0 points.;
• There will be a written report required, including a results table.
TASKS
• DATA
o Load UCI raisin dataset
o Assign the attribute names and source from the .txt file
o Display details about the data, using DataSet attributes & assigning known details from UCI repository
o Prepare data1 for training & validating models, reserving a subset of rows as a hold-out set for testing (see Data
Engineering tips in footnotes)
o Unknown data point2 has original feature values: [188888, 666, 333, 0.77, 191919, 0.74, 1666].
• PERCEPTRON Classifiers
o Generate Simple PERCEPTRON classifiers from training data & record hyperparameter values & results (to table)
for each new model below:
▪ Generate 1st model using all defaults
▪ Next model: change the learning rate, until you see improvement
▪ Next model: change the number of epochs, until you see improvement
o Create a final Simple PERCEPTRON model w
est hyperparameters from above, or additional choices if
appropriate/available, (fit to training data)
o Record results when tested on training data, and separately on test data. Support your results and assessment
with at least two different performance metrics,
o Predict for Unknown data point. Record the prediction and any speculation on this outcome.2
• ENSEMBLE modeling
o Use AdaBoost –or—EnsembleLearner to generate ensemble models, starting with defaults.
o Next Models: Alter hyperparameter or learner, as appropriate, and record results (to table)
o Record results when tested on training data, and separately on test data. Support your results and assessment
with at least two different performance metrics,
1 - As mentioned in TIPS, you should follow best practices, to include: shuffle, convert-classes-to-Numbers, normalize features,
and train-test-split. Be aware of when to pass full DataSet, examples only, features only, targets only, training-set only, test-
set only.
2 Unknown Data Point is to be used to predict output class for a single data point , as requested in last bullet of the 3 model types.
https:
archive.ics.uci.edu/ml/datasets/Raisin+Dataset
o Predict for Unknown data point. Record the prediction and any speculation on this outcome.2
• NEURAL NETWORK modeling
o Generate multi-layer perceptron classifiers & record hyperparameters & results (to table)
▪ Generate 1st model using all defaults
o Record results for each of: #hidden layers/nodes is (1,10), (2,10), (3,10) , and
o Record results for each of: learning rate at 1.0, 0.1, .01, and <# of your choice>
o Create a final NNet model by selecting the best hyperparameters from above; record results.
o Record results when tested on training data, and separately on test data. Support your results and assessment
with at least two different performance metrics,
o Predict for Unknown data point. Record the prediction and any speculation on this outcome.2
DELIVERABLES: XXXXXXXXXXG# will be your group number.
• WRITTEN REPORT (.pdf)
o Overview describing your general approach to the problem (1paragraph);
o Analysis of performance (w/direct references to table);
o Summary of what worked/didn't, and recommendations for how you might improve with future efforts.
o Table of results (min. of 1 row per model)
• G#p2.ipynb – CLEAN source file.
o Include a general header (names, etc.) & generous comments to document your code
o You are required to include ALL dependencies in your submission.
o Remove any unnecessary code (unused functions, troubleshooting, etc.)
o Explicitly note (w/comments) any classes or functions you wrote or modified
• G#p2.pdf – .ipynb notebook,
o Visible: all code cells & comments,
o Visible: outputs from successful run & demonstrations that support your report & results table
P2 Written Report
Project 2: Raisin Feve
Group Number: 02
By:Amritvir Randhawa
Written Report
The general approach to the problem:
This collaborative assignment required us to create classification models using AIMA learners,
precisely simple perceptron, ensemble, and neural networks. To accomplish this, we have (1)
loaded and prepared the dataset for training; (2) generated a simple perceptron classifier and
tuned it based on learning rate and a number of epochs; (3) generated an Adaboost-based
ensemble model and tuned it based on the 'K' number; and (3) generated a neural network model
and tuned it based on hidden layer sizes, learning rate, epochs, and activation function. Each cell
lock has been explained using comments, and meaningful variable names have been used. The
original paper (Cinar et al., 2020), documents available on the aimacode/aima-python repository,
and other websites have been consulted to derive the solution.
Analysis of performance:
(i) The good performance (85.8% on ALL datasets) of the simple perceptron model is an
indication of its power. The high precision XXXXXXXXXXshows the reproducibility of the results. Table 1
shows the experiments with the learning rate. The accuracy stagnates at about 0.1. Table 2 shows
the experiments with epochs. The accuracy does not appreciably increase after 700 epochs. With
these hyperparameters tuned, the accuracy of the perceptron model increased to 87.2% on the
unseen validation dataset.
(ii) The performance of the untuned adaboost-based ensemble classifier (85.7%) was similar to
that of the perceptron model. 10 different values of the hyperparameter K were tried. It was seen
that a plateau of accuracies formed around K = 35 and K = 40 (Table 3). By setting K to 35, the
accuracy of the ensemble model increased to 87.2% on the unseen validation dataset.
(iii) A multi-layer perceptron (here refe
ed to as a neural network) classifier gave 85.4%
accuracy when trained using 80% of the data and tested on ALL the data values. Multiple values
of hyperparameters such as hidden layer sizes (Table 4), learning rate (Table 5), epochs (Table 6),
and activation function (Table 7) were tried and the ones giving the highest accuracies were
applied to the final model. The final model accuracy (85.6%) was a mere 0.2% up from the
initial run.
Summary and Recommendations:
This study showed how to implement 3 different classifiers and how to tune thei
hyperparameters. An interesting finding is that although ensemble-based and neural
network-based classifiers added more complexity to the classification, a simple perceptron model
still gave comparable accuracies without much tuning. The neural network hyperparamete
tuning requires more work. It has the capability to give better results. The approach adopted to
derive the best hyperparameter values is not co
ect. A grid-based approach, that is, a
combination of all hyperparameters much be tried to see which combination of hyperparamete
values gives the best results. Finally, the results could also be compared by implementing some
standard machine learning classifiers such as random forest, decision trees, and support vecto
machines - so as to support
efute the findings of Cinar et al., 2020.
Table of results (1 row per model):
(1) For a simple perceptron model, the hyperparameter learning rate
Learning rates = [ XXXXXXXXXX XXXXXXXXXX]
Accuracies = [0.5, XXXXXXXXXX, 0.86, XXXXXXXXXX, XXXXXXXXXX,
0.83, XXXXXXXXXX, XXXXXXXXXX, 0.82, XXXXXXXXXX]
(2) For a simple perceptron model, hyperparameter epochs
Epochs = [ XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX]
Accuracies = [0.5, XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX,
XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX, 0.87, 0.87,
XXXXXXXXXX, 0.87, XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX,
XXXXXXXXXX]
(3) For the ensemble model, hyperparameter K
K = [ XXXXXXXXXX XXXXXXXXXX]
Accuracies = [ XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX,
XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX, 0.86, 0.86,
XXXXXXXXXX, XXXXXXXXXX]
(4) For the neural network model, hyperparameter hidden layers/nodes
Hidden layer choices = [[7], [14], [20], [7, 14], [7, 14, 7], [14, 28, 14]]
Accuracies = [ XXXXXXXXXX, XXXXXXXXXX, 0.86, XXXXXXXXXX, 0.5,
XXXXXXXXXX]
(5) For the neural network model, the hyperparameter learning rate
Learning rates = [ XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX
XXXXXXXXXX]
Accuracies = [0.5, XXXXXXXXXX, 0.86, 0.86, XXXXXXXXXX,
XXXXXXXXXX, XXXXXXXXXX, 0.83, XXXXXXXXXX, XXXXXXXXXX,
XXXXXXXXXX, 0.82, 0.82, 0.82, 0.82, XXXXXXXXXX, 0.82, 0.82,
XXXXXXXXXX, 0.82]
(6) For the neural network model, hyperparameter epochs
Epochs = [ XXXXXXXXXX XXXXXXXXXX]
Accuracies = [ XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX,
XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX, XXXXXXXXXX,
XXXXXXXXXX, XXXXXXXXXX]
(7) For the neural network model, the hyperparameter activation function
Activation functions = ['sigmoid', 'relu', 'tanh', 'elu', 'leaky_relu']
Accuracies = [ XXXXXXXXXX, 0.5, XXXXXXXXXX, 0.5, 0.5]