This is the continuation for Part I and we are going to train the model and see what that means.
Train the model
Before you train the model you will do an extra and simple step which is to extract features from your dataset, just click on the generate feature and you will see a nice representation of your dataset. Besides it looks nice this representation is an early indication if your data is good or not. You will see a 3D representation of each of your images displaying where it falls, if you see some clear cluster pattern in that representation then you can expect a good result. As these are images you cannot manipulate it is possible to see a complex clustering pattern, but clear anyway. In the case you are the one generating the images (as I did for the Anomaly detection, see the previous posts) then you can see an even clearer separation. To extract features just go to the image block in your impulse and click "extract features", the process will be automatic.
Once the features are extracted you can train the model, just go to the learning block, adjust some parameters if you feel confident about what you are doing and click on start training. Depending on the amount of provided data this process could take some time, the training is done in the cloud and when finished you will see some statistics about the result. As you can see in the picture this model has a 100% of accuracy, there is something to say here. That 100% refers to the training dataset and it doesn’t mean that your model will have that accuracy when providing unexpected data as input.
When the training process is done, you will see a confusion matrix, some metrics such as accuracy and the visual representation of how data was classified during the learning process, remember that the data being used belongs to the training dataset only.
As you can see in the picture everything went wonderful so far!
Validating or Testing the model
Once the model is trained you will test it. To test the trained model means exposing it to the data you specified in the Validation or Testing dataset. Go to the test model menu entry and can select all the pictures on the test dataset and start the process, once the test is finished you will see how good it went. You will see each picture followed by the prediction. In this example, I left some pictures from another dataset (pictures with more complicated backgrounds) to make the example more realistic; most of the images were classified OK (OK, means the model classified them as expected) and some of them didn’t. As you can see in the figure, the accuracy percentage is a little bit lower (95.50%) than the one displayed after training and you will also see the samples that went wrong. Here is where the fine tuning begins.
Review, corrections, retraining
Sometimes you need to make changes to the models itself, sometimes it is a matter of rearranging your dataset. In this case, you can examine what went wrong and fix it. There are some pictures that are out of place such as a Lion and a part of furniture, that is a bad picture because you don't expect to see that in the real scenario, so, you can remove those. The other picture is a good one, the exception is the angle and some heavy shadows over the Jaguar. If that scenario can happen, then the best thing you can do is to move that picture to the training set in order to allow the model to learn from those variants too. By clicking on the menu of each picture you will see the options to remove them or to send them to the training set. These are examples of bad pictures to be removed and good pictures to be included in the training set.
Once you are finished cleaning your dataset, you will retrain the model again and test it again. In this case, I did so, I tested the model again and the accuracy went up to 99%. This is a process that you will have to repeat until you are satisfied with the results. It is also a good idea to inspect the datasets even if things went OK the first time.
After repeating this process you will have an idea about when you model is good and when it needs improvements. In the scenarios where you model is less accurate the best thing you can do is to get more data and test the model with those examples and then use them on the training set. Once you have a trained model you can perform a live classification (just for one picture), so you don't need to test the model again, it is just a matter of uploading a new sample and test it against the model, this will allow you to fine tune your model even more. Don't fool yourself, challenge your model with complex pictures or pictures you think you can expect in the real scenario.
Edge Impulse has several nice tools to allow you keep things under control in this iterative process, each time you feel you made some advances then it is good time to store the version, in order to do so you just need to go to the version control add a description for your current version and that is it. Version control and notes are good tools to understand later what you did and the results you got.
Deploy the model, extract the model and copy it to the Arduino Portenta !!!
And now, the moment of truth, take the model to run on the Arduino!!!
Edge impulse has several ways to deploy models, whoever the best way to deploy the model to the Arduino Portenta today is the openMV option. As I said before, edgeImpulse is still on preview mode regarding the Portenta, you could deploy the C++ Library or the Arduino Library however a vital component (the feature extractor) is not implemented yet thus it won’t be that useful. The only useful thing in this regard is that you will see an estimate of the resources being consumed by the model.
The openMV deployment is just a .zip file containing the optimized tensor Flow model. When you connect the Portenta to the computer it will be exposed as a regular USB drive, you just need to download the model, unzip it and copy the file into the Portenta and that's it. The downloaded package comes with a Python script that will enable the camera and perform the inferences in an endless loop. As you can see in the video, the performance and accuracy are impressive indeed, 3.4FPS and most of the time the score is near 1! Even when I point the camera to Jaguars pictures or posters I didn’t include in the training set. Do you want to see it running ? there you go!
A short conclusion! Edge Impulse is a pretty handy tool to work with tinyML, the same for the openMV but most of all, the Arduino Portenta is FAST! at running the models; in some future posts I will cover more models to work with sound which are also important for Big Cat Brother and of course I will cover some of the useful features openMV implements that are also used by BCB in some scenarios. Stay Tuned!