Background

Our project is a website that uses machine learning to classify the facial expressions in a photo into one of 4 emotions: happy, sad, neutral, or angry. We used Flask to create the web application portion and Tensorflow for the machine learning portion.

Timeline

Our project had three major milestones, detailed below:

Milestone 1

• Finalize designs and prototypes for website.
• Develop basic frontend of website.
• Learn the basics of what machine learning is, as long as the major steps that go into image classification.
• Find emotion identifying datasets to train model on.

• Feature to upload and display images on the website.
• Ability to pre-process images to identify face, crop image, and resize to correct dimensions.
• Train machine learning model that reliably classifies the right emotions based off an image.

• Integrate classification model onto the website.
• Final polishes, including final design touches and improving classification accuracy.

Challenges

Our team faced several challenges during the machine learning portion of our project. Firstly, it was a challenge to bring up the accuracy of our machine learning classification accuracy rate, since our dataset’s resolution was too small (48x48). Secondly, overfitting was also an issue since our dataset was too small. We were able to partially solve this by randomly rotating and flipping images before training them, but it still affected the final accuracy.

We also ran into several small issues that stalled our progress in development while working out the frontend of our application. Working with Flask for the first time was a challenge for us as we tried to figure out Flask-specific details to file structures such as where to place all our static files. Getting used to Git, integrating the web application with the machine learning portion, and working with OpenCV for the first time also posed a challenge for us. While trying to integrate the webcam OpenCV, we ran into the problem of circular dependency between our files that we couldn’t quite wrap our head around. We ended up leaving out the webcam portion, but this is something we would like to figure out with more time.

Takeaways

Full-Stack Developers (Acelyn Choi, Subin Kim, Shelly Wu): Although we ran into many challenges with using Flask, it was a valuable learning experience, especially since all of us were unfamiliar with the framework. We were familiar with HTML and CSS, but using them with Flask was slightly different, which required us to dive deeper into how Flask works and learn more about it. We also learned the shortcut to a hard reload (Ctrl+Shift+R) that solved our problem of browser caching.

ML Developers (Aditya Gupta, Aarushi Singh): We learned that machine learning isn’t too hard to set up but is very hard to master - there were a lot of parameters and each one had an effect on the final accuracy. The structure of the model, the number of classes used, the quality of the dataset, and the time taken to train were all factors that we had to consider in training the model.

Looking Forward

Looking forward, there are several improvements we could make to our project:

Firstly, our final accuracy could have been improved by training on a larger dataset next time. Additionally, we could look into working with with distributed big data frameworks such as Spark. Finally, we could integrate Amazon s3 and cloud storage to handle larger datasets to work with.