Design Engineering
Showcase 2020

Automation technology in the food industry reduces physical and cognitive demands on production line operators. Perhaps the same tech could also reduce errors and assist decision-making in home cooking? How might automation augment the cooking skills of busy parents and professionals, for example?

Kitchens pose very different design engineering challenges to industry, however, as home cooking requires multi-purpose tools rather than specialised machines. Robot arms can mimic human-kitchen interaction, but these are too large and expensive to be feasible for home use. For multi-purpose sensing, cameras can detect a wide variety of cooking information, but there are currently no datasets for training cooking image classification algorithms.

With OnionBot, I wanted to see if there was a way to integrate industry automation techniques and machine vision into a simple robot that fits on a countertop.

OnionBot tackles automation of pan cooking. The camera and thermal camera mounted above the stove track cooking progress, and data is processed by a Raspberry Pi. The servo motor automatically controls induction stove power setting.

The goal of this project is automation without unnecessary complexity. After all, we are not replacing the chef but simply giving them multi-tasking superpowers! There is no actuator more flexible and dextrous than the human: a touch screen interface called ‘sous-chef’ provides instructions, reminders, and alerts. OnionBot takes care of the pan so that the chef can focus on culinary creativity. This human-in-the-loop approach is a first of its kind in cooking robotics research.

Food image classification research has reported poor real-world results; food images often have tricky-to-define features and a lot of environmental variation. A cooking device must tackle these perception problems.

OnionBot introduces two improvements.

1. The fixed stove-top camera view ensures a consistent environment.
2. Researchers typically explore general classification, attempting to identify characteristics from 1000s of potential classes. OnionBot instead establishes a milestone-based approach, labelling only key events at which actions occur (milestones) for a single recipe. Each classification model must identify just 10s of classes, dramatically simplifying the perception challenge.

As these datasets don’t currently exist, OnionBot includes an interface for easy creation of labelled datasets of cooking images. With the control panel you simply click along with each milestone as you cook, and labelled images are automatically uploaded to the cloud where they can be accessed by model training platforms. Training is streamlined using Google AutoML, allowing models to be trained for new recipes with just a few clicks.

This diagram shows the complete system. I wrote the OnionBot firmware in Python, and frontends for the control devices in HTML/CSS/JavaScript.

Check out the video above to see a demonstration of the prototype! OnionBot can automatically proceed through a complete pasta and tomato sauce recipe. It provides hands-free instructions and autonomously controls pan temperature. I also added boiling-over detection and stirring reminders using the camera. This simple recipe only scratches the surface of what could be possible in the future with machine vision.

This prototype demonstrates an initial proof-of-concept showing the possibilities of automation in home cooking, but large training datasets will be critical to success of machine vision systems. I propose that autonomous cooking should be approached in the same way that Tesla Motors approaches autonomous driving. Rolling out a ‘fleet’ of networked OnionBot devices would enable crowd-sourcing of massive labeled pan-cooking image databases.

Large cooking image databases don’t currently exist. The OnionBot dataset (which would also include rich metadata on ingredients, temperature, recipes, corrective inputs, and so on) could enable new research into cooking with AI.

OnionBot is open-sourced to encourage further research into home cooking automation. This project is perfect for makers and hobbyists, as it comprises off-the-shelf components and accessible code. Long term, a community of collaborators and early adopters could crowd-source:

• The massive cooking image dataset.
• A database of recipe vision models.
• Advanced deep learning functionality, beyond classification.
• New ‘product’ hardware design.

Check out the project GitHub to join the community.

In the short term, I hope to continue to develop this project. Planned hardware iterations will bring down the component costs and new machine vision models will further expand functionality. I would love your feedback, support and ideas! Please reach out to me at any of the contact details on this page.