Embedded Systems Projects
Table of Contents
Raspberry Putt Mini-Golf Game
The inspiration for this project was my background playing golf for fun as a kid and competitively as a high school student. We have created a mini-golf game that utilizes Pose Estimation to detect a user’s golf stance, an IMU remote controller alongside with dedicated-hardware to sense the user’s swing, and a dynamic user interface created in Unity to allow the user to play golf virtually. GitHub Link.
VERSION 2
VERSION 1
CONTROLLER
Raspberry Pi Swing Classifier
I designed a swing classifier using the 9-axis IMU’s accelerometer data and a Raspberry Pi Zero WH for its WiFi capabilities with MQTT. I sensed a wide range of different motions that the user could make: No motion, Motion, Back Swing, Down Swing, Complete Swing, and Swing Power. To sense the swing gesture, I used a hard-coded classifier rather than a machine-learning classifier. Before classification begins, there is a 5 second calibration period in which the user is prompted to get into their starting position. At the end of the calibration period, the users’ unique starting position is stored and taken into account with their swing. Next, classification begins; once a complete swing is sensed, a MQTT publish statement is sent to the server and retrieved by Unity. After the successful transmission, the program will terminate. In the case that there is no swing detected, an MQTT string “noSwing” will be published to the server and received by Unity, which results in the “startClassifier” string being resent to the python file. The classification function will be invoked in a cyclic manner until a swing is sensed. GitHub Link.
Joystick Game Controller Serial Input
In this project, I created a game controller using an Arduino Nano, Joysticks, and Buttons. The Arduino Nano read inputs from the input devices and assigned a “character value” to each respective input. Joysticks were assigned: Right (r), Left (l), Up (u), and Down (d). The Buttons were assigned High (1) and Low (0). The Arduino was then connected directly to a PC and the data was transmitted directly to a Python script. The Python script leveraged the “pyautogui” and “pyserial” libraries to translate the “character values” from the serial port into keyboard actions on the users’ PC. Lastly, I created a basic snake game to allow the user to test the keyboard inputs and to play the game with the joystick input device. GitHub Link.
Gesture Recognition Device
In this project, I created a device that had gesture recognition capabilities using an Arduino Nano and its Inertial Measurement Unit. My recognition model was gesture recognition. The categories/classes (Target Gestures) included: punches, arm flexing, and wrist flip/rotation. In order to trigger the sample window the sensor inputs must be above a certain threshold level. In my code, the acceleration threshold was set to 2.5 G’s of force and the sample window was about 1 second of sampling (about 119 samples/second). Features that were used to train my model included: x/y/z acceleration and x/y/z gyroscope input values. The Serial Plotter made it much easier to visualize the response of the Arduino to my arm movements. I was able to see that different motions resulted in very unique response curves. After collecting the data, I used the TinyML library to train my models and generate loss functions for each trial. Next I converted my trained model to Tensor-Flow Lite, which was then encoded into an Arduino Header File. Lastly, I included this trained model header file into my final Arduino testing code, to properly recognize my motions. GitHub Link.
Color Recognition Device
i. In this project, I created a device that had color recognition capabilities using an Arduino Nano and its Proximity & Color Sensors. My recognition model was color recognition. The categories/classes (Target Colors) included: Blue, Green, and Orange. In order to trigger the sample window the object must be well-lit and the object must be close to the sensor. Additionally, the sample window was dependent on whether the conditions for lighting and closeness were met. Features that were used to train my model included: Red/Green/Blue ratios and Closeness index. The Serial Plotter was useful to see the relative difference in color patterns with respect to different objects. I was able to see that different objects had unique ratios of RGB outputted to their response curves. After collecting the data, I used the TinyML library to train my models and generate loss functions for each trial. Next I converted my trained model to Tensor-Flow Lite, which was then encoded into an Arduino Header File. Lastly, I included this trained model header file into my final Arduino testing code, to properly recognize my motions. GitHub Link.
WiFi MQTT with BLE Servers/Clients
In this project, I created several Clients & Servers using an Arduino Nano 33 BLE Sense, an ESP32 DevKit, and an MQTT Protocol with Ubidots. Using the Arduino Nano’s on-board sensors, I captured the Proximity and IMU data. Next, the information is processed through the Arduino and sent as “read/notify characteristics” by creating a peripheral (server) via the Arduino Nano’s BLE, Bluetooth capabilities; it is openly accessible to any clients with the UUID. Lastly, we utilized the ESP32 as the receiving-end of the information by generating a Client that can access the given IMU/Proximity characteristics on the BLE Server. Next, MQTT transmits the data that we received on the ESP32 to our final clients that were subscribed to the respective data. This process was achieved by utilizing an MQTT Broker that was designed in the Arduino IDE to take our data from the ESP32 and send it to our Ubidots client via a WiFi/TCP connection rather than the previously used BLE connection. Our Ubidots client, that is subscribed to the IMU/Proximity “topics” is now able to read values from the original sensors on the Arduino in real-time. Essentially, any topic that is published to the MQTT Broker will be available to the clients that are subscribed to that specific topic. This model is often referred to as the mail center model because the MQTT acts as a package delivery service for clients. The ultimate goal of this project was to become familiar with BLE and MQTT data transmission, and to capture real-time sensor data from a remote location. GitHub Link.
