Wrist Gesture Sensing with ESP32 and an accelerometer
Posted: Tue Mar 19, 2024 11:20 pm
This is an experiment to see if a computing device can make sense of gestures made with a person's wrist. It identifies the Fabulous Four Gestures of next, previous, accept, and cancel. It is extensible to train and identify additional gestures. It uses an ESP32 and LIS3DHTR accelerometer. It uses a Movement Observation Matching algorithm I designed to make sense of a person's wrist movements.
I created a 5 minute video briefing on Wrist Gesture Sensing at https://www.youtube.com/watch?v=nNwV_FyjRPc
Human interfaces to computing devices often benefit from having 4 primary controls. I call these the Fabulous Four Gestures: Next - person's goal is to move to a subsequent choice; Roll your fist counter clockwise, to the left. Previous - person's goal is to return to a prior choice; Roll your fist clockwise, to the right. Accept - person's goal is to approve a selection; I adopted the American Sign Language (ASL) movement to say yes, take a hand and make it into a fist and bob it back and forth, resembling a head nodding. Cancel - person's goal is to disapprove a selection; I adopted ASL to say no, take your first two fingers and tap them with your thumb, resembling a mouth saying no.
Serial monitor shows status and progress to recognize gestures.
Sensing gesture
Comparing gestures
Accellerometer gesture next 0.70, 0.74, 0.81, 0.83, 0.77,
Accellerometer gesture previous 0.67, 0.48, 0.39, 0.46, 0.55,
Accellerometer gesture cancel 0.55, 0.53, 0.57, 0.55, 0.58,
Accellerometer gesture accept 0.51, 0.49, 0.53, 0.57, 0.60,
Sum of averages, gesture recognized: next
Highest average, gesture recognized: next
In the case where no gesture is recognized the code shows:
Sum of averages, gesture inconclusive
Highest average, gesture inconclusive
Movement Observation Matching
Movement Observation Matching (MOM) algorithm detects a person's wrist movements. It is a template system. It compares a person's movements to a library of recorded templates. Movement and template recordings begin with detection of a person's movement. A variety of techniques determine the conclusive determination of movements to the templates.
Templates last 2 seconds, with 50 frames of accelerometer X, Y, Z data and 40 milliseconds between each frame. Recording begins after the person begins movement. Movements are the same duration and frames as templates.
MOM uses multiple templates for the same gesture to identify gestures. The default settings record 5 templates for each gesture. Then compare a persons latest gesture to the templates.
MOM Algorithm
Settings in Accelerometer.h control the types of gestures, the number of templates for each gesture type, the number of frames, and the template duration.
MOM uses these matching algorithms:
Sum of averages, chooses the gesture type with the overall highest average matching ratio. Ratio must be 55% or higher to be selected.
Highest average, chooses the gesture type with the highest single matching ratio. Ration must be 55% of higher to be selected.
MOM is extensible to include additional matching algorithms.
Find the implementation in the ReflectionsOS repository here: https://github.com/frankcohen/Reflectio ... ometer.cpp
Use Arduino IDE 2.3 or later. Code requires changes to be compatible with Arduino IDE 1.x nor Platform IO.
Requires these libraries:
LISDHTR Accelerometer, https://travis-ci.com/Seeed-Studio/Seee ... nch=master
ESP32 board, https://github.com/espressif/arduino-esp32
We have these planned improvements to MOM:
Template effectiveness learning. MOM recognizes and promotes templates correctly recognized, and denegrates and removes templates that do not recognize movements. It will do this by observing a person's movements after detecting a gesture. For example, when a person gestures "previous" after MOM detects a "next" gesture more than 50% of the time it is fair for MOM to identify the "next" gesture as incorrectly identified.
Automatic new template detection. MOM recognizes new templates from movements that do not match templates. For example, when a person makes 5 "next" gestures then an unidentified new gestures then another 5 "next" gestures, MOM adds the unidentified movement gesture as a new "next" gesture.
Template sharing between people. MOM shares templates with high success ratings among other devices.
MOM has yet to be proven in these instances:
Is there a difference in recognition between left and right handed people?
What impact does a benign essential tremor (ET), Parkinson's disease, Zlzheimer's disease, and Multiple Sclerosis (MS) have on gesture recognition?
Do individuals gesture differently?
Reflections is an open-source hardware and software platform for building entertaining mobile experiences. Details at https://github.com/frankcohen/ReflectionsOS
I created a 5 minute video briefing on Wrist Gesture Sensing at https://www.youtube.com/watch?v=nNwV_FyjRPc
Human interfaces to computing devices often benefit from having 4 primary controls. I call these the Fabulous Four Gestures: Next - person's goal is to move to a subsequent choice; Roll your fist counter clockwise, to the left. Previous - person's goal is to return to a prior choice; Roll your fist clockwise, to the right. Accept - person's goal is to approve a selection; I adopted the American Sign Language (ASL) movement to say yes, take a hand and make it into a fist and bob it back and forth, resembling a head nodding. Cancel - person's goal is to disapprove a selection; I adopted ASL to say no, take your first two fingers and tap them with your thumb, resembling a mouth saying no.
Serial monitor shows status and progress to recognize gestures.
Sensing gesture
Comparing gestures
Accellerometer gesture next 0.70, 0.74, 0.81, 0.83, 0.77,
Accellerometer gesture previous 0.67, 0.48, 0.39, 0.46, 0.55,
Accellerometer gesture cancel 0.55, 0.53, 0.57, 0.55, 0.58,
Accellerometer gesture accept 0.51, 0.49, 0.53, 0.57, 0.60,
Sum of averages, gesture recognized: next
Highest average, gesture recognized: next
In the case where no gesture is recognized the code shows:
Sum of averages, gesture inconclusive
Highest average, gesture inconclusive
Movement Observation Matching
Movement Observation Matching (MOM) algorithm detects a person's wrist movements. It is a template system. It compares a person's movements to a library of recorded templates. Movement and template recordings begin with detection of a person's movement. A variety of techniques determine the conclusive determination of movements to the templates.
Templates last 2 seconds, with 50 frames of accelerometer X, Y, Z data and 40 milliseconds between each frame. Recording begins after the person begins movement. Movements are the same duration and frames as templates.
MOM uses multiple templates for the same gesture to identify gestures. The default settings record 5 templates for each gesture. Then compare a persons latest gesture to the templates.
MOM Algorithm
Settings in Accelerometer.h control the types of gestures, the number of templates for each gesture type, the number of frames, and the template duration.
MOM uses these matching algorithms:
Sum of averages, chooses the gesture type with the overall highest average matching ratio. Ratio must be 55% or higher to be selected.
Highest average, chooses the gesture type with the highest single matching ratio. Ration must be 55% of higher to be selected.
MOM is extensible to include additional matching algorithms.
Find the implementation in the ReflectionsOS repository here: https://github.com/frankcohen/Reflectio ... ometer.cpp
Use Arduino IDE 2.3 or later. Code requires changes to be compatible with Arduino IDE 1.x nor Platform IO.
Requires these libraries:
LISDHTR Accelerometer, https://travis-ci.com/Seeed-Studio/Seee ... nch=master
ESP32 board, https://github.com/espressif/arduino-esp32
We have these planned improvements to MOM:
Template effectiveness learning. MOM recognizes and promotes templates correctly recognized, and denegrates and removes templates that do not recognize movements. It will do this by observing a person's movements after detecting a gesture. For example, when a person gestures "previous" after MOM detects a "next" gesture more than 50% of the time it is fair for MOM to identify the "next" gesture as incorrectly identified.
Automatic new template detection. MOM recognizes new templates from movements that do not match templates. For example, when a person makes 5 "next" gestures then an unidentified new gestures then another 5 "next" gestures, MOM adds the unidentified movement gesture as a new "next" gesture.
Template sharing between people. MOM shares templates with high success ratings among other devices.
MOM has yet to be proven in these instances:
Is there a difference in recognition between left and right handed people?
What impact does a benign essential tremor (ET), Parkinson's disease, Zlzheimer's disease, and Multiple Sclerosis (MS) have on gesture recognition?
Do individuals gesture differently?
Reflections is an open-source hardware and software platform for building entertaining mobile experiences. Details at https://github.com/frankcohen/ReflectionsOS