Semaphore with MediaPipe Pose, retuned for smaller humans

Overview#

Flag semaphore is a signal alphabet where every letter is a pair of arm angles. That makes it a perfect party trick for a pose detector: point MediaPipe at a webcam, bucket each arm direction to the nearest 45 degrees, and look up the pair in a dictionary.

Everything Is Hacked ↗ (Fletcher Bach) built exactly that in Semaphore ↗, a full-body keyboard where you type by waving your arms. The LICENSE file says GPL-3.0, but the terms that matter are in a commit message: “Do whatever you want with my code, just don’t make it boring”. His demo video ↗ shows the whole thing in action, and the project later evolved into a full-body game controller called Gamebody ↗.

I took the project to a classroom setting where junior high and high school students would be the ones signaling. That exposed the one thing the original did not account for: it was tuned on one adult body, at one distance from one webcam. This post covers how the original reads a signal, where body size actually matters, and what I changed so that every size of human in the room could type.

How the original reads a signal#

MediaPipe’s Pose solution returns 33 body landmarks per frame in normalized image coordinates, each with a visibility score. The signaling logic in semaphore.py is small:

1. Each arm is the triplet (shoulder, elbow, wrist). A limb counts as “pointing” when the elbow angle is within 20 degrees of straight and the forearm is at least 0.8 times the upper arm, so a bent or relaxed arm never registers.
2. The arm direction is the angle from shoulder to wrist, collapsed to the nearest 45 degrees. Eight directions per arm.
3. The pair of directions is looked up in the SEMAPHORES dictionary, which follows the standard alphabet closely. (90, 90) is space, or enter when the numerals layer is active.

Everything in that chain is scale-invariant. Angles do not care how tall you are, and the two sanity checks are ratios: forearm versus upper arm, both drawn from the same body in the same frame.

The keyboard’s extra commands ride on other gestures: hands over the mouth for backspace, a squat for the numerals layer, open hands for shift, a jump to repeat the last character, leg lifts for command and control, and crossed-arm-plus-leg-angle combinations for arrow keys.

Where body size actually matters#

I went through the code expecting to rewrite the pose math for kids and did not need to. The weak point is narrower than that. Three gestures are thresholded on absolute distances in normalized frame units, and those constants were clearly tuned on one adult filling one webcam:

MOUTH_COVER_THRESHOLD = .03   # hands over mouth, frame-fraction distance
SQUAT_THRESHOLD       = .1    # hip-to-knee vertical distance
JUMP_THRESHOLD        = .0001 # hips rise/fall per frame

A smaller person, or anyone standing farther from the camera, shifts every one of these. The mouth-to-palm gap shrinks, so backspace fires when the hands are nowhere near the face. The hip-to-knee gap shrinks, so the squat layer latches on while standing. The hip rise of a jump compresses below the threshold, so repeat stops working. Nothing is wrong with the pose math; the constants are just calibrated for a body that is not twelve years old.

The fork#

My changes are one commit on top of the original, repurposing it from a keyboard emulator into a self-contained alphabet display. Concretely:

• The keyboard injection is gone. The original types through the keyboard library, which needs Accessibility permissions and an editor to type into; the fork keeps an on-screen string instead, where space and delete edit the string directly and enter commits it and saves the frame as a PNG. The demo becomes a single window a projector can show, with no other app involved.
• The squat layer now yields uppercase, and the squat band widened from 0.10 to 0.12. The original mapped the squat layer to digits and symbols; uppercase letters are the more useful second layer for a classroom.
• JUMP_THRESHOLD dropped from 0.0001 to 0.00008, because a smaller body produces a smaller apparent hip displacement per frame. Without this the repeat gesture simply never fires for the youngest participants.
• A two-second cooldown between characters (NEW_CHAR_TIMEOUT) makes one pose one letter. Kids hold a signal, it registers, and there is time to read it before the next one.
• The leg gestures are gone, all commented out. Command, control, and the arrow-key combinations require balance and coordination that consume attention better spent on the alphabet itself.
• Selfie flip, landmark drawing, and recording are forced on, and the running string renders in a white box in the bottom right corner so it stays readable on a projector.

The result runs on an Apple Silicon Mac with mediapipe-silicon 0.9.2.1 and OpenCV, one process, one window.

The fix I would make next#

The honest residue is MOUTH_COVER_THRESHOLD, still an absolute 0.03 of frame distance. For students who appear small in frame, hands near the face can still trip a false delete. The codebase already contains the right pattern: crossed arms and open hands are measured against the mouth width, which scales with the person. Backspace wants the same treatment, normalizing the mouth-to-palm distance by a body segment such as the shoulder width, and I would fold that in before the next workshop rather than retune the constant per room.

Closing#

The original Semaphore is a beautifully scoped project: pose landmarks, a dictionary of angle pairs, and a week of readable code. None of that needed touching to work for students. The lesson from the classroom pass is that a gesture UI tuned on one body is a calibration document, and the fix is knowing which three constants carry that calibration. Credit again to Everything Is Hacked for the original and for the most generous commit message in open source.