AI/ML
Hand-Gesture Drawing Tool
Draw in the air: real-time hand tracking turns fingertip position into geometric constructions on a live camera feed, with both hands working independently.
- Built
- 2025
Overview
A computer-vision drawing tool with no input device: MediaPipe's 21-point hand model tracks both hands on a webcam feed, a pinch starts and ends a line, and the result is a geometric construction drawn in the air.
Two problems make this harder than it sounds. The first is that a pinch is ambiguous, because thumb and forefinger approach at an angle, so a pure distance threshold either fires early or refuses to fire at all. The second is that hands shake, and a raw landmark stream is unusable for drawing straight lines.
Both are solved in the signal rather than in the interface: distance and angle together for gesture detection, and a One Euro filter for smoothing, which suppresses tremor at low speed without adding the latency a fixed-window average would introduce at high speed.
What it does
- Dual-hand recognition. Both hands draw simultaneously and independently, colour-coded.
- Pinch detection combining fingertip distance and approach angle.
- Temporal smoothing with a One Euro filter, tuned to keep fast strokes responsive.
- Dynamic connection: new lines snap to existing endpoints or to the nearest point on an existing segment.
- Live distance readouts between endpoints as a construction is drawn.
- A triple-pinch exit gesture, so the application can be closed without touching a keyboard.
How it works
- MediaPipe's 21-landmark hand model, tracked per hand per frame with independent state machines.
- Vector projection onto line segments to find the nearest point for snapping, rather than only testing endpoints.
- One Euro filtering, a speed-adaptive low-pass filter, which is the standard answer to the jitter-versus-lag tradeoff in interactive tracking.
- Transparency overlays for connection hints and proximity indicators, so guidance never obscures the canvas.
Problems and answers
Distance alone misclassifies pinches, because fingers meet at an angle.
Combine distance with approach angle before declaring a pinch.
Hand tremor makes straight lines impossible.
Apply a One Euro filter, which smooths hard when slow and lightly when fast.
Joining a new line to an existing one needs sub-pixel intent.
Project the cursor onto nearby segments and snap to the projection, not just to endpoints.
Two hands drawing at once means two independent gesture states.
Track per-hand state separately and colour-code the feedback so the user can tell them apart.
Where it goes next
- Gesture-based shape recognition, snapping freehand strokes to geometric primitives.
- Depth estimation for drawing in three dimensions.
- Per-user gesture calibration learned over a session.