The idea
The cheapest piano ever made: a sheet of paper and a webcam. You print a page of piano keys, point a camera at it, and play with your fingers on the paper. No touch sensor, no hardware — which means the computer has to infer a key press from vision alone. That inference problem is the whole project.
How it works
- Print the keyboard — a sheet of piano keys with ArUco markers at the corners
- Each frame, the webcam detects the ArUco markers
- A homography maps the camera image onto the flat keyboard plane — angle and perspective stop mattering
- MediaPipe Hands tracks the hand landmarks in real time
- A smoothing filter stabilizes the fingertip stream — raw landmarks jitter
- Finger-speed analysis: a sharp downward motion inside a key's area reads as a press; a hover doesn't
- The mapped key triggers its note
Everything is Python: OpenCV for the camera pipeline and ArUco detection, MediaPipe for hand tracking, and the geometry — homography, coordinate mapping, velocity — computed on top.
The hard problems
- A webcam can't see touch. There is no depth information, so "pressed" versus "hovering above" must be inferred from 2D motion — the finger-speed calculations exist precisely to separate an intentional strike from a hand passing over the keys.
- Jitter versus latency. Smooth the landmarks too little and keys misfire; smooth too much and the instrument feels laggy. An instrument's latency budget is brutal — players feel tens of milliseconds — so the filter is a constant trade-off.
- Perspective. Without correction, where a key sits in the image depends entirely on the camera angle. The ArUco markers plus homography make the layout position-independent — the sheet can sit at an angle, or even shift mid-play, and the key boundaries follow.
- Hands occlude the sheet. The very act of playing covers parts of the paper — markers live at the corners so tracking survives a hand in the middle of the frame.
Status & what's next
A working local prototype — the loop from printed paper to sounding notes runs end to end. On the list: velocity-sensitive volume so striking harder plays louder, chords and polyphony, and profiling the pipeline to shave latency. The dream version runs in the browser — MediaPipe ships a JS build — embedded right here on this page, so you could print the sheet and play it without installing anything.
Why it's on a .NET portfolio
Because it's the opposite of my day job, and that's the point. Web products are about reliability and users; this is about geometry, signal processing and real-time constraints. Different stack, different failure modes — same habit of taking a problem apart until it works.