Find any button on a flat touchscreen, by sound.
Kiosks, ticket machines and appliance panels are smooth glass, so blind people can't feel where the buttons are. TouchGuide reads the screen through a phone camera and steers your fingertip with sound, speech and vibration. Put on the blindfold and try to order lunch.
Your pointer is your fingertip. Hover to explore and click to tap. On a phone, drag to explore and tap to press. With a keyboard, focus the screen, move with the arrow keys and tap with Enter. Sound map: the beep comes from the side the button is on, a higher pitch means move up, faster beeps mean closer, and a steady chord means tap.
Point it at a real screen.
Take a photo of a kiosk, ticket machine, card terminal or appliance panel. TouchGuide reads it and turns it into a touch map on this phone. Explore the map with your finger to learn the layout, then reach for the real thing.
Checking whether Claude can read photos in this view…
No photo yet. Take one, choose one, or start with the sample ticket machine.
Drag a finger or your pointer over the photo to hear what is where. Pick a button on the right and TouchGuide guides your finger to it on the photo, then tells you where it sits on the real screen.
How TouchGuide works
One phone and no changes to the kiosk. The camera sees the screen and your hand at the same time, and TouchGuide runs this loop many times a second.
See the screen
A vision model finds every button, key and label in the camera frame, with its position and current state.
Track the fingertip
Hand tracking finds the tip of your index finger in the same frame, about 30 times a second.
Guide
Stereo sound says left or right, pitch says up or down, and the beeps speed up as you get close. A steady chord means tap.
Watch for changes
After each tap it reads the screen again, says what changed, and catches pop-ups like time-out warnings.
Learn the sound map
Wear headphones if you can. Each button plays the cue you would hear in that situation.
What is real in this prototype
- The guidance engine is real: spatial sound, speech, vibration, explore by touch, and the step planner that picks the next button.
- On the Try it page, the practice devices tell TouchGuide what they show. That stands in for the camera's screen reading, and your pointer stands in for fingertip tracking.
- On Real screens, the screen reading is real. Claude looks at your photo and returns each control with its position.
What came before
Researchers have worked on this for about ten years. VizLens and StateLens from Carnegie Mellon used crowd workers to label kiosk screens. Toucha11y from the University of Maryland placed a small robot on the kiosk to press buttons. blindAR from FAU Erlangen ran on HoloLens glasses. Each needed people, extra hardware, or a headset. TouchGuide needs only the phone in your pocket.
Next steps
- Native app with live hand tracking on the phone's camera.
- Private mode for card terminals: frames never leave the phone while you enter a PIN.
- Shared kiosk maps, so a layout one person reads is ready for the next.
- Camera glasses, so both hands stay free.
60-second demo script for judges
- Turn on sound and pick the food kiosk. Press Watch a demo run so the room hears a virtual fingertip follow the beeps to each button.
- Hand a judge headphones. Start the mission without TouchGuide, blindfolded. After 20 seconds of silence, press Give up.
- Start it again with TouchGuide. Point out that the beeps move between ears, rise and fall, and speed up near the button.
- Wait on a screen for 35 seconds to show the time-out pop-up being caught and handled.
- Open the results and compare the two runs.
- Go to Real screens and take a photo of any machine in the room.