Current AI interfaces reduce human expression to typed text, leaving out what the body communicates without words. This project investigates how somatic signals can become a primary mode of input.
Five experiments across typing rhythm, breath, pulse, facial expression, and movement each confirmed the body carries contextual information a chat window can't reach, yet each returned responses to the screen rather than back to the body.
These findings converge in Residual Motion, a prototype for solo ACL rehabilitation where pose detection and a wearable knee sensor feed a particle visualisation and an AI voice designed to witness rather than instruct.
Experiment 1
p5.js
TYPING ANALYSIS
Experiment 2
Arduino, THREE.js
BREATH
Experiment 3
Arduino, THREE.js
HEART RATE
Experiment 4
MEDIAPIPE, SHADERS
MOOD
Experiment 5
MOVENET, p5.js
Body moVEMENTS
Residual Motion
MOVENET, ESP32
RECOVERY
Experiment 1
p5.js
TYPING ANALYSIS
Experiment 2
Arduino, THREE.js
BREATH
Experiment 3
Arduino, THREE.js
HEART RATE
Experiment 4
MEDIAPIPE, SHADERS
MOOD
Experiment 5
MOVENET, p5.js
Body moVEMENTS
Residual Motion
MOVENET, ESP32
RECOVERY
PART ONE
Semester 1 focuses on developing visual and interaction aesthetics
for each bodily input modality and exploring how different
technologies can shape the felt experience of embodied AI interaction.
Before designing direct body-based inputs, the first experiment aims to examine whether existing text input already carries embodied information. Built on p5.js, the prototype tracks micro rhythms of writing like keystroke timing, hesitation patterns, revision rhythms, and pause durations.
INPUT: KEYBOARD
The prototype designed with two phases, participants first transcribe a fixed poem, then write freely about their day. While data is captured in the background.
Phase 1: Poem transcription. The interface enforces accuracy, advancing only on correct input. Participants described this as "meditative", isolating the physical act of typing from thought.
Phase 2: Freestyle writing. Pauses lengthen as participants "think about feelings" and search for words—the visible labor of translating internal states into text.
User observations: Testing with six participants revealed that typing styles also changes what pauses mean.
Breath Ring captures breath intensity using an Arduino microphone sensor and translates it into dynamic WebGL shaders. By inviting mindful, controlled breathing, the experiment makes invisible physiology visible, creating an immediate biofeedback loop that directs attention to the breath itself.
INPUT: Breath using Arduino sound sensor
Users blow into the sensor while the visuals mirrors their breath's loudness, duration, and rhythm through a real-time envelope system.
Pulse Check translates heartbeat into light. A finger placed on a KY-039 optical sensor feeds pulse data through Arduino to a Three.js shader visualisation, rendering the body's rhythm as glowing, breathing form. The experiment slows interaction to physiological time, making invisible circulation visible. By requiring sustained touch and stillness, it creates contemplative space between body and screen, a counterpoint to frictionless interfaces.
INPUT: HEART RATE using Arduino HEARTBEAT sensor
Finger on sensor captures pulse, Arduino calculates BPM and streams data to Three.js shader where heart rate drives breathing speed, rotation, and glow intensity
A experiment that uses MediaPipe Face Landmarker to analyse facial blendshapes (smiles, frowns, brow movements) and translates expressions into soft shifting colours through WebGL shaders.
Like a digital mood ring, the interface accumulates emotional signals over time, creating an ambient visual atmosphere that mirrors felt states without requiring verbal articulation.
INPUT: WEBCAM + MediaPipe Face Landmarker
MediaPipe blendshapes detect facial expressions and WebGL shader creates an atmospheric presence
15-second session captures users' movements, Gemini interprets those patterns as emotion and provide a reflective response
An installation that invites participants to move freely for 15 seconds, staying present with how their body feels. As they move, shader-based trails and particles paint their movement in real time.
MoveNet measures velocity, spatial reach, and intensity. These motion data are then passed to the Gemini API, which generates short poetic reflections projected back onto the participant's torso.
INPUT: WEBCAM + Tensorflow, Movenet
MoveNet keypoint tracking generates shader particles, energy of the movements effects particle intensity
setting up the projection and on the fly code tweaks
PART TWO
Semester 2 narrows the catalogue into a single project — applied to the long, quiet space of recovery.
01 — CONTEXT
ACL recovery takes months, and between clinic appointments most of the work happens at home, the same exercises repeating day after day with no one there to confirm the form is right. Residual Motion is built for that in-between space, a home rehab system for the hours that would otherwise be spent alone with a recovering body.
02 — SENSING
Two sensors run in parallel. The laptop's webcam handles whole-body posture, with MoveNet tracking seventeen keypoints at thirty frames per second, enough to follow a full exercise from a few metres away. A small custom module strapped to the knee, built around a Seeed nRF52840 with its onboard IMU, streams six-axis motion over Bluetooth at fifty hertz, picking up the fine tremor and joint angles a camera can't resolve.
CAMERA
17 keypoints @ 30 fps
MoveNet · TensorFlow.js
KNEE IMU
6-axis motion @ 50 Hz
Bluetooth streaming
03 — VISUALISATION
Movement is rendered back as a particle figure, roughly 260,000 GPU particles drawn in two layers, one filling the body's silhouette and another tracing the skeleton. The particles settle when the body holds still and scatter as it moves, mapping each limb in real time so the motion of the exercise becomes something you can see as you perform it.
04 — COMPANION
Instead of asking how you feel, the app streams movement data to an AI language model in the background, rep count, form quality, range of motion, knee angle, tremor, and lets it speak a sentence or two aloud when something shifts. The AI isn't there to correct or instruct, only to register what the body is doing and occasionally reflect it back in its own voice. Most of the time, it says nothing at all.
PART THREE
Code as the Canvas is a short publication on two semesters of building with Claude Code, a record of the experiments, learnings, and recommendations that came out of working with a tool that can read the codebase, write into it, and hold the whole project in context. Part reflection, part practical guide for designers stepping into the same territory.
Drag a corner or click the edges to turn a page.