Chen Liang

This work wouldn’t be possible without the amazing collaborators and advisors on this project - Yuxuan Liu, Martez Mott, and @anhongguo.bsky.social , thank you so much! (8/8)

01.08.2025 03:33 — 👍 0    🔁 0    💬 0    📌 0

We envision HandProxy will expand the design of existing speech interfaces in XR, and hope this can be implemented on the system level to provide a uniform, cross-application input solution for accessible control and input automation of hand interactions. (7/n)

01.08.2025 03:32 — 👍 0    🔁 0    💬 0    📌 0

A user study on various hand interaction tasks showed a 91.8% command execution accuracy, with an average of 1.09 attempts per speech command. HandProxy was able to interpret diverse commands users issued during tasks, and users found it flexible, effective, and easy to use (6/n)

01.08.2025 03:32 — 👍 0    🔁 0    💬 0    📌 0

To do so, HandProxy continuously captures speech input and uses LLM to decompose it into a list of hand control primitives. They are used to calculate the 3D hand joints for the hand movement, and joint data is streamed to the virtual environment for hand control (5/n)

01.08.2025 03:32 — 👍 0    🔁 0    💬 0    📌 0

Users can issue various commands, such as gesture (pinch, grab the cube), timing (stop, undo), movement (move up, twist left), or high-level interaction goals (reduce the volume, increase brightness) where HandProxy interprets and decomposes it into detailed hand controls (4/n)

01.08.2025 03:32 — 👍 0    🔁 0    💬 0    📌 0

A figure showing a comparison between the existing approach and the proposed HandProxy approach. The existing approach usually relies on a list of discrete, predefined commands such as "Activate the lever," "Select the cube," "Open the box," "Close the box," "Resize the window," and "Resize the object". The proposed HandProxy approach supports more natural, expressive interactions using varied and context-rich phrases like "{punch, push, cut} the cube," "grab the {lever, fruit}," "{move, twist} the {hand, object}," "put apple into the basket," "pinch up the window corner," and general control commands like "stop, continue, undo." These inputs are decomposed into virtual proxy hand movements, which then interact directly within a virtual environment.

HandProxy introduces a virtual hand as an interaction proxy. It expands the affordance of speech interfaces to support expressive hand interactions. Users can initiate hand interactions through natural speech, and the system interprets and decomposes it to hand movements (3/n)

01.08.2025 03:31 — 👍 0    🔁 0    💬 0    📌 0

a figure showing possible factors causing hand interactions to be unavailable, including situational impairments (e.g., occupied hands, fatigue after long term use), environmental constraints (e.g., confined physical space, virtual object out of reach), and upper-body limitations (spasm or tremor, rapid fatigue, poor coordination, difficulty controlling distance/direction)

Hand interactions are widely used in XR but are not always feasible, e.g., due to (situational) impairments, environmental constraints. Speech can be used as an alternative, but is often limited to initiating basic gestures and system controls (2/n)

01.08.2025 03:28 — 👍 0    🔁 0    💬 0    📌 0

Introducing HandProxy, a speech-controlled virtual proxy hand that interprets the user’s natural speech input and performs hand interactions in XR on the user's behalf. Accepted at #IMWUT 2025 (1/n)
Paper: bit.ly/4fjdEJN
Demo Video: bit.ly/40KXtPy

01.08.2025 03:25 — 👍 1    🔁 0    💬 7    📌 0