MediaPipe Arm Tracking for Mixamo FBX Characters

**Real-time arm and leg control using MediaPipe Pose in TouchDesigner**

Overview

This project demonstrates how to use MediaPipe Pose detection to control arm and leg  movements on Mixamo-rigged FBX characters in TouchDesigner. Perfect for interactive installations, live performances, avatar rigging for game development, or learning motion capture basics! 

My goal with this was to create a HD mediapipe responsive avatar to feed into stream diffusion. There are several reasons why you might prefer this method of character rigging to wire into stream diffusion. It doesn't require a clean/stable background or good lighting, you can rig any kind of character utilizing all open source tools (all credited below), and it provides a much heavier reference for stream diffusion than the ipadapter. It's also helpful for people currently (as of January 2026) utilizing stream diffusion locally who can't currently run control nets on newer 5xxx amd 6xxx machines locally. The main downside is that the current retarget code I've uploaded is a limited range of motion versus straight media pipe. 

Features

 **Simple Setup** - Get running in 10 minutes

 **Real-time Tracking** - 60fps arm and leg movement

 **Natural Motion** - Smooth, clamped arm and leg rotations

 **Extensible Code** - Easy to add more body parts (elbows, knees, wrists)

 **Well-Documented** - Complete guides included

 What It Does

- Tracks arm up/down movement (90° range in each direction)

- Tracks legs moving outwards 60° range out to each side and 30° in front 

- Positions character based on your body location

- Works with any Mixamo-rigged FBX model

- Runs in real-time with minimal lag

What's Included

 **retarget_code.txt** - Complete tracking code

 **README.md** - Full documentation

 **QUICKSTART.md** - 10-minute setup guide

 **TECHNICAL_BREAKDOWN.md** - Code explanation

 **FAQ.md** - Troubleshooting & common issues

Quick Preview

System Requirements

- TouchDesigner 2023.11760+ (tested on 2025 Commercial)

- Webcam

- Mixamo-rigged FBX model

- Basic TouchDesigner knowledge

 Getting Started

Option 1: Use the .toe file

1. Download the .toe project file

2. Update the FBX path to your model

3. Stand in front of webcam

4. Done!

Option 2: Build from scratch

1. Follow QUICKSTART.md

2. Learn the setup process

3. Upload 3 images doing a T-pose from different angles into Rodin HyperAI to create a 3d model
4. Clean it up in Blender (or equivalent) and upload 3d model into Mixamo to get a rigged model

X3 images > Rodin HyperAI > Blender > Mixamo = rigged fbx

 Key Learnings from Development

Through building this project, I discovered:

- **MediaPipe coordinate system** - Understanding normalized 0-1 coordinates

- **Bone hierarchies** - How Mixamo rigs are structured

- **Rotation axes** - Which rotations control which movements

- **Clamping strategies** - Preventing unnatural 360° rotations

- **Real-time optimization** - Keeping code fast for 60fps

Challenges Overcome

1. **Finding neutral positions** - Each model has different default rotations

2. **Coordinate translation** - Converting MediaPipe 0-1 to world units

3. **Natural movement** - Determining the right sensitivity multiplier

4. **Model orientation** - Getting spine/leg rotations correct

5. **Channel naming** - MediaPipe output structure

Future Extensions

This is a foundation that can be extended with:

Elbow bending

Knee tracking

Wrist/Hand/finger tracking

Full body IK

Motion recording

Multi-person tracking

Technical Details

**Language:** Python

**Framework:** TouchDesigner

**ML Model:** MediaPipe Pose

**Run Time:** Real-time (60+ fps)

**Code Lines:** ~130

Why I'm Sharing This

I spent hours debugging and testing to get this working smoothly. By sharing this fully documented project, I hope to:

- Save others the trial-and-error process

- Provide a foundation for more complex tracking

- Demonstrate clean code practices in TouchDesigner

- Contribute to the community that's helped me learn

Credits & Thanks

- **Daydream.live** - hosting cohort and server based option for stream diffusion

- **MediaPipe** - https://github.com/torinmb/mediapipe-touchdesigner

- **Mixamo** - Free character rigging

- **TouchDesigner Community** - Endless help and inspiration

- **Rodin Hyper AI** - opensource tool to create 3D model

- **Blender** - clean up 3D model

- **Mixamo** - free tool with adobe account for rigged model

- **hyperbodied** - Development and documentation

Includes:

- Complete code

- Documentation (README, Quick Start, Technical Breakdown, FAQ)

- Example .toe file 

 License

Free to use and modify. Attribution appreciated but not required.

 Questions?

Feel free to ask in the comments or reach out! I'm happy to help troubleshoot or explain anything.

**Project Status:** Version 2, will be updated further with elbows, knees, wrists, and head movements. I will additionally finish the architecture and code for the 2D puppet rigging and add that file. 

**Difficulty:** Beginner-Intermediate

**Last Updated:** January 2026

---

**v2.0** - second release with arm tracking and leg tracking

---