Driving a Unity Humanoid Avatar with MediaPipe, No Depth Camera Required

Motion capture used to mean a studio. Reflective markers on a bodysuit, a ring of infrared cameras, someone in another room watching a skeleton move on a monitor. That setup is still how most film and AAA game animation gets made, and it is still expensive, slow to set up, and out of reach for almost anyone who is not a studio.

What changed is that a normal webcam can now do a version of this. Google’s MediaPipe can look at a single 2D camera feed and estimate a full 3D skeleton from it: not just where your shoulder is on the screen, but roughly where it is in space. That is a useful piece of technology sitting in a free, open library. The part that was missing was the second half: taking those 3D points and actually using them to move a character. Most examples stop at drawing dots on a video feed. Almost nothing shows you how to go from those dots to a rigged, animated 3D avatar inside a game engine.

I built that second half. This is how it works, and the code is on GitHub if you want to use it yourself.

What MediaPipe actually gives you

MediaPipe’s pose model looks at a video frame and returns 33 points: nose, shoulders, elbows, wrists, hips, knees, ankles, and a few more. Each point comes with two sets of coordinates. One is where that point sits on the flat image, useful for drawing an overlay on the video. The other is a 3D position in space, measured relative to the middle of your hips, as if a virtual camera were watching you from a fixed spot. That second set is the interesting one, and it is the part most tutorials ignore. A 2D point can tell you a wrist moved up and to the left on screen. It cannot tell you whether the arm reached forward or stayed close to the body, because both of those look identical from directly in front of a camera. The 3D landmarks solve that. They are what makes it possible to reconstruct an actual pose in three dimensions from one ordinary camera, instead of needing two or three cameras to triangulate depth the old way.

Points are not a skeleton

Here is the part that is easy to underestimate before you try it: 33 floating points in space are not a character. A Humanoid avatar in Unity is a hierarchy of bones, each one a child of the bone above it, each one only able to rotate around its own joint. The shoulder does not know where the wrist is. It only knows its own rotation, and the elbow’s rotation depends on the shoulder’s, and the wrist’s depends on both. So the real problem is not detecting a pose. It is translating a cloud of points into a chain of joint rotations that respects how a skeleton actually works, and doing it every single frame, thirty times a second, without the character’s joints twisting into something anatomically impossible the moment you turn sideways.

The approach: measure a direction, apply a rotation

The trick that makes this tractable is to stop thinking about positions and start thinking about directions. Take the shoulder and the elbow. MediaPipe gives you both as points in space. Subtract one from the other and you get a direction vector: which way the upper arm is currently pointing. Do the same thing once, at the very start, while the person is standing in a known reference pose (a simple T-pose works well), and you get a second direction: which way the upper arm bone points when the avatar is doing nothing at all. Now you have two directions for the same bone: where it started, and where it is now. The rotation that turns one into the other is exactly the rotation you need to apply to that bone on the avatar. Unity has a single function for this, Quaternion.FromToRotation, which takes two direction vectors and hands back the rotation between them. Apply that rotation to the avatar’s upper arm bone, and the upper arm follows the tracked arm. Repeat that same calculation, independently, for every major joint pair: shoulder to elbow, elbow to wrist, hip to knee, knee to ankle. Each bone only cares about its own two points, which is what keeps the whole thing from needing to understand the full kinematic chain explicitly. The chain sorts itself out because Unity’s own bone hierarchy applies each rotation on top of its parent’s automatically.

The spine, chest, and hip twist are handled a little differently, since a believable torso needs more than one bone reading the same movement independently. Those get blended so the rotation is distributed up the spine instead of concentrated in a single joint, which is closer to how a real torso actually bends.

Why calibration matters more than the tracking itself

Everything above depends on that first reference pose being right. If the “at rest” direction you captured does not match how the avatar’s rig is actually built, every single frame after that inherits the error. Small mistakes here do not look like small mistakes. They look like an arm that is subtly, permanently wrong, in a way that makes the whole thing feel broken even though the tracking is fine. There are two ways to get that reference pose. One is live: stand in a T-pose in front of the camera, hold still for a few seconds, and the system captures whatever MediaPipe sees at that moment. That works, but it depends on the person’s pose actually being clean, and on tracking being stable in that instant. The other way is to calibrate against a single reference photograph instead of a live person. Run MediaPipe once against a clean, unambiguous photo of someone in a T-pose, capture that as the reference, and use it every time the app starts. No holding still, no relying on a good live frame, and it is repeatable: the same reference photo produces the same calibration every time.

Running it entirely on the device

None of this needs a server. MediaPipe’s pose model ships as two small neural networks, one that finds a person in the frame and a second that reads their detailed pose from that crop. Both convert cleanly to ONNX and run inside Unity through Sentis, Unity’s own on-device inference engine. The whole pipeline, camera capture, detection, pose estimation, and avatar retargeting, runs inside a single Unity process. No Python, nothing talking to a remote API, nothing that stops working the moment you lose internet.

What this is actually for

I built this after spending a while on a different pose-tracking system for mobile sports games, tracking a player’s bat and body from a phone camera. That work stayed in 2D, good enough for judging a swing but never meant to drive a full 3D character. This project is the natural next step: given that a plain camera can now recover a real 3D pose, what does it take to put that pose onto an actual rigged avatar, the kind you would use in a game or a VTuber-style stream?

The answer turned out to be smaller than I expected once the direction-vector trick clicked, but getting there meant working through a fair number of ways to get it subtly wrong: mismatched coordinate systems that mirrored every forward and backward motion, a calibration bug that made every gesture feel sluggish, a flickering overlay caused by updating visibility on frames where nothing new had actually arrived. None of that shows up in the finished code. All of it shaped how the finished code is written.

The full implementation, the retargeting script, the calibration system, the ONNX conversion steps, and a setup guide, is public. It works with any Humanoid-rigged avatar, not a specific character bundled with the project. If you drop it into a project with your own model, it should track you the moment you press play.