Why auto-reframe loses the subject

Auto-reframe takes a wide shot and produces a vertical one by deciding, frame by frame, which part of the picture to keep. When it works you stop noticing it exists. When it fails it does something worse than a bad crop: it swings, and a swinging frame reads as a mistake in a way a slightly-off static crop never does.

Here is what it is actually doing, and why the failures cluster where they do.

It is tracking salience, not meaning

The thing being followed is not "the speaker". It is a moving estimate of where the interesting pixels are: faces, motion, contrast, sometimes text. That estimate is recomputed constantly and smoothed over time so the crop glides rather than jumps.

This is a good approximation of attention and a poor approximation of intent. The two come apart in predictable places:

Two people talking. Salience says both faces matter, because both faces are faces. Meaning says the one who is speaking matters. A tracker without audio alignment will sit between them, drift toward whoever moves more, and cross the frame every time the other one gestures. The result is a clip that appears to be looking for something.

A cutaway. The wide shot goes to a slide, a product, a hand. Salience follows the new high-contrast object, then snaps back when the speaker returns. Two swings for a two-second insert.

A speaker who moves in and out of frame. Someone who paces gives the tracker a target that leaves and re-enters. Every re-entry is a decision, and consecutive decisions in opposite directions look like indecision.

Anything where the point is off-centre on purpose. A demo where the screen matters more than the person, a reaction where the joke is someone else's face. Salience picks the loudest thing in the frame. Loudest is not the same as relevant, and no amount of tracking quality closes that gap. It is a different question.

Why the smoothing is the hard part

The obvious fix for a jittery crop is more smoothing, and it works right up until it doesn't. Heavy smoothing turns a fast cross-frame move into a slow one, which is not an improvement: a slow pan to nothing in particular is more distracting than a quick one, because it lasts longer and it looks deliberate.

There is also a lag cost. A crop that is smoothed enough to be calm is, by construction, behind the action. If someone leans forward to make a point, the frame arrives after they do.

The honest position is that this is a trade with no free setting. Calm frames that lag, or responsive frames that twitch. What a tool can do is give you the dial and make the failure visible. MakeRoll draws the detected subject boxes on the preview and shows the crop it chose, so you can see what the tracker latched onto. When it picked the wrong thing, you drag the crop box yourself.

What actually fixes it

Lock it when the shot allows. A talking head that stays put does not need tracking. A static crop, placed once, is more stable than any tracker and takes five seconds to set. Most podcast footage is this case. Reach for tracking when the subject genuinely moves, not by default.

Cut, don't pan. If the clip has two speakers, the vertical version usually wants two crops with a cut between them, not one crop travelling. This is what a human editor does with the same footage, and it is the single largest quality difference between a clip that was reframed and one that was edited.

Fix the moment, not the setting. The failures are local: three seconds around a cutaway, one swing at a re-entry. Adjusting a global sensitivity to fix three seconds degrades the other fifty-seven. Drag the crop box by hand on that clip, or switch its layout: fill, fit, or a facecam and gameplay split.

Watch it once at speed, muted. Sound covers a lot. Without it, the frame's behaviour is all there is to look at, and a swing that felt fine in context becomes obvious.

When to give up on vertical

Some shots do not survive the crop, and recognising them early saves more time than any correction.

A wide with four people at a table has no vertical inside it. A screen share where the text matters is unreadable at 9:16 no matter where the crop sits. A shot whose composition is the point (a landscape, a room, a scale gag) stops being that shot when two-thirds of it is gone.

For these, the answer is not a better tracker. It is a different frame: letterbox the original into a vertical canvas with a background, or accept a square, or pick a different moment from the same source. A clip that admits its footage was horizontal reads as a choice. A clip that crops that footage badly reads as carelessness, and the viewer does not know which tool made the decision.