The Gaussian Is Enough:
Flow-Matching Priors Do Not Help
When Fine-Tuning Large Behavior Models

What is the action prior? Left: a flow-matching policy generates an action by transporting a random sample a₀, drawn from a prior distribution p₀ over action chunks, to the target action. The action head is conditioned on the observation embedding e produced by the encoder and on the diffusion step t. By default p₀ is a standard Gaussian; it is the only component we vary. Right: samples from two priors we consider, executed as actions in simulation before flow-matching transport, on the held-out DumpVegetablesIntoContainer task from LBM Eval. The Gaussian 𝒵 produces incoherent, random motion. A_pre, the zero-shot actions of the pretrained policy, already makes partial task progress before fine-tuning on task-specific data.

Why a closer prior might help. Flow matching only requires paired samples from the prior and the target, so any distribution can serve as the prior without changing training or inference: the Gaussian is a convention, not a requirement. Recent work exploits this flexibility when training from scratch: BRIDGER (Chen et al., 2024) learns a CVAE action prior, Cocos (Dong et al., 2026) a condition-dependent Gaussian, action-to-action flow matching (Jia et al., 2026) starts from demonstration actions, and EfficientFlow (Chang et al., 2026) adds equivariance guarantees. Replacing the Gaussian with such a closer-to-target prior substantially improves success rates on LIBERO, MetaWorld, and Adroit: if the prior is already near the target, the policy only needs to traverse a short transport distance. Whether this benefit carries over to fine-tuning a pretrained LBM is the question we study.

Experimental Setup

We compare seven action priors: the standard Gaussian, three constructed directly from the pretrained LBM with no auxiliary training, and three baselines from the literature. Within each experiment, only the prior differs; task, data, architecture, optimizer, and training steps are held fixed.

Seven action priors (hover or tap a name):

Our study is guided by three research questions:

• Q1. Do non-Gaussian priors that are measurably closer to the target improve policy fine-tuning performance?
• Q2. Do informed priors have an advantage across data fractions, particularly in the low-data regime, where the added inductive bias may be most beneficial?
• Q3. Is the effect of prior choice consistent across model architectures, and across simulation and hardware?

Four experimental settings address these questions:

• (1) Controlled test (Q1): a prior constructed to be measurably close to the target, using LBM 1.0 on five LBM Eval tasks in simulation (14,000 rollouts).
• (2) Full-scale fine-tuning in simulation (Q1, Q2): LBM 1.0 on eight LBM Eval tasks unseen during pretraining, at five fine-tuning data fractions (56,000 rollouts).
• (3) Bimanual hardware (Q3): blind A/B testing on five bimanual tasks with two Franka FR3 arms (1250 rollouts).
• (4) Other LBM architectures (Q3): π_0.5 and GR00T N1.5 on 34 RoboCasa365 tasks (54,400 rollouts).

Results

Analyses

How do priors shape the training dynamics and model internals?

(1) Fine-tuned encoders are equally predictive of the target actions

LogME is a training-free measure of feature quality: the log marginal evidence of a Bayesian linear regression from the fine-tuned encoder's embeddings to the ground-truth actions. Scores are statistically inseparable across priors at every data fraction, so the prior leaves no detectable effect on the quality of the encoder's features.

(2) Encoder representations diverge, action predictions do not

We measure how far fine-tuning moves each model from the pretrained one (per-observation ℓ₂ distance). Action predictions change by a comparable amount across priors, but encoder embeddings diverge sharply: the Gaussian moves the encoder least, A_pre roughly 2 to 2.5× more. One plausible reason: the Gaussian was also the pretraining prior, so fine-tuning with 𝒵 shifts only the target, whereas A_pre shifts both prior and target, and the encoder compensates.

(3) Encoder training matters more than prior choice

If the encoder dominates, degrading it should hurt far more than changing the prior. We cut the encoder's learning rate to 1/10 of the action head's. In simulation (eight tasks, 100% data; 𝒵, E_pre+σ_E𝒵, Cocos), success drops from 56–59% to 41–43% for every prior, with no measurable prior effect within either learning rate. On hardware (five tasks, Gaussian prior), success drops from 37% to 28%. No change of prior in this study has an effect of comparable magnitude.

Discussion

Implications for practitioners. The standard Gaussian prior is sufficient for fine-tuning LBMs with a flow-matching action head; learned priors add engineering complexity with no measurable benefit. What matters is the quality of the pretrained representations. Without pretraining, learned priors may still be valuable.

Limitations. Our findings cover three LBMs and roughly 10 to 400 fine-tuning demonstrations per task. Much larger or smaller models, other pretraining data, or much larger fine-tuning sets might behave differently, and the encoder-dominance explanation remains a hypothesis at other scales.

Open questions.

• Where is the transition? At 5% data, a learned prior significantly outperformed the Gaussian. Mapping where the prior begins to matter as a function of dataset size, task complexity, and model capacity would tell practitioners when a learned prior is worth its cost.
• Why do different priors yield divergent encoders of equal quality? Loss-landscape geometry, plasticity, and linear mode connectivity are plausible directions.
• Does encoder divergence have downstream consequences? Whether these structurally different encoders differ in robustness to distribution shift, continual-learning stability, or catastrophic forgetting is open.

BibTeX

Preprint: arXiv:2609.27070

@article{xu2026gaussian,
  title={The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models},
  author={Xu, Chen and Shah, Rishi and Kress-Gazit, Hadas and Nishimura, Haruki and Itkina, Masha},
  journal={arXiv preprint arXiv:2609.27070},
  year={2026}
}