STAR: Steering Action Representations with Proprioception for VLA Inference

STAR motivation, probing experiment, residual block, layer-wise object awareness, and benchmark results — **TL;DR:** VLAs must recognize the object to manipulate and the target of the action. If a task-relevant object is masked, the model's hidden states should change substantially. Yet, we find that this response weakens in late layers, suggesting that task-relevant visual evidence is not fully preserved near the action output. STAR mitigates this decline during inference, requires no training, and improves task performance.

Abstract

Vision-language-action models (VLAs) map language instructions and visual observations to robot actions, showing strong capabilities across robotic manipulation tasks. However, in a probing experiment, we find that action representations near the action output do not fully preserve task-relevant visual grounding. Motivated by this late-layer grounding gap, we introduce STAR, for STeering Action Representations, a training-free inference method for pre-trained VLAs.

STAR applies an exponential moving average over late-layer residual updates, allowing later action representations to inherit object-aware update directions from earlier layers without model updates or architectural changes. It further uses recent proprioceptive states to adapt the mixing strength, using conservative mixing when execution dynamics change rapidly. Experiments on robot manipulation benchmarks show that STAR improves late-layer object awareness and increases average task success rates by 4.0 percentage points.

Method

STAR leaves the first two-thirds of the Transformer unchanged. Across the final third, it maintains an exponential moving average of residual updates, so each late layer combines its current update with the update trend accumulated from preceding layers. This helps visual evidence formed earlier persist until action decoding. Recent proprioceptive states adjust the mixing strength at every control step.

Comparison of native VLA inference and STAR residual mixing — Left: native inference applies each residual update directly. Right: STAR mixes the update trend across late layers. The coefficient β_t is computed once per control step from recent proprioceptive history.

Training-free residual steering

Let Δh_t^k be the residual update produced by late layer k. STAR combines this current update with the accumulated update trend m_t^k−1. The resulting trend m_t^k replaces the original update passed to the next layer. A single coefficient β_t is used throughout the forward pass at control step t.

m_t^k = β_tm_t^k−1 + (1 − β_t)Δh_t^k

Keep early layers unchanged. They form language-conditioned visual evidence.
Mix late-layer updates. The EMA helps this evidence persist across depth.
Adapt during execution. Mixing is stronger during steady progress and more conservative during rapid transitions or fine adjustments.

Detailed Experimental Results

Native, DTP, and STAR use the same frozen backbone checkpoints. DTP is another training-free inference baseline for VLAs (Li et al., 2026). All entries report task success rate (%).

SimplerEnv

We use the public SimplerEnv evaluation integration available for each backbone. The GR00T-N1.5 integration covers all six Google Robot tasks. The OpenVLA integration covers five and does not include Pick Object, so its average is computed over those five tasks.

Backbone	Method	SimplerEnv Google Robot Tasks						Avg.
Backbone	Method	Open Drawer	Place in Drawer	Pick Coke Can	Move Near	Close Drawer	Pick Object	Avg.
GR00T-N1.5	Native	26.7	4.0	69.3	68.7	45.0	32.3	41.0
	DTP	26.7	4.0	59.3	68.7	41.7	39.3	40.0
	STAR	32.0	4.7	77.3	76.7	47.3	40.0	46.3
OpenVLA	Native	7.4	0.0	65.3	56.7	22.2	—	30.3
	DTP	11.1	0.0	69.3	69.0	22.2	—	34.3
	STAR	18.5	3.7	70.0	61.7	25.9	—	36.0

LIBERO-PRO

Results under language, object-attribute, spatial-configuration, task, and visual-context variations.

Backbone	Method	LIBERO-PRO Perturbations on LIBERO-10					Avg.
Backbone	Method	lan	object	swap	task	env	Avg.
GR00T-N1.5	Native	81.2	40.4	0.0	6.0	73.8	40.3
	DTP	82.0	42.2	0.0	1.8	75.6	40.3
	STAR	82.8	48.0	2.2	6.8	76.4	43.2
π_0.5	Native	92.8	66.2	9.2	6.8	65.8	48.2
	DTP	90.4	64.6	8.0	7.8	67.6	47.7
	STAR	92.8	68.6	13.0	7.6	70.2	50.4

Real Robot

The task requires picking up a pear and placing it in a bowl. Both native inference and STAR use the same fine-tuned π_0.5 checkpoint. On the real-robot system with a single NVIDIA RTX 3090, STAR introduces only 3.3% additional inference latency relative to native inference.

Method	Successes / Trials	Success Rate
Native	17 / 40	42.5%
STAR	19 / 40	47.5%

Analysis

We analyze π_0.5 on LIBERO-PRO with two complementary diagnostics. Object masking tests whether replacing task-object pixels with the image's mean RGB value changes the hidden states used for action generation. Input-gradient saliency (Simonyan et al., 2013) tests whether those pixels influence the final predicted action. We also inspect attention maps to determine whether the hidden-state effect appears as a visible change in attention allocation.

Masking-based object awareness

We run the model on an original image and on a paired image in which the task-relevant object's pixels are replaced with the mean RGB value of the full image. At each layer, object awareness is the normalized L2 change in the action hidden state. A larger value means that removing the object's visual evidence has a stronger effect on the representation used to generate actions.

To check that the score is not merely responding to generic image corruption, we use two controls: the same number of masked pixels scattered randomly across the image, and a similar-area rectangle sampled from the background.

Object, random-pixel, and background masking controls; late-layer object-awareness scores; attention maps; and object attention mass under native inference and STAR — STAR makes late action representations more sensitive to task-relevant object evidence, while the control masks elicit weaker responses. Native inference and STAR show similar attention patterns, suggesting that STAR's effect is more evident in action hidden states than in attention maps.

Action-gradient dependence on the task object

The masking experiment measures hidden-state sensitivity. We next test whether the final predicted action actually relies on the task object. We differentiate a scalar score derived from the predicted action chunk with respect to the input image, then measure the fraction of total input-gradient magnitude that falls inside the task-object region. A higher score means that the selected action depends more strongly on pixels belonging to the key object.

Normalized target-object input-gradient evidence across LIBERO-PRO splits — Across LIBERO-PRO, STAR places more input-gradient evidence inside the task-object region than native inference. The paired differences show that this trend holds across evaluation samples.