Patch and Control: Steering Behavior of Large Vision-Language Models via Latent Activations

We propose a novel approach to steer the behavior of Large Vision-Language Models (LVLMs) through activation patching, a mechanistic interpretability technique that modifies a model's internal representations. By extracting latent activations from image-captioning prompts conditioned with behavioral control statements and injecting them into a separate inference pass, we enable implicit conditioning without modifying the input or fine-tuning the model. Our experiments with LLaVA-1.5 demonstrate that patched activations yield semantically coherent and controllable outputs, as measured by BLEU, BERTScore, cosine similarity, and COMET. We find that early decoder layers offer the highest fidelity for behavior transfer.

• ›First extension of activation patching to Large Vision-Language Models (LVLMs), enabling multi-modal behavior steering.
• ›Implicit conditioning without input modification or fine-tuning, purely through latent space injection.
• ›Empirical finding that early decoder layer patching yields highest fidelity for behavior transfer (BLEU, BERTScore, COMET).
• ›Activation patching as an interpretable, robust alternative to prompt engineering for LVLM control.

Control tokens C = {c1, …, cNC} from a behavioral prompt (e.g., 'Act like a pirate') are processed through LLaVA-1.5. Activations a_l at extraction layer l_e are captured and patched into a clean inference pass at patching layer l_p. Image tokens I from the vision encoder are handled separately. The patched model generates output O conditioned implicitly on the control without seeing it in context. We study all decoder layer combinations and report fidelity metrics.

Evaluated on 100 Flickr30k images against 10 hand-written control statements, at full patching strength (α = 1.0). BLEU stays low by design — the goal is to follow the control, not to reproduce the gold caption — so the semantically grounded metrics carry the signal.

• ›Control transfers far better than image (BERTScore 0.87 vs 0.75) — image activations are load-bearing for caption content in a way conditioning is not.
• ›Patching both beats patching the image alone, because both activations then come from a jointly-aligned representation instead of forcing patched visual features to reconcile with an unpatched control context.
• ›Self-patching (extract and patch at the same layer) beats both forward and backward patching, and early decoder layers beat late ones — there is a 'conditioning sweet spot' early in the stack.
• ›Characteristic controls transfer most reliably (BERTScore 0.882) versus linguistic (0.859) and persona (0.858): behavioural traits are more robustly encoded than identity or style.
• ›Control prompt length has no measurable effect, making the method length-agnostic.

• ›In-domain only — extraction and patching currently use the same image.
• ›Compute limited the layer sweep to 5×5 depth combinations out of a 32-layer decoder, and evaluation to 100 of Flickr30k's 31,000 images.