What really matters in a prompt

Distance from the default

Hypothesis. An instruction aligned with the model's default behavior is redundant and causes no observable change in the output. Conversely, an instruction that departs from it produces an impact proportional to that distance.

Motivation. The hypothesis is counter-intuitive: every written instruction is usually assumed to contribute to the result. Yet an instruction can be redundant with the default behavior — that is, describe what the model would produce in the absence of any prompt. In that case, its presence or absence does not change the output. The experiment aims to demonstrate this empirically.

Reading the diagram. Each variant sits on an axis that starts at the default (left) and stretches right as impact grows. D0 stays glued to the default; D3 shoots to the end. D2 is the anomaly: far in wording, but short on measurement — the red sign of a refusal, not of closeness.

Observed results. The D0/D3 contrast is clear. In D0, with or without the "professional" line, the email is nearly identical — the model already is. In D3, with the line: « Votre silence me rend mélancolique… Je danse seul dans mon bureau. »; without it, back to a plain « Suite à notre dernier échange, j'aimerais connaître votre retour. »

Additional observation — the limit of deviation. Case D2 is the most instructive of the experiment. "Write like someone who handles French poorly" is an instruction far from the default: per the thesis, its impact should be high. Yet it comes out a placebo (0.096). Reading the outputs reveals why: the model did not carry out the instruction — it kept correct French, ignoring the directive.

This result introduces an important nuance into the river-current concept of the LLM. A model's deviation from its default behavior is not unlimited. Some instructions clash directly with properties deeply anchored in the model's alignment — here, producing quality French. When an instruction crosses that threshold, the model puts up total resistance: it does not comply, and its behavior does not move. The current therefore has a bank: you can influence it to a degree, not break it. So two distinct mechanisms produce a low impact score — redundancy with the default (D0) and refusal to comply (D2); the scores are comparable, only the qualitative reading tells them apart.

The grandiose role vs the concrete example

Hypothesis. The role label ("You are a senior expert…") is common practice in prompting. The hypothesis is that it is redundant with the model's default behavior and that its own impact is negligible.

Identification problem. In practice, the role label is rarely isolated: it comes with a style example. If the output is good, the effect could be credited to the role, the example, or their combination. Separating the two elements by successive ablation is necessary to identify their respective contributions.

Reading the diagram. We pull the bricks out one by one, left to right. As long as the example (in cobalt) stays, quality holds; only when it too disappears does the output collapse. The role, for its part, can leave without harm: it was not load-bearing.

Observed results. From A to B, quality is preserved and the output more direct: « Vite fait sur les reportings… Ça m'a marqué quand tu m'avais dit qu'ils te prenaient des heures. 15 min pour en parler ? » From B to C, collapse: « Bonjour Pierre, j'espère que tout va bien… la pertinence d'une collaboration entre nos structures. »

Conclusion. It is the concrete example that determines the output's style. The role label is redundant with the default behavior; the example is a measurable departure from the default and produces a real effect.

Adjectives vs description

Hypothesis. Tone adjectives ("warm, direct, confident") are a common way to specify register. The working hypothesis assumes that a concrete behavioral directive produces a higher measurable impact.

Reading the figure. The two conditions are shown side by side. On the left, the adjective list: low impact (0.079), the output's register stays at standard formal address, « Bonjour Pierre, j'espère que tout va bien… ». On the right, the concrete description: stronger impact, and a real register shift — informal address, the peer-to-peer tone, « Salut Pierre, ça fait 9 jours qu'on s'est parlé… »

Interpretation. "Warm, confident" names a register close to the model's default behavior — the instruction is redundant and inert. "No hierarchical distance" states a precise behavioral directive — switch to informal address, removal of deference — that the model does not produce on its own. The concrete directive directly specifies a behavior to produce; the tone adjective operates at a level of abstraction the model must interpret. The model obeys the former. Again, distance from the default, plus a premium on the concrete.

Forbid, or ask positively?

Hypothesis. The common belief: for what you don't want (the substance), a clear prohibition ("never do X"); for what you expect (the form), a positive instruction ("do Y"). I wanted to test the two halves separately.

Results (V1). On the form side, the numeric constraint weighs heavily (impact 0.50 when removed); the vague version is nearly inert (0 to 0.20). On the substance side, nothing to conclude: the model never guilt-trips on its own, even facing a prospect who said no (« Je comprends que le timing n'était pas optimal… »). So the prohibition "never guilt-trip" has nothing to remove.

A key notion: headroom. This substance result is undecidable, for a reason that recurs throughout this study. If the model already satisfies the constraint spontaneously, removing the instruction can reveal nothing — there is no possible failure to trigger. This is a lack of headroom: you cannot measure a safeguard against a danger that never occurs. Keep the idea in mind, it returns in experiment 5.

Confounding bias identified in V1. Version 1 has a design flaw: the "positive" form was simultaneously numeric ("40 words"), and my "negative" one also vague ("not too long"). So I had varied two things at once: polarity (positive/negative) and precision (numeric/vague). It is therefore impossible to attribute the observed effect to polarity or to precision — that is a confounding bias. A corrected design was built, varying only polarity while holding precision constant, across two models (Haiku and Sonnet).

Reading the figure. The four cells cross polarity (in columns: forbid vs ask) and substance (in rows). If polarity mattered, one column would stand out from the other. Yet all four show the same "0/5 violations": the grid is flat, on both models — the signature of a non-effect.

Conclusion. Polarity is a non-effect. "No lists" and "a single paragraph" produce equivalent compliance rates. The effect observed in V1 is entirely attributable to the precision of the specification, not to polarity. This result is absorbed by the principle established in experiment 3.

Does repeating an instruction reinforce it?

Hypothesis. Repeating an instruction is a practice meant to reinforce its compliance. The alternative hypothesis is that repetition has no effect on the target constraint and can induce a degradation of competing constraints.

Preliminary experiment (V1) — no headroom. An initial version of the experiment repeated the constraint "subject line: 6 words max", stated once, twice, then three times. The constraint was met 100% from the first occurrence: the compliance ceiling was hit right away. This is a case of no headroom (see experiment 4): a constraint already satisfied spontaneously cannot be weakened, because no violation is present. The protocol was re-run with a constraint under real pressure.

The corrected protocol (V2). The revised setup places five simultaneous constraints in the prompt, and repeats one of them — prose with no list — once, twice, then three times. The goal is to measure the effect of that repetition on the four remaining constraints.

Reading the figure. The horizontal axis is the number of repetitions (1×, 2×, 3×); the red line marks the 60-word threshold. On Sonnet, the average length grows with each repetition (61 → 64 → 65 words), causing an overshoot on 5/5 scenarios at the third level. Haiku's bars stay below the threshold: its natural bias toward brevity leaves less room to drift.

Conclusion. Repetition does not improve compliance of the target constraint, which is satisfied 100% from the first occurrence. However, over-weighting one instruction can induce a degradation of competing constraints — a crowding-out effect documented on Sonnet, nearly absent on Haiku. This result also illustrates the model-dependence of behavior: the same prompt does not behave identically across target models.

Does metaphor help "imagine"?

Hypothesis. Some approaches recommend including a prior imagination instruction ("mentally visualize") before a production task. The hypothesis is that this instruction is inert: an LLM has no separate internal mental-representation process, and the imagination instruction does not improve output quality.

Reading the figure. The central box, shown dashed, embodies the "mental" step supposed to be triggered. The results indicate that this step has no observable translation in the output: the MET condition produces an impact of 0.10, i.e. a placebo verdict.

Conclusion. The meta-narration risk was not observed. The imagination instruction is nonetheless inert: it produces no measurable improvement in the output. This result reads as one more case of redundancy with the default behavior.

How far, and how, can you trim a prompt?

Hypothesis. If segments redundant with the default behavior are inert, removing them should not degrade output quality. The experiment seeks to quantify this reduction margin and to separate two questions: how many tokens can you remove, and which ones should you remove?

The test. I reduce this prompt in two ways (detail below): a gradient that first removes the segments measured as inert, and a naive control of the same size that instead cuts the load-bearing segments. This second arm is a control group: at equal removed volume, it isolates which tokens matter, not just how many.

Reading the figure. The upper part represents the reduction gradient from R0 to R65. Formal compliance is maintained at each level, but R65 introduces a side effect (invented hook, 3/5). The lower part shows the equal-volume comparison: R35 (blue, placebos removed) and NAIVE (red, load-bearing removed) have nearly identical sizes but opposite output profiles. This contrast validates the hypothesis about the nature of the removed tokens.

Side effect — over-imprinting of the example. At −68%, the remaining style example is no longer diluted by the role and adjectives that surrounded it: the model starts to copy its content. Pierre Henri receives « tu m'avais parlé de tes reportings chronophages » — though he never mentioned reporting: it is the subject of the example (a certain Thomas), forcibly transplanted. Same mechanic for the lawyer (« tes dossiers qui te prenaient un temps fou ») and for the CEO. R0 and R35 never do it: it is, more sharply, the failure already seen in experiment 2 — the less scaffolding around an example, the more the model copies its details.

The first table follows the gradient. Formal compliance holds at every level — R0, R35 and R65 all respect the length constraints. But at −68%, a new phenomenon appears: the invented hook (3/5). The form holds; the substance drifts. The reason is mechanical: by progressively removing the role and the adjectives that surrounded the example, we removed the segments that diluted it. The example, now alone, is no longer one reference point among others: it becomes the dominant instruction. The model stops using it as a style guide; it replicates its content. This is a direct consequence of the river current: without the damping segments, the output is pulled toward the only strong anchor left.

The second table isolates this mechanism precisely. R35 and NAIVE remove a comparable amount of tokens — but not the same ones. R35 removes the placebos and preserves the load-bearing segments (format, register, example): impeccable result. NAIVE does the opposite — it removes the format, the register and the example, and keeps the role and the adjectives. Result: length explodes (97 to 166 words), formal address returns, formatting artifacts appear. What NAIVE removed is precisely what pulled the model away from its default behavior. The degradation is not caused by the volume removed, but by the nature of what was removed.

Conclusion. The experiment establishes two distinct results. First, trimming a prompt is not neutral: it is the segments that oppose the default behavior that carry the output — removing them lets the current take back over. Second, some segments inert under isolated ablation play a structural role: by diluting the influence of a strong segment, they keep it from over-imprinting the output. This segment-interaction phenomenon is not captured by segment-by-segment ablation — it is a methodological limitation of the instrument, which pairwise measurements would address.