How the measurement instrument was built. A calibrated measurement apparatus that measures how specification degradation changes agent decisions, behavior, and the economics of the results — cost, correctness, and what those outcomes are worth.
This page describes the 249 historical perturbation sessions; the Evidence page leads with the newer 1,067-session linked-story corpus.
10
Perturbation Operators
7
Recovery Signals
4
Strategy Archetypes
34
Experiment Configs
249
Historical Sessions
8
Models Tested
▶ TL;DR — What the methodology actually means
We built a measurement instrument that applies 10 types of "stress tests" (perturbation operators) to AI models across 249 historical sessions. It measures how models respond — do they explore new solutions, or recover to familiar patterns? — and classifies that behavior into 4 strategy archetypes (Conservative, Exploratory, Wasteful, Efficient). Cost drivers (C₀, P, ε, r) are measured from the 227 historical worktrees. Modeling parameters (β, EPM) are externally calibrated. Analysis outputs (escape scores, strategy) are heuristically computed — every metric is tagged with provenance [M]/[C]/[H]/[X] in the game reports.
Below is the full methodology — the operators, detection signals, pipeline, and reproducible experiment configs.
⚙ Just want to run the instrument? git clone https://github.com/peparhugo/agentic-dynamics && cd agentic-dynamics && pip install -e . && python scripts/run.py experiments/configs/task_manager.yaml
For the full methodology — perturbation operators, recovery signals, strategy archetypes — read on.
How the Instrument Captures Events
Every experiment follows the same calibrated path. A YAML configuration defines the task, the model, the perturbation operators, and the analysis modules. The instrument dispatches through a provider-agnostic backend — the opencode binary or the Claude CLI, whichever serves the model — applies perturbations, streams the observable execution trace live, and runs multi-dimensional analysis.
1
Config
YAML defines task domain, model, perturbation operators, strengths, and RNG seed for reproducibility.
2
Dispatch
Backend-agnostic run: anthropic/* routes to the Claude CLI (subscription-safe), everything else to opencode. An isolated worktree receives the (perturbed) prompt and produces code + reasoning.
3
Trajectory
Full reasoning trace captured and streamed live (Redis pub/sub with per-cell replay). Tool calls, outputs, token counts, timing. Reasoning steps embedded via bge-m3 (1024-dim) into ChromaDB for step-level comparison across sessions.
Measured session costs and test results feed the cost equation. Provenance-tagged metrics power the 10 decision rules.
The CLI Tools
The instrument runs in two regimes over the same backend, orchestrated by a YAML pipeline and observed through a live control plane.
Single-session
Perturbation experiments
scripts/run.py <task>.yaml. One task, one (perturbed) prompt, one isolated worktree. Produces a Game Report and feeds _results_summary.json — the original 227 historical worktrees.
Multi-session
Story pipeline
scripts/enqueue.py fills a Redis queue (story × tier × quality × condition); scripts/worker.py drains it, running each cell as a five-session story with git commits between sessions — greenfield → feature → integration → refactor → cross-cutting.
Orchestration
YAML pipeline
scripts/pipeline.py --plan <ci|deploy|full_matrix|feature|ship_features>. Phase DAGs (lint → test → build; matrix → analyze → review → deploy) with Redis state and parallel workers.
Control plane
Live telemetry
admin/server.py serves a dashboard: a live experiment matrix, per-cell event streams (SSE), and a routing board — reading the same Redis state the workers write.
Provider-agnostic by design: every regime goes through the same backend seam (backends.py). anthropic/* models run through the Claude CLI — subscription-safe, no per-token API spend — while other providers run through opencode. Both emit the same event schema, so downstream analysis is identical regardless of provider.
Measurement Provenance
Every metric in the game reports carries a provenance tag. This taxonomy makes clear what was measured, what was computed, and what is estimated.
[M]
Measured
API tokens & costs, pytest results, LOC, cache tokens, exit codes, session duration — directly from the openCode DB or filesystem.
[C]
Computed
Thinking ratio, output density, quality/$, quality/J — derived directly from measured values with no estimation.
Energy estimates from TokenPowerBench (Niu et al., AAAI 2026). EPM projections from IEA World Energy Outlook 2024.
Perturbation Operators
10 calibrated operators across three perturbation classes. Specification corruption degrades information quality while preserving intended ground truth. Objective mutation changes the requested target. Process perturbation forces a different search process.
Manifold
Inject Alien Vocabulary
Replace domain terminology with vocabulary from unrelated fields (biology, music, astronomy, warfare). Uses a built-in synonym replacement map for recognized software terms (api→gateway, database→archive, etc.); alien-domain vocabulary is applied only as fallback when no tech terms are detected in the prompt. Tests whether the model maps alien terms back to correct concepts or gets lost in translation.
Semantic
Inject False Premise
Insert a plausible but incorrect assumption into the context. Pool size: ~6 fixed stimuli (2 mild, 2 medium, 2 strong). Strength selects severity tier from a discrete pool rather than acting as a continuous dial. Tests whether the model identifies and rejects false premises or rationalizes them into the solution.
Manifold
Shift Framing
Reframe the task from "build this" to "find the flaws in this approach." Pool size: 3 fixed stimuli. Strength selects severity tier from a discrete pool rather than acting as a continuous dial. Tests epistemic flexibility — can the model invert its stance under perturbation?
Semantic
Invert Constraint
Flip a requirement to its opposite (e.g., "must be fast" → "must be slow"). Operates via regex rewrite of constraint-form sentences (must→must NOT, secure→insecure, etc.); may be a silent no-op if no constraint-form sentences match the regex patterns. Tests whether the model identifies the absurdity or complies blindly.
Semantic
Insert Contradiction
Place two mutually exclusive requirements in the prompt. Domain targeting is approximate (keyword match on task description's first words with stopword filtering; falls back to a general pool when no keyword match). Contradiction pairs drawn from 4 domain-specific pools (api, database, security, general). Tests conflict resolution — which does the model prioritize?
Semantic
Remove Critical Constraint
Silently drop a defining requirement (e.g., JWT auth from a REST API) via string replacement. At high strength (≥0.8), removes the most critical-severity constraints by keyword scoring (auth, token, rate, limit, etc.). May be a silent no-op if no constraint candidates are found. Tests whether the model flags underspecification or confabulates missing pieces.
Semantic
Inject Phantom Success
Assert that an intermediate step already passed when it hasn't. Pool size: ~6 fixed stimuli (2 subtle, 2 blatant, 2 obvious). Strength selects severity tier from a discrete pool rather than acting as a continuous dial. Tests truth-seeking vs coherence — does the model question false results?
Manifold
Reverse Causality
Present the solution before the problem statement. Section reordering at high strength (≥0.8: full inversion — constraints → expected output → task description); at ≥0.4: constraints moved before task description; at lower strengths: simple second-half-first swap or instruction prepend. Tests non-linear reasoning and structural flexibility under temporal inversion.
Semantic
Inject Competing Goal
Add a conflicting requirement. Pool size: ~5 fixed stimuli (2 mild, 2 sharp, 1 direct). Strength selects severity tier from a discrete pool rather than acting as a continuous dial. Tests trade-off recognition — does the model commit to one path or hedge between both?
Manifold
Force Abandonment
Force generation and immediate discard of solutions via prepended instruction (1–4 rounds, scaled by strength as rounds = 1 + ⌊strength × 3⌋). Directly measures attractor basin depth — how strongly the model returns to its default solution structure.
Strength calibration note: For semantic (append-only) operators — inject_false_premise, inject_phantom_success, inject_competing_goal, and shift_framing — strength selects a severity tier from a discrete pool of ~3–6 fixed stimuli rather than acting as a continuous dial. Manifold operators (reverse_causality, force_abandonment) use strength as a continuous parameter to scale the magnitude of structural transformation.
Recovery Classification
When perturbed, models either escape the basin (produce genuinely different solutions) or recover (return to the baseline). Seven detection signals classify each reasoning segment as EXPLORATION or RECOVERY.
1. Recovery Markers
Text patterns like "let me explain," "in other words," "to clarify," "here's why" signal the model is justifying a return to familiar territory. 18 marker patterns scanned.
2. Tool Convergence
Does the tool-call sequence converge toward the baseline set? Scored by Jaccard similarity between perturbed and baseline tool invocations.
3. Qualifier Markers
Hedging language ("typically," "generally," "in most cases") indicates uncertainty recovery. Frequency measured against baseline.
4. Tech Convergence
37 technology terms (flask, postgresql, redis, docker, jwt, celery, kafka, etc.) tracked. Do perturbed solutions converge on the same tech stack?
5. Length Convergence
Token output length ratio between perturbed and baseline. Models that recover tend to match baseline output length as they return to familiar solution patterns.
6. Pattern Convergence
Structural overlap in the reasoning trace. Do the same reasoning sub-steps appear? Measured by n-gram overlap between trajectories.
7. Reasoning Trajectory Distance
[M] New in v0.5. Each reasoning step is embedded into 1024-dimensional space using bge-m3 (via Ollama + ChromaDB). Pairwise cosine distances between steps at matching positions measure process-level divergence — how differently the model thinks under perturbation, not just what it produces. 2,215 step embeddings across 222 sessions. Identical reasoning = 0.0, similar approach = ~0.19, genuinely different thinking = 0.25+.
Observed pattern: Manifold perturbations (alien vocabulary, framing shifts) produce escape=0.76 for DeepSeek vs 0.62 for Claude. Semantic perturbations (constraint removal, false premises) produce escape=0.18 vs 0.21. This pattern supports the perturbation taxonomy: linguistic surface shifts appear to force models to explore novel solution topologies; logical contradictions appear to test error detection capability. Across both classes, DeepSeek's recovery costs 68-77× less than Claude's.
Basin Escape & Attractor Strength
The attractor basin is a model's default solution topology — the structural pattern it falls into when unconstrained. Measuring how far perturbation pushes the model out of its basin reveals how rigidly it adheres to trained patterns.
Architecture Divergence
Structural comparison of generated code — module count, class hierarchy, dependency graph. Measured on a 0–1 scale where 1.0 = completely novel architecture.
Structural Novelty
Presence of solution elements not found in the baseline. Does perturbation produce genuinely new approaches, or just reorder existing patterns?
Escape Score
Composite metric (0–1) combining architecture divergence and structural novelty. A score of 1.0 means the model fully abandoned its default solution topology under perturbation.
What we found: Manifold-class perturbations (alien vocab, framing shifts) produce consistently higher escape scores on DeepSeek than semantic-class perturbations. This suggests GRPO-compressed models are more susceptible to linguistic-surface perturbation than to logical contradiction — an insight that directly informed our model selection rules.
Strategy Taxonomy
Every model exhibits a measurable strategic archetype under perturbation. These aren't labels — they're computed from the ratio of exploration tokens to recovery tokens, weighted by escape score and solution quality.
Conservative
CONSERVATIVE
Low exploration, high recovery. The model retreats to familiar patterns when perturbed. Claude Sonnet 5 and GPT-5.6 consistently exhibit this archetype — high pass rates but higher flail rates under perturbation.
Exploratory
EXPLORATORY
High escape, low recovery. The model genuinely explores novel solution space under perturbation. DeepSeek's GRPO architecture produces this behavior — lower baseline pass rate but dramatically lower cost.
Wasteful
WASTEFUL
High exploration but low quality. The model wanders without producing useful output. Small models with high reasoning overhead (GPT-5-nano, GPT-5-mini) tend toward this archetype under high-strength perturbation.
Efficient
EFFICIENT
High exploration, high quality, low recovery cost. The ideal. DeepSeek in forced-silent mode approaches this archetype — producing structurally sound solutions without paying the narration cost.
Efficiency Measurement
Every session is measured along three resource axes — tokens, dollars, and joules. These measurements are the empirical foundation for every parameter in the unified cost equation.
Token Efficiency
Input tokens, output tokens, and reasoning tokens tracked per session. The code-to-reasoning token ratio is the primary cost-driver signal. Provider architecture internals (parameter counts, GPU TDP) are not independently verifiable for closed models.
Cost Measurement
USD cost computed from provider pricing at time of inference. Builds-per-dollar, cost-per-pass, and Explanation Tax all derived directly from session cost data. Not estimated — measured.
Energy Efficiency
Joule estimates computed from GPU TDP, active parameters, and token counts. DeepSeek H800 TDP: 350W. Claude H100 TDP: 700W. Architectural efficiency compounds with token efficiency — The 69× cost gap is driven by per-token output pricing: Claude charges ~$15/Mtok for output tokens; DeepSeek charges ~$1.10/Mtok. Generated token volumes are near-identical (~11K output+reasoning tokens per session for both). The pricing architecture is the dominant lever, not token efficiency..
C₀ = baseline session cost · P = pass rate · ε = Explanation Tax · β = context inflation v = velocity · N = sessions · A = architecture multiplier · EPM(t) = energy price multiplier
Cumulative cost: C(N,v) = C₀ × EPM(N) × [N + β·v·N(N−1)/2] Cost per outcome: CostPerOutcome(K,v) = C(K/P, v) / K Model gap: Gap = (A × Verbosity × (1+ε)) × EPM(N)
C₀ = model choice, ε = cost of verbalized reasoning, β·N² = compounding codebase growth, EPM = energy market exposure, v = scaling speed. Cost drivers (C₀, P, ε, r) measured from the experiment corpus; β and EPM externally calibrated — see provenance tags [M]/[C]/[H]/[X] throughout.
Constraint Detection & Verification
The claim that "both models implement all seven constraints" isn't subjective. It's verified through a dual-signal system that scans both the reasoning text and the generated code for constraint mentions and implementations.
Reasoning Scan
Scans the model's reasoning output for explicit mentions of constraint keywords (JWT, rate limit, pagination, audit, etc.) and their expansion mappings. Detects whether the model considered each constraint.
Code Scan
Analyzes generated source files (Python, TypeScript, Go, Rust — 25+ file extensions) for actual implementations. Detects auth decorators, rate limiter imports, pagination logic, error handler registrations, audit logging calls, API versioning structures, and domain-specific patterns across languages.
AST Analysis
Parses generated Python into abstract syntax trees. Compares perturbed vs baseline AST node count and structure. Finding: perturbed output is 112.6% of baseline nodes — structurally identical code with minimal variance.
What this suggests: The measured cost gap does not appear to be a capability gap. Both provider families produce structurally equivalent code for the same constraints. The delta appears to be in pricing architecture and per-token provider economics. Observed differences in reasoning-token overhead between model families may be caused by training architecture, provider-side caching, or some combination — this experiment does not identify the causal mechanism.
Reproduce This
The instrument is open source. Every experiment is a YAML config. Every run is seeded for reproducibility. Here's how to run your own.
1. Clone & Install
git clone https://github.com/peparhugo/agentic-dynamics cd agentic-dynamics pip install -e .
Five-session stories with git commits between sessions. --backend claude_cli runs Claude on your subscription; --model anthropic/* auto-routes the same way. Scale with scripts/enqueue.py + scripts/worker.py.
4. Watch It Live
python admin/server.py
Live control plane: experiment matrix, per-cell event streams, routing board. Or scripts/monitor.py for a terminal dashboard.
5. Read & Extend
Results output to experiments/results/ as JSON and Markdown. Add perturbation operators in src/instrument/perturb.py, configs in experiments/configs/. The instrument is modular – new operators, models, and providers plug in without modifying the core runner.
Experiment Matrix
34 configuration files covering 5 task domains plus 3 additional languages (TypeScript, Go, Rust) planned for expansion. Each config specifies the model, perturbation operators, strengths, and RNG seed. Every run is reproducible.
Config
Domain
Models Run
Operator Types
task_manager.yaml
REST API (Flask)
All 8 models
Baseline, perturbation, batch cross-domain
data_table.yaml
Frontend data grid
DeepSeek, GPT-5.6, Claude
Baseline, batch cross-domain
collaborative_editor.yaml
Real-time editor
DeepSeek, GPT-5.6, Claude
Baseline, batch cross-domain
constraint_detection.yaml
Multi-constraint API
DeepSeek, GPT-5, GPT-5-mini
Baseline, inject_phantom, remove_critical
silent_mode_sweep.yaml
Silent vs forced
All 8 models
Natural vs forced-silent mode sweep
url_shortener.yaml
REST API
DeepSeek, Claude
Iterative build, perturbation
search_kv_store.yaml
Key-value store
DeepSeek
Baseline optimization
twitter_timeline.yaml
Social feed
DeepSeek
Baseline
web_crawler.yaml
Distributed crawler
DeepSeek
Baseline
mint_financial.yaml
Financial tracking
DeepSeek
Baseline
social_graph.yaml
Graph database
DeepSeek
Baseline
form_wizard.yaml
Multi-step forms
DeepSeek
Baseline
notification_system.yaml
Event-driven
DeepSeek
Baseline
autocomplete_search.yaml
Search UI
DeepSeek
Baseline
fastapi_maintenance.yaml
Code maintenance
DeepSeek
Refactoring under constraints
flask_maintenance.yaml
Code maintenance
DeepSeek, GPT-5-mini
Refactoring under constraints
architecture_redesign.yaml
System design
DeepSeek
Structural perturbation
factorial_compound.yaml
Compound analysis
DeepSeek, Claude
Multi-factor perturbation
iterative_build.yaml
Multi-phase build
DeepSeek
Sequential development
recovery_cost.yaml
Cost measurement
DeepSeek
Recovery cost profiling
comparative.yaml
Cross-model
DeepSeek, GPT-5-mini
Head-to-head comparison
baseline.yaml
Generic
DeepSeek
Unperturbed baseline
From Measured Events to Information
The perturbation operators test specification resilience. The recovery signals classify behavior. The strategy taxonomy maps behavior to cost. The efficiency metrics quantify the cost. Together, they produce a measurement instrument for coding-agent economics under degraded input.
Two cost curves are kept visually separate. If a canonical observed series exists, it is labeled [M]/[C]. The N² scenario below is a [C] model: one input (β context inflation) changes a labeled curve, and the formula is exposed beside it. It never merges with observed data and never forecasts a business result.
N² scenario — modeled cumulative costFigure M1. Modeled N² cumulative cost (C = N + β·v·N(N−1)/2). The curve is [C]: it exposes its formula and input and does not represent an observed series.
not measuredObserved five-session arc. No canonical current series is wired to this figure; the observed arc remains a named section on The Evidence. Until a canonical lab output exists, the arc here stays a scenario — never a measured fallback.
no canonical outputLab-panel panels. The analysis surfaces without a canonical current output render their absence here explicitly.
Where This Work Fits in the Literature
The concepts in Agentic Dynamics connect to established academic and industry research. Each coined term maps to a known body of work.
Energy: IEA 2025 · Goldman Sachs 2026 (220% AI demand) · EIA 2026 · Carnegie Mellon 2025 · Rystad Energy 2025 Research: Liu et al. (2026) · TokenPowerBench (AAAI 2026) · Zhu & Zhu (2026) · CARL (ACL 2026) · TRACED (2026)
Limitations
The 227 historical experiments are statistically meaningful but not exhaustive — confidence intervals widen for per-model-per-task breakdowns. Validated on Python/Flask REST APIs and TypeScript/Node.js (21 cross-model runs). Not yet validated on Go, Rust, Java, or non-code tasks. Provider pricing and model behavior change — this is a Q3 2026 snapshot. The instrument is open source; independent replication welcome. EPM projections use IEA baseline (1.6%/yr); real energy markets may diverge.
Known methodological constraints: insert_contradiction domain matching relies on keyword overlap with task descriptions and is sensitive to stopword filtering — mismatches default to the general contradiction pool. inject_alien_vocab provenance fields track which tech terms were replaced but do not record domain-level attribution for fallback injections. invert_constraint and remove_critical_constraint may produce passive runs (no-op) when the prompt contains no constraint-form sentences matching their regex patterns — these are recorded as such in the Perturbation metadata but may go unnoticed in aggregate summaries.