The 2,000-Token Apology

Last month I watched an enterprise agent hallucinate its way through a deployment checklist because someone kept "helpfully" adding more instructions to the system prompt. The final prompt was 2,143 tokens. The agent skimmed the first 300, ignored the rest, and still pushed to production. We didn't have a context problem. We had a prioritization problem.

Every time the bot broke something, the team stuffed another paragraph into the prompt. "Please confirm destructive actions." "Please summarize the plan." "Please re-state the goal." By the end, the agent was carrying around a phone book it never read.

If your incident report includes the sentence "We already told the model not to do that," you have prompt bloat, not an alignment breakthrough.

Prompt Bloat Is Process Debt

We shove everything into the prompt because it feels safe. The prompt never goes down. But bloated prompts create three forms of debt:

  • Latency debt: you waste time re-sending context the model already saw five seconds ago.
  • Cognitive debt: models have attention budgets; noise buries the one instruction that matters.
  • Maintenance debt: every change requires editing a sacred text nobody wants to touch.

Just-in-time (JIT) context fixes the problem by turning the prompt into an operating manual and pushing everything else into callable tools.

What JIT Context Actually Means

JIT context is the agent equivalent of modern supply chain management: keep a minimal inventory in the prompt and fetch detailed parts only when the workflow requires them.

1. Operating Manual

Identity, tone, approval gates, collaboration rules. Short. Stable. Always loaded.

2. Tool Crate

Structure helper, formatting guide, API schema, pricing tables. Fetched on demand.

3. Execution Timeline

The choreography that tells the agent when to call which tool. This is the "JIT" brain.

Component 1: The Prompt Is an SOP, Not a Wiki

The prompt should read like a Standard Operating Procedure. Mine now follows this structure:

  1. Greeting & positioning: "You are the Orchestrator…"
  2. Workflow map: Planning → Approval → Execution → Presenting.
  3. Approval gates: the exact block the UI listens for.
  4. Communication rules: progress emojis, error handling, tone.
  5. Tool catalog (high level): what is available, not how to use it.

If the prompt includes JSON schemas, SQL migrations, or 20 bullet points of mitigation instructions, you don't have an SOP—you have a knowledge base disguised as a greeting.

Component 2: The Tool Crate Carries the Weight

Everything that changes frequently or requires detail lives in a tool. Examples from our stack:

  • Structure Helper: returns outlines, word targets, image placements, and size guidance.
  • Formatting Standards: explains spacing, image styling, and pre-publish checklist.
  • Pricing Sheets / Schema Docs: tiny dedicated tools that return the authoritative version.

Tools are versioned, testable, and cached. Updating an image strategy no longer means touching a 700-line prompt—just ship a new tool response.

Component 3: The Execution Timeline Is the Glue

JIT context fails when the agent doesn't know when to fetch the extra context. We solve that with a locked sequence:

  1. Planning: call article_structure_helper_tool to pick a format.
  2. Drafting: write section-by-section but reference the structure payload instead of guessing.
  3. Execution: after approval, call generate_images_for_post_tool, replace placeholders, then fetch formatting standards.
  4. Publishing: only after applying the formatting checklist do we call post_to_wordpress_tool.

The agent always knows the next tool to call, so every detail arrives just when it's needed—never earlier, never forgotten.

The Stateless Problem: Why Breadcrumbs Matter

Here's the uncomfortable truth about LLMs: they're stateless. Every time the model responds, it forgets where it was. There's no persistent "current step = 3" variable in the model's head. Each turn is a fresh start.

So how does a stateless model execute a 4-step workflow without getting lost?

Answer: Breadcrumbs.

Breadcrumbs are contextual markers you embed in each turn that tell the model "You are here. Next is there." The model reads the breadcrumbs, figures out its position in the workflow, and knows what to do next.

What Good Breadcrumbs Look Like

Our execution flow uses explicit state markers:

User Approves Plan

Agent Turn 1:

🔄 STEP 1/4: Generating images...
[calls generate_images_for_post_tool]

Agent Turn 2 (after tool response):

✅ STEP 1/4 Complete: Images generated

🔄 STEP 2/4: Replacing placeholders with real URLs...
[performs replacement]

Agent Turn 3:

✅ STEP 2/4 Complete: URLs replaced

🔄 STEP 3/4: Fetching formatting standards...
[calls get_html_formatting_standards_tool]

Agent Turn 4:

✅ STEP 3/4 Complete: Formatting applied

🔄 STEP 4/4: Publishing to WordPress...
[calls post_to_wordpress_tool]

Notice the pattern:

  • Step counter: "STEP 2/4" tells the model where it is
  • Status marker: "✅ Complete" confirms the last step finished
  • Next action: "🔄 STEP 3/4: Fetching..." tells the model what's next

Each message creates a trail the stateless model can follow. When the model wakes up for the next turn, it sees "✅ STEP 2/4 Complete" and knows it should start STEP 3.

What Bad Breadcrumbs Look Like

Anti-Pattern: Vague Progress

✅ Images created!

Now posting...

No step counter. No clear state. The model has to guess whether it replaced placeholders or fetched formatting standards.

Anti-Pattern: No State Markers

[calls tool]
[calls another tool]
[calls another tool]
Done!

Silent execution. No breadcrumbs. If something fails mid-workflow, the model can't recover because it has no idea where it stopped.

Separation of Concerns: Prompt vs. Tools vs. Breadcrumbs

The JIT model works because we split responsibilities across three layers:

Layer Responsibility Example
Prompt Workflow map, approval gates, tone, tool catalog "You are the Blog Poster Agent. Follow this 4-step execution flow..."
Tools Domain knowledge, formatting rules, schemas, data retrieval get_html_formatting_standards_tool returns spacing rules on demand
Breadcrumbs State tracking, progress markers, next-step signals "✅ STEP 2/4 Complete: URLs replaced. 🔄 STEP 3/4: Fetching formatting..."

The prompt is the constitution—it defines what the agent is and how it behaves. It never changes mid-execution.

Tools are the reference library—called only when the workflow needs specific knowledge. They're versioned, cacheable, and independently updatable.

Breadcrumbs are the state machine—they guide the stateless model through multi-turn execution by creating a trail of "where I was, where I am, where I'm going."

Without breadcrumbs, your agent is hiking through a forest with no map. It might reach the destination, or it might wander in circles. With breadcrumbs, it follows the trail—even if it's stateless and forgets the last 10 steps.

Case Study: The Blog Poster Agent

We recently refactored our Blog Poster agent using this JIT model. Here's what changed:

Problem Old Behavior JIT Fix
Image strategy drift Prompt described "try 2 images" with no specifics. Structure tool now returns exact placements, types, and sizes JIT.
Formatting inconsistencies Prompt contained a 90-line spacing guide the model ignored. Formatting tool called right before posting; model only sees rules when needed.
Mid-execution confusion Agent sometimes skipped steps or called tools out of order. Explicit breadcrumbs ("STEP 2/4 Complete") guide stateless model through workflow.
Approval slippage Agent sometimes skipped confirmation because the rule was buried mid-prompt. Approval gate lives in SOP as non-negotiable checkpoint; UI enforces it.

The result: smaller prompts, faster execution, reliable multi-turn workflows, and updates that touch tools—not core behavior.

Anti-Patterns to Avoid

  • The Infinite Prompt: "Just add one more reminder." Delete three lines for every one you add.
  • The Tool Phone Book: dumping every schema into a single tool response. Split them.
  • Tool Guessing: forcing the agent to infer when to call a tool. Spell it out in the execution timeline.
  • Silent Execution: calling tools without announcing what you're doing. No breadcrumbs = no state recovery.
  • Vague Progress: "Working on it..." instead of "STEP 2/4: Replacing URLs...". The model needs explicit markers.
  • Stateful Assumptions: assuming the model "remembers" what happened last turn. It doesn't. Leave breadcrumbs.
  • Silent Updates: updating prompts without changelog or versioning. Treat the SOP like code.

Implementation Checklist

Prompt Layer:

  • Audit every sentence in the prompt. If it changes monthly, move it to a tool.
  • Define the workflow as a numbered sequence with clear start/end points.
  • Include approval gates as explicit blocks the UI can parse.
  • Version your SOP. "Prompt v17" is a real asset, not tribal knowledge.

Tool Layer:

  • Give each tool a single responsibility and a schema the agent can trust.
  • Return structured data (JSON), not prose. The agent should parse, not interpret.
  • Version tools independently from the prompt for clean rollbacks.
  • Log tool calls. If the agent never calls a tool, remove it from the flow.

Breadcrumb Layer:

  • Emit "STEP X/Y" counters in every progress message so the model knows its position.
  • Mark completed steps with status symbols (✅) before announcing the next step.
  • Always announce the next action before calling a tool ("🔄 Fetching formatting standards...").
  • Never execute tools silently—breadcrumbs are the state machine for stateless models.
  • Test recovery: pause execution mid-workflow and verify the agent can resume from breadcrumbs.

The Payoff

JIT context is not just about shrinking prompts. It's about building a system where:

  • The prompt stays human-readable and defines behavior, not knowledge.
  • Tools stay modular and deliver context exactly when needed, not 500 tokens too early.
  • Breadcrumbs guide stateless models through multi-turn workflows without guesswork.
  • Changes touch tools or execution logic, not the core identity of the agent.

When you inevitably change your mind about image guidelines or formatting rules, you update a tool—not the soul of the agent. When execution gets confused, you check the breadcrumbs—not rewrite the entire workflow. When a new engineer joins, they read the SOP—not a 2,000-line manifesto.

That's how you keep agents sharp, fast, maintainable, and reliable across dozens of execution turns. Minimal prompt, maximal leverage, explicit state tracking.

And when someone asks "How does this agent work?", you can point to three things: the prompt (behavior), the tools (knowledge), and the breadcrumbs (state). Not a monolithic blob that does everything and explains nothing.