Mastering LLM Context & State Management in Autonomous Agents

Introduction

Building production-grade autonomous AI agents is no longer just about crafting a clever prompt. As agents take on multi-step tasks — debugging codebases, orchestrating workflows, conducting research — the real engineering challenge becomes state: what the agent remembers, what it forgets, and how it persists knowledge across sessions, restarts, and multi-agent handoffs. A 200K-token context window is not infinite memory; it is a budget that must be actively managed like RAM.

This technical analysis breaks down the three load-bearing pillars of agent reliability:

1. Prompt engineering — how system prompts encode identity, instructions, and constraints.
2. Context window management — sliding windows, rolling summaries, and eviction strategies that keep agents coherent over long horizons.
3. State persistence — file-based memory (CLAUDE.md, AGENTS.md, scratchpads), structured checkpoints, and multi-agent state sharing.

We ground each section in runnable Python and reference real patterns used by production agent frameworks. This guide targets tech leads, AI engineers, and startup founders building agent systems that must survive more than a single chat turn.

---

1. Prompt Engineering: Identity, Instructions, and Constraints

The system prompt is an agent's constitution. Unlike a chat message, it is the persistent instruction set that shapes every downstream generation. Production agents fail when system prompts are vague, mutable across sessions, or fail to encode hard constraints (safety, output format, tool-use boundaries).

Structural Anatomy of a Production System Prompt

A robust system prompt is not a single paragraph — it is a layered document with distinct sections:

Here is a Python implementation that builds a system prompt from modular sections and injects live runtime state:

from dataclasses import dataclass, field
from typing import Callable

@dataclass
class PromptSection:
    name: str
    builder: Callable[[dict], str]
    priority: int = 0  # lower = earlier in prompt

@dataclass
class SystemPromptAssembler:
    sections: list[PromptSection] = field(default_factory=list)

    def add(self, section: PromptSection):
        self.sections.append(section)
        return self

    def render(self, ctx: dict) -> str:
        ordered = sorted(self.sections, key=lambda s: s.priority)
        blocks = [f"## {s.name}\n{s.builder(ctx)}" for s in ordered]
        return "\n\n---\n\n".join(blocks)

# --- Section builders ---
def identity_builder(ctx):
    return f"You are {ctx['agent_name']}, a {ctx['role']}."

def tools_builder(ctx):
    tool_list = "\n".join(f"- {t['name']}: {t['desc']}" for t in ctx['tools'])
    return f"You have access to the following tools:\n{tool_list}"

def constraints_builder(ctx):
    rules = "\n".join(f"{i}. {r}" for i, r in enumerate(ctx['constraints'], 1))
    return f"Hard constraints (never violate):\n{rules}"

def state_handoff_builder(ctx):
    return (
        f"At session start, read the memory file at `{ctx['memory_path']}` "
        f"to restore prior context. Append a summary of each completed "
        f"task to that file before terminating."
    )

# --- Assemble ---
assembler = SystemPromptAssembler()
assembler.add(PromptSection("Identity", identity_builder, priority=0))
assembler.add(PromptSection("Tools", tools_builder, priority=1))
assembler.add(PromptSection("Constraints", constraints_builder, priority=2))
assembler.add(PromptSection("State Handoff", state_handoff_builder, priority=3))

ctx = {
    "agent_name": "Atlas",
    "role": "senior backend engineering agent",
    "tools": [
        {"name": "run_bash", "desc": "execute a shell command"},
        {"name": "read_file", "desc": "read a file from disk"},
    ],
    "constraints": [
        "Never commit changes without explicit user approval.",
        "Always run the test suite before declaring a task complete.",
    ],
    "memory_path": ".agent/AGENTS.md",
}

system_prompt = assembler.render(ctx)

The key insight: by making the prompt composable, you can version-control it, unit-test each section, and inject live runtime state (current working directory, available tools, loaded memory) without rewriting the prompt string.

---

2. Context Window Management: The Memory Budget

Every token inside the context window has a cost — inference latency, monetary expense, and attention dilution. Research consistently shows that LLMs suffer from "lost in the middle" degradation: information in the center of a long context is recalled less reliably than at the edges. The context window is therefore a budget to be managed, not a bucket to be filled.

2.1 Sliding Window Memory

The simplest strategy: keep the most recent N messages and drop older ones. This is the default for most chat systems, but it is lossy — it forgets early instructions and facts with no summarization.

from collections import deque

class SlidingWindowMemory:
    def __init__(self, max_messages: int = 20, system_prompt: str = ""):
        self.system_prompt = system_prompt
        self.window = deque(maxlen=max_messages)

    def add(self, role: str, content: str):
        self.window.append({"role": role, "content": content})

    def render(self) -> list[dict]:
        msgs = [{"role": "system", "content": self.system_prompt}]
        msgs.extend(self.window)
        return msgs

    def token_estimate(self) -> int:
        # rough: 1 token ≈ 4 chars
        chars = len(self.system_prompt)
        for m in self.window:
            chars += len(m["content"])
        return chars // 4

When to use: short-lived interactions where early context is not critical. When not to use: any agent that runs more than ~15 turns or must recall facts from the beginning of a session.

2.2 Rolling Context Summaries

To preserve long-term information without keeping every message, summarize evicted messages and inject the summary as a synthetic system message. This is the backbone of virtually all production long-horizon agents.

import json

class RollingSummaryMemory:
    def __init__(self, llm_summarize, threshold_tokens=3000,
                 keep_recent=6, system_prompt=""):
        self.llm_summarize = llm_summarize  # callable: str -> str
        self.threshold = threshold_tokens
        self.keep_recent = keep_recent
        self.system_prompt = system_prompt
        self.summary = ""
        self.messages: list[dict] = []

    def _est_tokens(self, text: str) -> int:
        return len(text) // 4

    def add(self, role: str, content: str):
        self.messages.append({"role": role, "content": content})
        if self._total_tokens() > self.threshold:
            self._compress()

    def _total_tokens(self) -> int:
        return self._est_tokens(self.summary) + sum(
            self._est_tokens(m["content"]) for m in self.messages
        )

    def _compress(self):
        to_summarize = self.messages[:-self.keep_recent]
        retained = self.messages[-self.keep_recent:]
        transcript = "\n".join(f"{m['role']}: {m['content']}" for m in to_summarize)
        prior = f"Prior summary:\n{self.summary}\n\n" if self.summary else ""
        new_summary = self.llm_summarize(
            prior + "Summarize the following, preserving key decisions, "
                    "facts, and unresolved questions:\n" + transcript
        )
        self.summary = new_summary
        self.messages = retained

    def render(self) -> list[dict]:
        msgs = [{"role": "system", "content": self.system_prompt}]
        if self.summary:
            msgs.append({"role": "system", "content": f"[Conversation Summary]\n{self.summary}"})
        msgs.extend(self.messages)
        return msgs

The summary is a lossy compression of history. The keep_recent window preserves the exact, high-fidelity recent turns for in-context reasoning. The compression trigger (token threshold) is the critical tunable — too low and you summarize too often (losing detail); too high and you risk exceeding the model window.

2.3 Eviction Strategies Compared

---

3. File-Based Persistence: CLAUDE.md and Scratchpads

Context windows are volatile — they vanish when the session ends. For agents that must survive restarts, you need persistent memory on disk. The most battle-tested pattern is a markdown "memory file" that the agent reads at startup and writes to during/after work.

3.1 The CLAUDE.md / AGENTS.md Pattern

Popularized by Claude Code and widely adopted, this is a markdown file at the repository root that encodes project context, conventions, and prior decisions. The agent treats it as a long-term memory store.

from pathlib import Path
import datetime

class AgentMemoryFile:
    def __init__(self, path: str = "AGENTS.md"):
        self.path = Path(path)

    def load(self) -> str:
        if self.path.exists():
            return self.path.read_text(encoding="utf-8")
        return ""

    def append(self, section: str, body: str):
        existing = self.load()
        stamp = datetime.datetime.now(datetime.timezone.utc).isoformat()
        entry = f"\n\n## [{stamp}] {section}\n{body}\n"
        self.path.parent.mkdir(parents=True, exist_ok=True)
        self.path.write_text(existing + entry, encoding="utf-8")

    def rewrite(self, content: str):
        """Full overwrite — use sparingly, loses history."""
        self.path.write_text(content, encoding="utf-8")

A well-structured AGENTS.md looks like:

# Agent Memory

## Project Context
- Repo: payment-service (Go 1.22, PostgreSQL)
- Test command: `go test ./... -race`
- CI: GitHub Actions on push

## Conventions
- Never commit directly to main; use feature branches.
- All public functions need doc comments.

## Decisions Log
- [2026-08-09] Chose pgx over database/sql for prepared-statement caching.
- [2026-08-10] Migrated idempotency keys to a dedicated table.

The agent reads this at startup and appends new decisions as it works. Over time, this file becomes the institutional memory of the project.

3.2 The Scratchpad Pattern

A scratchpad is a working memory — ephemeral notes the agent writes mid-task to offload intermediate reasoning from the context window to disk. Unlike the memory file, scratchpads are task-scoped and can be discarded.

import uuid, tempfile, os

class Scratchpad:
    def __init__(self, task_id: str | None = None, base_dir: str = ".agent/scratch"):
        self.task_id = task_id or uuid.uuid4().hex[:8]
        self.base = Path(base_dir) / self.task_id
        self.base.mkdir(parents=True, exist_ok=True)

    def write(self, name: str, content: str):
        (self.base / f"{name}.md").write_text(content, encoding="utf-8")

    def read(self, name: str) -> str:
        p = self.base / f"{name}.md"
        return p.read_text() if p.exists() else ""

    def list_notes(self) -> list[str]:
        return [f.stem for f in self.base.glob("*.md")]

    def cleanup(self):
        import shutil
        shutil.rmtree(self.base, ignore_errors=True)

Use cases: storing intermediate SQL results, plan drafts, tool outputs too large for the window, or a running list of hypotheses. The agent writes "plan_v3.md", reads it back next turn, and avoids re-deriving.

---

4. Multi-Agent State Sharing

When multiple agents collaborate — a planner, a coder, a reviewer — they need a shared state store. Passing full context between agents is expensive and error-prone. The production pattern is a shared, structured state object (often a JSON file or lightweight KV store) that each agent reads and writes atomically.

import json, fcntl
from pathlib import Path

class SharedState:
    """File-locked shared state for multi-agent coordination."""
    def __init__(self, path: str = ".agent/shared_state.json"):
        self.path = Path(path)

    def _default(self):
        return {"tasks": {}, "log": [], "artifacts": {}}

    def read(self) -> dict:
        if not self.path.exists():
            return self._default()
        with open(self.path, "r", encoding="utf-8") as f:
            try:
                return json.load(f)
            except json.JSONDecodeError:
                return self._default()

    def update(self, fn):
        """Atomically read-modify-write under a file lock."""
        self.path.parent.mkdir(parents=True, exist_ok=True)
        with open(self.path, "a+", encoding="utf-8") as f:
            fcntl.flock(f, fcntl.LOCK_EX)
            f.seek(0)
            try:
                state = json.load(f)
            except (json.JSONDecodeError, ValueError):
                state = self._default()
            state = fn(state)
            f.seek(0)
            f.truncate()
            json.dump(state, f, indent=2, ensure_ascii=False)
            fcntl.flock(f, fcntl.LOCK_UN)
        return state

    def claim_task(self, agent_id: str, task_id: str) -> bool:
        def _claim(state):
            t = state["tasks"].get(task_id)
            if t and t.get("owner"):
                return state
            state["tasks"][task_id] = {"owner": agent_id, "status": "in_progress"}
            state["log"].append({"agent": agent_id, "action": "claim", "task": task_id})
            return state
        before = self.read()
        after = self.update(_claim)
        return after["tasks"][task_id]["owner"] == agent_id

The file lock (fcntl.flock) ensures that two agents racing to claim the same task do not corrupt state or double-claim. This is the lightweight equivalent of a database transaction, suitable for co-located agents.

---

5. Putting It Together: A Resilient Agent Loop

Combining all four layers — structured prompts, rolling summaries, file memory, and shared state — yields an agent that survives restarts, scales across long task horizons, and coordinates with peers:

def agent_loop(task: str, agent_id: str):
    memory_file = AgentMemoryFile(".agent/AGENTS.md")
    shared = SharedState()
    scratch = Scratchpad()

    # 1. Load persistent memory → inject into system prompt
    project_context = memory_file.load()
    system_prompt = assembler.render({
        **ctx, "memory_path": ".agent/AGENTS.md",
        "project_context": project_context,
    })

    # 2. Initialize rolling memory with the prompt
    memory = RollingSummaryMemory(llm_summarize=summarize_fn,
                                  system_prompt=system_prompt)

    # 3. Claim the task atomically
    if not shared.claim_task(agent_id, task):
        return  # another agent got it

    # 4. Run the agent turn loop
    memory.add("user", task)
    for turn in range(max_turns):
        response = llm.chat(memory.render())
        memory.add("assistant", response.content)
        if response.tool_calls:
            scratch.write(f"turn_{turn}_tools", json.dumps(response.tool_calls))
            results = [execute(tc) for tc in response.tool_calls]
            memory.add("tool", json.dumps(results))
        if response.done:
            break

    # 5. Persist the outcome
    memory_file.append("Task Complete", f"Task `{task}` resolved by {agent_id}.")
    scratch.cleanup()

---

Conclusion

The gap between a toy agent and a production agent is almost entirely about state. Prompt engineering defines what the agent knows; context management determines how much it can hold at once; persistence ensures it survives; and multi-agent sharing lets agents collaborate. Treat each as an engineering discipline with its own tests, telemetry, and failure modes. The agents that scale are not the ones with the biggest context windows — they are the ones that manage memory as deliberately as any other resource.

---

Data Sources

This analysis synthesizes patterns from production agent frameworks and published research:

• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024) — attention degradation in long contexts.
• Anthropic, Claude Code documentation and AGENTS.md convention (2024-2026).
• OpenAI, "Practices for Building Reliable Agents" engineering guidance (2025).
• LangChain / LangGraph memory module source and documentation.
• Microsoft AutoGen multi-agent framework architecture notes.
• Production post-mortems from autonomous agent deployments (2024-2026), aggregated from public engineering blogs.

---

Promotional Companion: 60-Second Video Guide

This article is part of a multi-channel content launch promoted across Twitter, YouTube Shorts, and our landing page. The companion 60-second video distills the four state-management layers — sliding windows, rolling summaries, file-based persistence (AGENTS.md / scratchpads), and multi-agent state sharing — into a fast-paced visual walkthrough for engineering audiences.

Core claim: Scale your AI engineering capabilities with enterprise-grade prompt architecture and state management guides.

Watch the explainer and read the full technical guide collection on our campaign page:

Read the full guide → hum.pub/author/tech-nexus

Targeting tech leads, AI engineers, and startup founders building production autonomous agents.