Clinical Coding Agent for Unstructured Notes

This case study intentionally describes the system at the architecture and engineering level without exposing proprietary data, prompts, internal schemas, clinical rules, company knowledge assets, or implementation details.

I designed and built a production agentic AI system that predicts medical coding from unstructured clinical notes. The agent converts messy clinical text into structured reasoning steps, retrieves relevant domain context from a company-curated clinical knowledge graph, and routes work across models based on task complexity.

---

The Problem

Clinical documentation contains important coding signals, but those signals are often scattered across long, noisy, and inconsistently written notes. A reliable system needs to identify clinically relevant conditions, connect them to the right supporting evidence, and reason through ambiguous cases without exposing users to a brittle one-shot prompt.

The goal was to build an agent that could support medical coding workflows while remaining observable, reviewable, and practical enough for production use.

---

My Role

I sole-authored the project end to end:

• Designed the agent workflow and prompt strategy
• Built the LangGraph orchestration layer
• Implemented condition extraction from unstructured clinical notes
• Integrated retrieval from a company-curated clinical knowledge graph
• Added model routing based on task complexity
• Designed a self-correcting evaluation and retry loop
• Instrumented the full system with on-premise Langfuse
• Collaborated with clinical SMEs for validation and iteration
• Took the system through deployment

---

System Design

The agent separates the workflow into smaller steps rather than asking a single model to perform the entire task at once.

Clinical note
  -> condition extraction
  -> knowledge graph retrieval
  -> evidence-aware coding prediction
  -> confidence evaluation
  -> retry or final answer

The architecture uses model routing so simpler extraction and formatting tasks can run on open-source models, while more complex clinical inference can be escalated to a stronger reasoning model. This made the system more cost-aware without treating every step as equally difficult.

---

Agentic Workflow

The core agent loop combines retrieval, reasoning, and self-evaluation:

This feedback loop helped reduce brittle outputs by giving the agent a structured way to recognize low-confidence responses and attempt correction before returning a result.

---

Observability

I deployed and integrated an on-premise Langfuse instance to make the system inspectable in production-like workflows. Tracing made it possible to debug individual runs, compare prompt and model behavior, inspect intermediate steps, and collaborate with clinical SMEs using concrete examples rather than vague model outputs.

---

Validation

Because the system operates in a clinical domain, validation required both technical and domain review. I worked with clinical subject matter experts to evaluate outputs, refine task decomposition, identify failure modes, and improve the agent's reasoning and confidence behavior over multiple iterations.

---

Key Takeaways

• Agent design matters most when the domain is ambiguous and high-stakes.
• Retrieval quality and task decomposition are as important as the model choice.
• Model routing can reduce cost while preserving stronger reasoning for harder cases.
• Self-evaluation and retry loops make agent behavior more robust than one-shot prompting.
• Observability is essential for debugging, validation, and SME collaboration.

---

Technical Stack