GPT-4o can validate a document in a single pass — but at 79% accuracy, one in five results was wrong. For Scoop's document processing pipeline, that wasn't close to good enough.
The core issue was subtle: the model would miss edge cases on its first attempt — misclassified fields, overlooked formatting violations, incomplete extraction checks. A human reviewer would catch these, but at thousands of documents per day the bottleneck was obvious. We needed the model to catch its own mistakes.
The self-critique loop
Rather than fine-tuning or prompt engineering a single pass, we built a three-node graph in LangGraph that lets GPT-4o review and revise its own validation output. The validator produces an initial assessment, the critic evaluates it for gaps or errors, and the revise node feeds the critique back for another pass. A conditional edge controls the loop: if confidence exceeds 0.95 or three iterations have elapsed, the graph exits.
from typing import Literal
from langgraph.graph import StateGraph
from typing_extensions import TypedDict
class ValidationState(TypedDict):
document: str
validation_result: str
critique: str
revision_count: int
confidence: float
def should_continue(state: ValidationState) -> Literal["critic", "__end__"]:
"""Exit when confidence is high enough or max revisions reached."""
if state["confidence"] >= 0.95 or state["revision_count"] >= 3:
return "__end__"
return "critic"
graph = StateGraph(ValidationState)
graph.add_node("validator", validator_node)
graph.add_node("critic", critic_node)
graph.add_node("revise", revise_node)
graph.set_entry_point("validator")
graph.add_conditional_edges("validator", should_continue)
graph.add_edge("critic", "revise")
graph.add_edge("revise", "validator")
workflow = graph.compile()In practice, most documents converge after 1–2 critique rounds. The average across our production dataset was 1.4 rounds per document — meaning the self-critique loop adds roughly 0.4 extra LLM calls on average, a cost well worth the accuracy gain.
Parallel batch processing
A self-critique loop is inherently sequential per document, but documents are independent of each other. We grouped incoming documents into batches of 50 and processed each batch concurrently using asyncio.gather(). This pushed throughput from roughly 3 documents per minute (sequential, with critique) to 200 documents validated in under 90 seconds.
A Streamlit dashboard gave operators real-time visibility into batch progress, per-document confidence scores, and flagged items — making it easy to spot patterns in validation failures without waiting for full batch completion.
Lessons learned
Calibrate the confidence threshold carefully. Setting the exit threshold too high caused the loop to exhaust all three iterations on nearly every document, tripling costs with minimal accuracy gain. Too low, and errors slipped through. We landed on 0.95 after benchmarking against a manually audited set of 1,000 documents.
Bound your loops. The three-iteration cap wasn't arbitrary — it was the point of diminishing returns. Beyond three passes, accuracy improved by less than 0.5% while latency and cost scaled linearly.
Batch size is a rate-limit tradeoff. Larger batches improved throughput but risked hitting API rate limits. Batches of 50 kept us well under the ceiling while maintaining the sub-90-second target for 200 documents.