The US government publishes stacks of reports every year — nursing home abuse investigations, USDA administrative reform, things like that. Dozens of pages of dense text. Nobody wants to read every word, but the information is genuinely useful. This project trains an AI to pull out a summary for you.

But AI has a problem — it hallucinates. It will confidently write things into the summary that never appeared in the source document. That's dangerous, especially for policy documents. So the key contribution of this project is: after the AI generates a summary, it checks its own work for fabrications. That's the "Self-Checking" part.

There's a second problem — the documents are too long for the AI to read in one pass. Our model can take in at most 16,000 tokens at a time, but government reports often run into tens of thousands of words. Everything past that limit just gets cut off and never processed.

The fix is called Divide & Conquer: split the document into chunks, summarize each chunk, then merge. We tried three different ways of doing the merge and ran experiments to see which works best.

Quick setup: our USDA report is long enough that we split it into 2 chunks. I'll walk you through each method using the actual content of those 2 chunks so you can see exactly what happens.

Exactly — that's the progression across the three methods. The key insight: don't try to make the big merge smarter, try to make each merge smaller. Map-Refine nails that principle.

Every metric below is computed on the same example — a sentence from the actual project data (the USDA report). Burn these two into your memory:

ROUGE-1 is the simplest: count how many words appear in both texts. But "how many words overlap" isn't enough — you also need to distinguish "the AI said too much" from "the AI said too little." That's where precision and recall come in.

The Venn diagram makes this obvious in one look:

From this diagram, three questions are answered immediately:

① How much of what the AI said is correct? → right circle: 5 overlap words ÷ 10 AI words = 50%. That's Precision.

② How much of the reference did the AI cover? → left circle: 5 overlap ÷ 8 reference words = 62.5%. That's Recall.

③ Combined score? → harmonic mean of those two — that's F1, which we call the ROUGE-1 score.

Because either one alone can be gamed.

If you only look at precision, the AI could say just one word — as long as it's correct — and precision is 100%. Useless summary.

If you only look at recall, the AI could copy-paste the entire source document for 100% recall. Not a summary either.

F1 is the harmonic mean — whenever either side goes extreme, F1 gets dragged down. Both have to be good for F1 to be good. That's why it's the more reliable combined metric.

ROUGE-1 only matches single words, which is too loose. If the reference says "business centers" and the AI says "business cafeteria", ROUGE-1 still counts "business" as overlap — but clearly the two phrases mean different things.

ROUGE-2's upgrade: take every pair of adjacent words as a unit (a bigram), and only count a match if the whole pair is identical. Same example, run through it:

Only 2 bigrams match: (business, centers) and (mission, areas).

"established business" and "created business" don't match at the bigram level, even though ROUGE-1 would have counted "business" as a match.

ROUGE-1 only asks: did this word appear? Order is ignored. So even if the AI says all the right words in a scrambled order, ROUGE-1 would still score high.

ROUGE-L adds: overlapping words also have to appear in the same relative order in both texts. It finds the "longest common subsequence" (LCS) — you can skip intermediate words, but you can't reorder them.

ROUGE-L checks: did the AI not only use the right words but also put them in a reasonable order? A good summary isn't just about having the right information — it has to be organized coherently.

Right — you just spotted ROUGE's biggest weakness: it only sees the surface form of words, not their meaning. "Established" and "created" are two completely different tokens to ROUGE — 0 points. Any human would say they mean the same thing in context.

BERTScore uses a neural network to understand meaning. Think of each word as a point in high-dimensional space — semantically similar words end up close together. BERTScore compares distances in that space, not surface spellings.

Yes, at the core it's Y or N — but there's a whole protocol behind it. Let me walk through the full flow.

First, why we need it: ROUGE and BERTScore both compare against the reference summary, but the reference is human-written and isn't the same as the source document. An AI could produce a high-ROUGE summary that still fabricates content. So we need a metric that compares directly against the original document.

Sharp question — it's one of the real limitations. The judge is also an LLM; it can misjudge, and complex reasoning can throw it off.

This approach is called LLM-as-Judge, and it's the mainstream way of evaluating faithfulness in NLP right now — human fact-checking every summary is too expensive. A stronger setup would use a more powerful model (like GPT-4) as the judge; our project uses the same model as both generator and checker, which has known limitations. We call this out in the limitations section.

Even so, the Unfaithful cases it flags are meaningful — at minimum, they show the summary doesn't hold up within the model's own reasoning framework, which is a useful signal.

ROUGE measures quality — whether what you said is correct — but not quantity — whether you said enough.

Picture this: reference has 10 sentences, AI writes only 2, but those 2 are the most important ones and every word is in the reference. ROUGE-1 scores high. Would you call that a good summary? It skipped 80%.

All five put together gives you the full evaluation framework:

ROUGE-1: did the words show up in the reference? (word-level quality)

ROUGE-2: did the word pairs show up? (phrase-level quality, stricter)

ROUGE-L: did the words appear in the right order? (structure)

BERTScore: how close is the meaning? (semantic, catches paraphrases)

Faithfulness: did the AI make anything up? (vs. source)

Coverage: did it say enough? (quantity)

Miss any one and you only see part of the story.

Numbers first, then we unpack them row by row.

Now row by row — can't just stare at numbers, have to read what they mean.

Numbers done. Now the conclusions. Three layers: what we solved, what we didn't, what comes next.

Conclusions in. What would the best next steps be? Three directions:

One-sentence wrap-up: D&C successfully solved the truncation problem, Map-Refine is the current best option, and the next priority is suppressing merge-step rewriting so ROUGE-2 and BERTScore can improve together instead of trading off.

Sharp question — shows you actually got it. Four layers of response:

One line: the ROUGE-2 drop isn't experimental failure — it's a known side effect of the merge step. We understand the cause, compensated with BERTScore, and pointed to a clear fix. That's good science.

These are real questions that came up during presentations of this project. If you're presenting this work, you need solid answers to all of them. The core theme is one thing: chunking is lossy compression — how do you handle the information loss?

Our model has a context window of about 16,000 tokens, but government reports can be 30,000 to 50,000 tokens. So we split sequentially with a fixed token budget — about 8,000 tokens per chunk. A typical USDA report ends up as 2 to 4 chunks.

Yes — each chunk shares about 200 tokens with the next one, a sliding window. That's roughly 5% of each chunk. The reason: if you hard-cut at exactly token 8,000, you might land in the middle of a sentence. The overlap ensures boundary context isn't lost — the end of chunk 1 and the beginning of chunk 2 share a small strip of text so the model has continuity.

Too little overlap risks splitting sentences. Too much overlap wastes the context window on repeated content. 200 tokens is the empirical sweet spot.

Yes — that's the core tradeoff of Divide-and-Conquer, and we take it seriously. Chunking is lossy: each chunk is summarized without seeing the other chunks, so cross-chunk context is lost. We address this in three ways:

First, overlap at chunk boundaries — 200 tokens shared between adjacent chunks, so sentences at the boundary aren't split.

Second, the choice of merge strategy matters enormously. MapReduce summarizes every chunk in isolation and merges once — maximum context loss. Map-Refine fixes this by carrying the draft forward: when it reads chunk 2, it already has the summary from chunk 1 as context. Cross-chunk information is preserved incrementally. This is why Map-Refine outperforms MapReduce on every positive metric.

Third, our metrics actually capture the effect. Coverage went from 63.7% to 86.2% — meaning D&C recovers most of the information that Baseline loses to truncation. BERTScore went up 18.6%, showing semantic quality improved, not degraded. If chunking were destroying too much context, these numbers would go down, not up.

In the current pipeline, no — we don't have a per-chunk quality gate. Every chunk summary goes directly into the merge step regardless of quality. That's a real limitation.

A clear next step would be to add a self-check at the chunk level too — not just at the final summary. If a chunk summary scores below a threshold on a quick faithfulness check, regenerate it before it enters the merge. We didn't implement this, but we know it's the right direction.

It would — but intentionally, we didn't. We used the same model family for both generation and judging. The reason: if we used GPT-4 as the judge, the Faithfulness score would look better, but we wouldn't know whether the improvement came from our pipeline design or just from having a stronger judge.

This is a proof-of-concept project. We want to isolate the effect of the D&C method itself. If the idea works even with a modest 7B model judging itself, it'll work even better with a stronger judge later. That's a much stronger claim than "it works, but only if you throw GPT-4 at it."

Larger context windows exist — GPT-4 has 128K tokens. But we're using Qwen 2.5-7B on an A100 with 4-bit quantization. At that model size, the effective context window is about 16K before quality degrades noticeably.

More importantly, research shows that even with larger windows, models tend to lose information in the middle of very long inputs — the "lost in the middle" problem. The model pays attention to the beginning and end but forgets things in the middle. Chunking with D&C actually forces the model to process every part of the document carefully, which is why our coverage goes up rather than relying on a long context window that might silently ignore the middle.

BLEU was designed for machine translation, not summarization. It focuses on n-gram precision and doesn't compute recall — but recall is critical for summaries because we need to know how much of the reference we covered. METEOR is closer but far less commonly reported in summarization papers, so our results wouldn't be comparable to prior work.

ROUGE has been the standard in summarization for 20 years. Every paper reports it. We chose it so our numbers are directly comparable — and we added BERTScore specifically because we knew ROUGE's paraphrasing blind spot would hurt us in a D&C pipeline.

No — the raw numbers look low because BERTScore applies baseline rescaling. Without rescaling, most scores cluster between 0.85 and 1.0, which makes everything look identical. The library subtracts a corpus-level baseline so differences become visible. So 0.09 means "0.09 standard deviations above the corpus mean," not "9% similar."

The important thing is the relative difference: Map-Refine is 18.6% higher than Baseline. That's a substantial and meaningful improvement.

Human evaluation. All our metrics are automated proxies. The gold standard would be having domain experts — people who actually read government reports — rate the summaries on a Likert scale for informativeness, fluency, and factual accuracy. We didn't do it because of time and cost, but it would be the strongest validation of whether D&C actually produces better summaries in practice, not just on automated benchmarks.