Knowledge Pull Requests decompose document updates into interpretable claim proposals and text diffs, separating what knowledge changes from how the text changes. Researchers at Johns Hopkins evaluated the approach on Wikipedia revisions across languages and query-driven report updates, finding that KPRs integrate more information per token generated than rewriting from sources or regenerating from scratch, while grounding question answering better than frontier models with search.
The method operates in three stages. First, an LLM decomposes sources into atomic, decontextualized claims—the same process applied offline to cache the main document's claims. Second, the system classifies each source claim as covered (already in the main), conflicting (contradicting existing content), or absent (new). Absent claims are filtered for relevance and routed to document sections; conflicting claims are flagged for human review rather than silently resolved. Third, each affected section is rewritten to integrate the approved claims, and a diff is generated against the original. The result is a ChangeLog: a claim proposal showing what knowledge is being added and where, plus a document diff showing the textual changes.
On Wikipedia, KPRs revised English articles using knowledge from other-language editions. The paper reports information precision (InfoP) of 0.878 for KPRs versus 0.837 for rewriting from raw text and 0.853 for conditioning on unfiltered claims. Information recall for added content (InfoR-A) reached 0.891 for KPRs against 0.716 for raw-text rewriting. On question answering grounded in the revised articles, KPRs achieved 69.2% accuracy on multilingual questions across seven models, compared to 48.4% for raw-text rewriting and 58.5% for unfiltered claims. On the hardest 100 multilingual questions that no dense model answered closed-book, a 7B model grounded on a KPR-revised article outperformed GPT Sol 5.6 with web search, which recovered only 38% accuracy.
The edit cost favored KPRs: they required 11.2 contiguous edit blocks for a reviewer to approve, versus 25.3 for unfiltered claims, while preserving 97.3% of the original article. On RAGTIME query-driven reports updated across two rounds, KPRs maintained higher precision and recall than baselines. In the conflict setting, where round-2 sources contradicted round-1 content, KPRs achieved 0.811 information precision and 0.782 information recall for retained content, compared to 0.666 and 0.625 for raw-text rewriting. The system withheld conflicting claims rather than resolving them implicitly, holding precision where text-conditioned methods were misled.
The implementation uses Qwen 3.5-27B for all LLM steps—claim decomposition, classification, routing, and rewriting. The paper acknowledges that every intermediate step requires an LLM call, making the pipeline computationally expensive; cheaper alternatives like lightweight encoders for containment classification or routing could reduce per-step cost. Human review was not evaluated: the experiments withheld flagged conflicts from the rewrite rather than having humans adjudicate them, leaving open the question of whether a ChangeLog actually makes editors faster and more accurate than a text diff.
For teams maintaining documents that evolve—Wikipedia articles, technical specs, intelligence reports, query-driven syntheses—KPRs offer a deployable pattern: operate over claims rather than raw text, surface conflicts instead of resolving them silently, and concentrate changes into reviewable blocks so that a human can assess what knowledge is actually changing before approving the rewrite.