Researcher Sergey V Samsonau has released sciwrite-lint, an open-source verification tool that automatically checks scientific manuscripts for citation accuracy and claim validity. The tool addresses growing concerns about quality assurance in academic publishing as AI-assisted writing accelerates paper production beyond what traditional peer review can handle.
Tool Runs Entirely on Researcher's Local Machine
sciwrite-lint operates completely on the researcher's computer using free public databases, a single consumer GPU, and open-weights models. No manuscripts are sent to external services, preserving privacy and confidentiality. The tool is available via pip install sciwrite-lint.
Comprehensive Verification Pipeline Checks Citations at Multiple Levels
The verification system performs several automated checks on each reference:
- Verifies that cited references actually exist
- Checks retraction status of cited papers
- Compares metadata against canonical records
- Downloads and parses cited papers
- Validates that cited papers support the claims made about them
- Follows citations one level further to check cited papers' own bibliographies
Each reference receives a reliability score aggregating all verification signals. The tool was evaluated on 30 unseen papers from arXiv and bioRxiv with error injection and LLM-adjudicated false positive analysis.
Experimental Extension Measures Research Contribution
Beyond citation integrity, sciwrite-lint includes an experimental SciLint Score that attempts to measure research contribution by implementing frameworks from philosophy of science. The system converts theories from Popper, Lakatos, Kitcher, Laudan, and Mayo into computable structural properties of scientific arguments. The integrity component is the core evaluated feature, while the contribution component is released as experimental code for community development.
According to the paper, traditional science offers two inadequate quality assurance options: journal gatekeeping that measures prestige rather than integrity, and open science with no quality assurance filter between AI-generated text and public record. Samsonau proposes a third option: measuring the paper itself through automated verification.
Key Takeaways
- sciwrite-lint is an open-source tool that automatically verifies scientific paper citations and claims, running entirely on researchers' local machines
- The verification pipeline checks that references exist, verifies retraction status, and validates that cited papers actually support the claims made about them
- Each reference receives a reliability score aggregating multiple verification signals
- The tool was evaluated on 30 papers from arXiv and bioRxiv
- An experimental SciLint Score attempts to measure research contribution using frameworks from philosophy of science