LLM-Assisted Research Verification Checklist

Academic researchers are increasingly incorporating AI into their research and writing processes. However, because of concerns about hallucinations with LLMs as well as ethical issues like copyright, it’s important to plan and document your use of AI. Below is a verification checklist you can use or modify to fit your research project. It includes considerations for the pre-research phase, the research phase, and the post-research phase. Not every item on the checklist will fit your project, so feel free to modify this checklist.

Pre-Research Phase

Model Documentation

• Document the specific LLM used (model name, version, provider)
• Record known limitations and biases of the chosen model
• Document any fine-tuning or modifications made to the base model
• Note the model’s knowledge cutoff date and potential implications

Prompt Engineering

• Document all prompts used in the research
• Validate that prompts are clear and unambiguous
• Test prompts with different phrasings to ensure consistency
• Record any prompt refinements and their justification

During Research Phase

Data Verification

• Cross-reference LLM-provided facts with primary sources
• Verify all citations and references provided by the LLM
• Document cases where the LLM generates incorrect or hallucinated information
• Maintain a log of verification attempts and their outcomes

Output Analysis

• Check for internal consistency across multiple LLM responses
• Validate mathematical calculations independently
• Review logical reasoning steps for completeness and accuracy
• Document any contradictions or inconsistencies in responses

Methodology Documentation

• Record all interactions with the LLM
• Document the decision-making process for accepting/rejecting LLM outputs
• Note any technical issues or limitations encountered
• Track time spent on verification versus direct LLM interaction

Post-Research Phase

Quality Control

• Have independent researchers verify key findings
• Compare LLM-assisted results with traditional research methods
• Identify potential biases introduced by LLM usage
• Document unexpected or anomalous results

Reproducibility

• Ensure all prompts and interactions are properly documented
• Verify that results can be reproduced by other researchers
• Document environmental factors that might affect reproducibility
• Create detailed instructions for replication attempts

Ethical Considerations

• Assess potential ethical implications of LLM usage
• Document measures taken to protect privacy and confidentiality
• Consider potential biases in research outcomes
• Evaluate the impact on different stakeholder groups

Publication Preparation

Transparency

• Clearly state the role of LLMs in the research
• Document limitations and potential biases
• Provide complete methodology for LLM usage
• Include example prompts and responses where relevant

Documentation

• Prepare supplementary materials detailing LLM usage
• Include verification procedures and results
• Document all modifications to LLM outputs
• Provide access to raw LLM interactions where possible

Peer Review Considerations

• Prepare responses to anticipated methodology questions
• Document validation procedures for peer review
• Include comparison with traditional methods
• Provide justification for LLM usage in research

Long-term Considerations

Archival

• Create backups of all LLM interactions
• Document the research environment and technical setup
• Archive prompts and responses in a retrievable format
• Maintain version control for research materials

Future Verification

• Plan for future replication attempts
• Document potential technological obsolescence issues
• Consider long-term accessibility of research materials
• Prepare for potential model updates or deprecation

Notes and Recommendations

• Regularly update this checklist based on new developments in LLM technology
• Adapt the checklist to specific research domains and requirements
• Consider institutional policies and guidelines regarding LLM usage
• Maintain open communication with other researchers about best practices

I acknowledge the use of Claude.ai to assist with generating this verification checklist. Here is a link to the artifact.