Improving LLM Accuracy and Minimizing Hallucinations with Query Refinement and Knowledge Graphs

Abstract

Recent developments in natural language processing (NLP) using large language models (LLMs) have transformed information retrieval systems. Problems still exist, however, in high stakes use cases where high accuracy is an essential requirement. A key issue is the hallucination problem, where models generate information unsupported by the underlying data, potentially leading to dangerous misinformation. This paper introduces a new approach addressing this gap by combining large language models (LLMs) with Query Refinement Technique and knowledge graphs (KGs) to improve question-answering system accuracy and credibility. Our approach employs LLMs to transform natural language questions into Cypher queries and complements this with a three-phase query-checking Module. The Module enforces syntactic correctness, semantic compatibility with KG schemas, and logical relationship integrity to enable proper information extraction from a knowledge graph in order to mitigate errors such as hallucinations. Evaluating on MedQA and Custom biomedical dataset for various tasks, our method drastically reduced hallucinations and achieved F1 rates of 91.1% (MedQA) and 86.0% (our dataset) with domain-fine-tuned models like Llama-3.1-8B-UltraMedical. Importantly, KG-validated data coupled with domain-fine-tuned models performed best amongst other LLM methods. Our query checker addressed crucial errors in 85% of the cases, correcting node-type mismatching and reversed relationships. Open-source models significantly improved through prompt engineering and algorithmic optimization and approached accuracy of closed-source LLMs. By grounding responses on the Unified Medical Language System (UMLS) KG, our system illustrates how structure-based knowledge verification can balance LLM flexibility with clinical precision. The method presents a roadmap for building reliable AI systems in mission-critical applications.