Knowledge2Model/GraphRAG

README

This repository contains scripts and tools for processing, indexing, and querying a SQL-based knowledge graph, plus a simple chat interface. Below is an overview of each component and instructions for exporting/importing the Neo4j database.

---

Scripts

Pipeline Flow

Stage 1: Data Preparation

1. db2annoy.py - Preprocessing
   • Reads models from SQLite database
   • Generates semantic embeddings using sentence transformers
   • Builds and saves Annoy index for fast similarity search
   • Outputs: model_metadata.json, model_embeddings.npy, models_index.ann, model_texts.npy.

Stage 2: Graph Database Setup

2. db2neo4j.py - SQL → Neo4j Conversion
   • Transfers model data from SQLite to Neo4j graph database
   • Creates nodes (Model, Problem, Library, Tag, Metric, HealthStatus)

Stage 3: Backend Service

3. chatbot_backend.py - Flask Backend (RAG Pipeline)
   • Implements two-stage retrieval:
     1. Semantic search using Annoy index (fast similarity matching)
     2. Graph queries using Neo4j (relationship-based filtering)
   • Connects to LLM (LMStudio/Ollama) for natural language generation

Stage 4: User Interface

4. chatbot_frontend.html - Web Interface
   • Minimal chat UI for interacting with the backend
   • Sends queries to Flask backend and displays responses

---

Testing & Debugging Tools

• semantic_search.py - Standalone Search Tester
  • Tests the Annoy index independently without running the full pipeline
  • Useful for debugging semantic search without Neo4j or LLM

---

Workflow Summary

Setup (run once or when data changes):

1. python db2annoy.py      # Build embeddings
2. python db2neo4j.py      # Populate Neo4j

---

Important Notes

Large Generated Files

The following files are generated by db2annoy.py and are not included in the repository due to their size:

File	Size	Purpose
model_embeddings.npy	~127 MB	Dense vector embeddings for semantic search
model_texts.npy	~828 MB	Original concatenated text from models
models_index.ann	~151 MB	Annoy index structure for fast retrieval
model_metadata.json	Small	Model IDs and metadata for results

Neo4j Database Dump & Restore

Export (Dump)

# Replace the backup_neo4j with the updated backup folder name
sudo neo4j-admin dump system \         
  --to-path=<path>/Knowledge2Model/GraphRAG/backup_neo4j

sudo neo4j-admin dump neo4j \
  --to-path=<path>/Knowledge2Model/GraphRAG/backup_neo4j

Import (Load)

# Replace the backup_neo4j with the updated backup folder name
sudo neo4j-admin database load system \
  --from-path=<path>/Knowledge2Model/GraphRAG/backup_neo4j \
  --overwrite-destination=true

sudo neo4j-admin database load neo4j \
  --from-path=<path>/Knowledge2Model/GraphRAG/backup_neo4j \
  --overwrite-destination=true

sudo chown -R neo4j:neo4j /var/lib/neo4j/data

Neo4j Credentials

Set environment variables or update your .env file:

NEO4J_URI="bolt://localhost:7687"
NEO4J_USER="neo4j"
NEO4J_PASSWORD="12345678"