Run Your Own Local RAG & Agent System: LangChain-Chatchat + Ollama

The Stack

• Stack: Streamlit UI + FastAPI backend + LangChain + Ollama models (Qwen2, Llama3, GLM-4).
• Capability: full offline RAG with knowledge-base Q&A, agent tools (search, database, arXiv), and multi-model support.
• OpSec: 100% local — no API keys, no cloud. FAISS vector store on encrypted disk. Apache 2.0 + MIT stack.

Setup Steps

01 — Install Ollama & Pull Models

• Install Ollama from ollama.com.
• Pull the LLM and the embedding model.

ollama pull qwen2:7b
ollama pull bge-m3
ollama list  # verify

02 — Install LangChain-Chatchat

• Create an isolated virtual environment.
• Install the package.

python -m venv ~/chatchat-env
source ~/chatchat-env/bin/activate
pip install langchain-chatchat -U

03 — Configure the Model Platform

• Set the default LLM + embedding model.
• Register Ollama as a model platform with its API base.

chatchat-config model --default_llm_model qwen2:7b
chatchat-config model --default_embedding_model bge-m3
chatchat-config model --set_model_platforms '[{...}]'

04 — Initialize the Knowledge Base

• Point DATA_PATH to your documents folder.
• Run the indexer.

chatchat-config basic --data_path ~/knowledge-base
chatchat-kb -r

05 — Launch & Use

• Start the API + WebUI with one command.
• Open localhost:8501, upload docs, and chat.
• Enable Agent mode for multi-tool chaining.

chatchat start -a
# WebUI → http://localhost:8501
# API  → http://localhost:7861

OpSec Hardening

Network Lockdown

• Bind to 127.0.0.1 only (default).
• Firewall-block ports 8501 and 7861.
• No telemetry — audit with grep.

Data Sovereignty

• FAISS vector store on an encrypted volume.
• No external API calls at runtime.
• 100% open-source stack (Apache 2.0 / MIT).