Personal LLM Assistant

Introduction

Building a personal LLM assistant combines several cutting-edge AI techniques to create a conversational agent that understands your specific domain, remembers context, and provides accurate, helpful responses. Unlike using a generic chatbot, a personal assistant can be tailored to your documents, preferences, and use cases. This project walks through the architecture and implementation of such a system.

Architecture Overview

RAG		Direct LLM		Tools
Pipeline		Generation		(search,
				calc)

Retrieval-Augmented Generation (RAG)

RAG grounds LLM responses in your actual documents, eliminating hallucination for factual queries.

- User asks	"What was our Q3 revenue?"
- Retrieved	["Q3 revenue was $2.3M, up 15% YoY..."]
Context	{retrieved_chunks}
Question	{user_query}
Answer	'''

Chunking Strategies

Fixed-size chunks	Split every 500 tokens
Pro	Simple, predictable
Con	May split mid-sentence or mid-concept
Semantic chunks	Split at paragraph/section boundaries
Pro	Preserves meaning
Con	Variable sizes, some too long
Recursive splitting	Split by paragraph, then sentence if too long
Pro	Balances meaning and size
Con	More complex implementation
Overlap	Include 50-100 tokens from adjacent chunks
Pro	Context continuity
Con	Increased storage

Fine-Tuning for Domain Adaptation

When RAG is insufficient (style adaptation, complex reasoning in domain):

Fine-Tuning Approaches

Full fine-tuning: Update all model parameters

- Requires significant GPU memory

- Risk of catastrophic forgetting

- Best when you have large domain dataset

LoRA (Low-Rank Adaptation)

- Freeze base model weights

- Add small trainable matrices to attention layers

- Original weight W → W + ΔW where ΔW = A × B (low rank)

- A is d×r, B is r×d (r << d, typically r=8-64)

- Trains only ~0.1% of parameters

- Can merge back into model for inference

QLoRA: LoRA + 4-bit quantization

- Base model in 4-bit (75% memory reduction)

- LoRA adapters in full precision

- Enables fine-tuning 65B models on single GPU

Training Data Preparation

Format	Instruction-Response pairs
"instruction"	"Summarize the key points of our privacy policy",
"input"	"[privacy policy text]",
"output"	"The privacy policy covers: 1) Data collection..."
"instruction"	"Draft a response to this customer complaint",
"input"	"Customer says delivery was late and product damaged",
"output"	"Dear [Customer], We sincerely apologize..."

Memory and Context Management

Context Window Management

- Most LLMs have 4K-128K token context windows

- Conversation history grows with each turn

- Strategy: Summarize old messages, keep recent ones verbatim

Memory hierarchy

1. Working memory: Current conversation (full text)

2. Short-term: Recent conversations (summaries)

3. Long-term: Vector DB of all past interactions

Implementation

if conversation_tokens > 0.7 × max_context:

old_messages = conversation[:-5] # keep last 5 verbatim

summary = llm.summarize(old_messages)

conversation = [summary_message] + conversation[-5:]

Tool Integration

- web_search(query)	Search the internet
- calculator(expression)	Compute math
- calendar(action, params)	Check/create events
- email(action, params)	Read/send emails
- code_executor(code)	Run Python code
User	"What's 15% tip on $47.80?"
LLM decides	Use calculator tool
Tool call	calculator("47.80 * 0.15")
Tool result	"7.17"
LLM response	"A 15% tip on $47.80 would be $7.17, for a total of $54.97."

Evaluation and Quality Assurance

Metrics for RAG quality

- Retrieval accuracy: Are the right chunks found?

- Answer faithfulness: Does response match retrieved context?

- Answer relevance: Does response address the question?

Testing approach

Create evaluation dataset of 100+ question-answer pairs

Run system on questions, compare to golden answers

Measure: Exact match, ROUGE score, human rating (1-5)

Iterate: Improve chunking, embedding model, or prompt

until quality meets threshold (e.g., 90% faithfulness)

Summary

Building a personal LLM assistant requires combining retrieval (for grounding in facts), generation (for fluent responses), memory (for conversation continuity), and tools (for real-world actions). RAG provides the most immediate value by connecting your documents to an LLM. Fine-tuning adapts behavior and style. The resulting system can serve as a domain expert, writing assistant, or productivity tool tailored specifically to your needs.

Introduction

Architecture Overview

RAG		Direct LLM		Tools
Pipeline		Generation		(search,
				calc)

Retrieval-Augmented Generation (RAG)

RAG grounds LLM responses in your actual documents, eliminating hallucination for factual queries.

- User asks	"What was our Q3 revenue?"
- Retrieved	["Q3 revenue was $2.3M, up 15% YoY..."]
Context	{retrieved_chunks}
Question	{user_query}
Answer	'''

Chunking Strategies

Fixed-size chunks	Split every 500 tokens
Pro	Simple, predictable
Con	May split mid-sentence or mid-concept
Semantic chunks	Split at paragraph/section boundaries
Pro	Preserves meaning
Con	Variable sizes, some too long
Recursive splitting	Split by paragraph, then sentence if too long
Pro	Balances meaning and size
Con	More complex implementation
Overlap	Include 50-100 tokens from adjacent chunks
Pro	Context continuity
Con	Increased storage

Fine-Tuning for Domain Adaptation

When RAG is insufficient (style adaptation, complex reasoning in domain):

Fine-Tuning Approaches

Full fine-tuning: Update all model parameters

- Requires significant GPU memory

- Risk of catastrophic forgetting

- Best when you have large domain dataset

LoRA (Low-Rank Adaptation)

- Freeze base model weights

- Add small trainable matrices to attention layers

- Original weight W → W + ΔW where ΔW = A × B (low rank)

- A is d×r, B is r×d (r << d, typically r=8-64)

- Trains only ~0.1% of parameters

- Can merge back into model for inference

QLoRA: LoRA + 4-bit quantization

- Base model in 4-bit (75% memory reduction)

- LoRA adapters in full precision

- Enables fine-tuning 65B models on single GPU

Training Data Preparation

Format	Instruction-Response pairs
"instruction"	"Summarize the key points of our privacy policy",
"input"	"[privacy policy text]",
"output"	"The privacy policy covers: 1) Data collection..."
"instruction"	"Draft a response to this customer complaint",
"input"	"Customer says delivery was late and product damaged",
"output"	"Dear [Customer], We sincerely apologize..."

Memory and Context Management

Context Window Management

- Most LLMs have 4K-128K token context windows

- Conversation history grows with each turn

- Strategy: Summarize old messages, keep recent ones verbatim

Memory hierarchy

1. Working memory: Current conversation (full text)

2. Short-term: Recent conversations (summaries)

3. Long-term: Vector DB of all past interactions

Implementation

if conversation_tokens > 0.7 × max_context:

old_messages = conversation[:-5] # keep last 5 verbatim

summary = llm.summarize(old_messages)

conversation = [summary_message] + conversation[-5:]

Tool Integration

- web_search(query)	Search the internet
- calculator(expression)	Compute math
- calendar(action, params)	Check/create events
- email(action, params)	Read/send emails
- code_executor(code)	Run Python code
User	"What's 15% tip on $47.80?"
LLM decides	Use calculator tool
Tool call	calculator("47.80 * 0.15")
Tool result	"7.17"
LLM response	"A 15% tip on $47.80 would be $7.17, for a total of $54.97."

Evaluation and Quality Assurance

Metrics for RAG quality

- Retrieval accuracy: Are the right chunks found?

- Answer faithfulness: Does response match retrieved context?

- Answer relevance: Does response address the question?

Testing approach

Create evaluation dataset of 100+ question-answer pairs

Run system on questions, compare to golden answers

Measure: Exact match, ROUGE score, human rating (1-5)

Iterate: Improve chunking, embedding model, or prompt

until quality meets threshold (e.g., 90% faithfulness)

Personal LLM Assistant

Introduction

Architecture Overview

Retrieval-Augmented Generation (RAG)

Chunking Strategies

Fine-Tuning for Domain Adaptation

Fine-Tuning Approaches

LoRA (Low-Rank Adaptation)

Training Data Preparation

Memory and Context Management

Context Window Management

Memory hierarchy

Implementation

Tool Integration

Evaluation and Quality Assurance

Metrics for RAG quality

Testing approach

Summary

Continue learning this concept

Personal LLM Assistant

Introduction

Architecture Overview

Retrieval-Augmented Generation (RAG)

Chunking Strategies

Fine-Tuning for Domain Adaptation

Fine-Tuning Approaches

LoRA (Low-Rank Adaptation)

Training Data Preparation

Memory and Context Management

Context Window Management

Memory hierarchy

Implementation

Tool Integration

Evaluation and Quality Assurance

Metrics for RAG quality

Testing approach

Summary

Continue learning this concept