Node2
All insights

Insights · Explainer

RAG: how AI answers from your data, not its memory

A language model only knows what it absorbed during training — frozen, generic, and prone to making things up. Retrieval-augmented generation (RAG) fixes that by fetching the relevant facts from your own documents and handing them to the model at the moment it answers.

The problem RAG solves

As covered in how LLMs work, a model predicts plausible text from frozen training weights. Ask it about your refund policy, last quarter's numbers, or a document it never saw, and it will confidently produce something that sounds right — but isn't yours.

RAG closes that gap without retraining: it retrieves the actual source text and places it into the model's context window, so the answer is grounded in real, current, specific information.

How it works, visualized

The pieces

  • Chunking: your documents are split into small passages so the right piece can be found and fit into context.
  • Embeddings: each chunk is turned into a vector (a point in meaning space). So is the question.
  • Vector search: find the chunks whose vectors are closest to the question's — i.e. the most semantically relevant text, not just keyword matches.
  • Augmented prompt: those top chunks are pasted into the prompt alongside the question.
  • Generation with citations: the model answers using the supplied text — and can point back to which source it used.

Why it beats just "training it on our data"

Fine-tuning bakes style and behaviour into the model; RAG supplies facts at answer time. You usually want RAG for knowledge that changes, is private, or must be cited — update a document and the next answer reflects it instantly, with no retraining.

Where it still goes wrong

  • Bad retrieval → bad answer: if the right chunk isn't found, the model fills the gap — garbage in, garbage out.
  • Chunking matters: split too coarse and you miss the detail; too fine and you lose context.
  • It's grounding, not certainty: RAG sharply reduces hallucination but doesn't eliminate it — citations let a human verify.

Where Node2 fits

RAG is how we make local AI genuinely useful: our Micro AI agents retrieve from your documents — on your own infrastructure, never a public cloud — and answer with citations you can check. Private by default, grounded by design.

See what we build

This explainer is for general understanding and simplifies some details. It reflects common RAG practice as of 2026.