threat-intel-rag started from a simple, annoying problem: I wanted a completely offline system that could parse the CISA Known Exploited Vulnerabilities (KEV) catalog and turn dense, inconsistent threat reports into clean structured JSON — CVE ID, vendor, required action — without hallucinating fields that don't exist in the source data.
The first version was just prompting a small local model with the retrieved context and asking nicely for JSON. It didn't work reliably enough to trust.
Where pure prompting breaks down
Small, consumer-GPU-friendly models are genuinely useful for RAG, but they're inconsistent at strict output formatting under load — especially with dense, jargon-heavy security text. Ask for structured JSON from raw retrieved chunks and you'll get fields invented from pattern-completion instinct rather than the actual source. For a threat-intel tool, an invented "required action" is worse than no output at all — it's actively misleading.
Prompting alone gives you probabilistic compliance. Threat intelligence needs deterministic structure.
Two-phase architecture
I split the pipeline into two genuinely separate phases rather than trying to solve retrieval and formatting in one model call:
- Phase 1 — LangChain Retrieval. Scrape and chunk the raw CISA KEV JSON feed, embed it, and store it in a Chroma vector database. This phase's only job is finding the right context — nothing else.
- Phase 2 — LoRA Extraction. A custom LoRA adapter, fine-tuned on
Qwen/Qwen2.5-0.5B-Instruct, takes the retrieved context and is trained specifically to output the target JSON schema — and only from what's actually in the context window, not from parametric knowledge.
Why fine-tune a 0.5B model instead of prompting a bigger one
The honest answer: constraints. This needed to run on a consumer GPU, fully offline, with no API cost. A 0.5B model with a LoRA adapter trained on exactly this extraction task outperforms a much larger general-purpose model being prompted zero-shot for the same structured task — because the small model isn't trying to be a generalist. It has one job, and PEFT training makes that job reliable instead of probabilistic.
This mirrors a pattern I keep running into in LLM security work: general-purpose capability and reliable, constrained behavior are often in tension. A system that needs to be trustworthy for one narrow task is usually better served by narrowing the model's actual degrees of freedom — through fine-tuning, retrieval constraints, or deterministic post-processing — than by hoping a bigger, more capable model happens to stay in its lane.