LLM Fine-Tuning: Your Dataset Is the Hard Part, Not the Run

The verdict. You have a domain the base model handles badly and you are about to fine-tune. The decision is not train-or-not; it is which gates your dataset must pass before you spend GPU-hours. For a security engineer building an internal smart-contract-audit assistant, our call was halt: fix the data and grounding pipeline first. Do not turn a polished bad dataset into a more expensive problem.

This is a field report from our internal SFT pipeline for Solidity repository audit work, recorded in June–July 2026. It is not a claim that a trained model beat a baseline. No trained-model outcome receipt exists in this record: no verified adapter artifact, no evaluation win, and no benchmark delta. What we have are failures caught before training, and a gate sequence worth copying.

1. The operator and the leak: your dataset looks finished because it can serialize

You run a small security or developer-tools team. The base model misses the domain patterns you care about, so a domain fine-tune feels like the obvious next move. You can already export ChatML. You may even have thousands of rows. That is precisely when the dataset starts lying: a file that parses is not necessarily a training set that teaches inspection, restraint, or source-backed answers.

Our internal pipeline was intended to create blind audit prompts and grounded targets from Solidity repositories. Its eight stages were: Selector → Parse Selector Output → Blind Validation → Target Writer → Target Sanitization → Citation Repair → Target Validation → ChatML Build + Export ([GBrain: projects/bastet-pipeline, July 2026]). The visible output was reassuring: v0 held 6.0M training tokens from 26 repositories, 77% of records scored 80–100 on its quality metric, and two repositories, yearn and balancer, were held out for validation ([GBrain: projects/bastet-pipeline, July 2026]).

Those receipts describe a pipeline state, not a model result. They also did not protect us from a deduplication defect that retained only 287 records from a much larger candidate set ([GBrain: projects/bastet-pipeline, July 2026]). That is the first rule: dataset quality is not one score. It is a chain of separate checks, and one broken link makes the aggregate number decorative.

2. The choice is a gate sequence, not a training recipe

An SFT dataset has to do two jobs at once. It must show the behavior you want, and it must withhold the answer long enough for the model to learn the behavior rather than copy a label. For audit work, that means source code in the prompt, a report in the target, and no leaked finding title, vulnerability class, or label in the training prompt ([GBrain: projects/bastet-pipeline, July 2026]).

Our pipeline used two agent stages: a selector chooses relevant code, then a target writer produces the audit target. Source was rendered into agent_input.md within a 120K-character inline budget; individual inputs were roughly 36–90KB ([GBrain: projects/bastet-pipeline, July 2026]). That detail matters because long-context data work has mundane failure modes too: shell ARG_MAX was approximately 100–200KB, so prompts had to travel through stdin or temporary files rather than command-line arguments ([GBrain: projects/bastet-pipeline, July 2026]).

The seven gates below are transferable. The exact figures are ours; the sequence is the point.

1. Dedup audit: exact first, semantic second. Count repeated serialized rows, repeated source artifacts, and near-duplicate prompts that differ only in wrapper text. We removed 191 exact duplicates in v0, then separately found the hash bug that left only 287 records ([GBrain: projects/bastet-pipeline, July 2026]). Your pipeline can silently eat most of its data while still producing a clean-looking JSONL.
2. Label hygiene: normalize before you stratify. Our corpus contained mojibake such as Re+AC0-erntrancy, unmapped external labels, and a bias toward High severity ([GBrain: projects-qwen36-bastet-lora; qwen36-bastet-lora-data, July 2026]). A severity chart built on broken labels is not a distribution; it is camouflage.
3. Negative coverage: teach the model what not to flag. An earlier corpus had 13 positive-only splits—no negative cases at all ([GBrain: projects-qwen36-bastet-lora, July 2026]). The curated SolidiFI v1.5 dataset later had 2,187 rows and 19.98% negatives ([GBrain: projects-qwen36-bastet-lora, July 2026]). Fixed versions, clean contracts, and false-positive traps teach abstention and discrimination; positives alone teach suspicion.
4. Length and truncation: measure the cutoff, not the average. A measured split had 137 train and 39 validation rows; 26% of train and 33% of validation rows exceeded 4K tokens, while 32% of train rows were truncated ([GBrain: qwen36-bastet-lora-data, July 2026]). If one third of examples end mid-reasoning, the model gets a lesson in stopping mid-reasoning.
5. Ground-truth leakage: make this the hard block. In our design, all other validation issues could warn; leaked finding titles, classes, or labels had to stop the record ([GBrain: projects/bastet-pipeline, July 2026]). This is the gate that kills a run on purpose.
6. Localization and citation: require a source span. The Web3Bugs grounding pilot failed because the model could not reliably map reports back to source ([GBrain: projects-qwen36-bastet-lora, July 2026]). The recovery condition was candidate recall above 70% before proceeding to LLM grounding ([GBrain: projects-qwen36-bastet-lora, July 2026]). Below that, fluent prose is not a finding.
7. Hold out repositories, not random rows. yearn and balancer were the two held-out validation repositories in v0 ([GBrain: projects/bastet-pipeline, July 2026]). Repository-level isolation is the relevant pattern when source families, templates, and coding style can otherwise leak across a random split.

3. Pick the cheapest path that can falsify the fine-tune

Start with the approach that can tell you a fine-tune is unnecessary. There is no dataset-size magic number here; our exports describe one internal audit domain, not a universal SFT requirement.

Approach	Use it when	Cost shape	Catch
Prompt + RAG on the base model	Test this first	$0 training; eval attention	Retrieval may solve the domain gap
Synthetic-only SFT	Controlled bug variants matter	Training plus synthetic review	Our 9,369 rows were entirely injected bugs
Agent-curated SFT	Source-grounded behavior is required	Data-generation hours plus review	Must pass blind-prompt and leakage gates
Full fine-tune vs LoRA	Data already passes gates	GPU-hours × rows × epochs	QLoRA execution here is unverified
Do nothing first	Base model meets your bar	$0	Stop before inventing a project

Prompt + RAG on the base model is the first experiment, not a consolation prize. Give the base model the relevant repository context, ask it to cite source spans, and score it on your held-out set. If it already clears the bar, stop. A fine-tune cannot justify itself by sounding more domain-specific.

Synthetic-only SFT is a coverage instrument, not proof of realism. Our sft_grounded_v1 export contained 9,369 ChatML rows, entirely SolidiFI synthetic injected bugs ([GBrain: projects-qwen36-bastet-lora, July 2026]). That may be useful for controlled vulnerability patterns. It is not evidence that the model learned to navigate production repositories, distinguish a true finding from a trap, or localize an explanation to real source.

Agent-curated SFT is where the pipeline earns its complexity. The selector-plus-target-writer design generated blind ChatML prompts, with sanitization, citation repair, and leakage validation downstream ([GBrain: projects/bastet-pipeline, July 2026]). A separate re-export recorded 352 prompts: 283 train, 69 validation, and 80 skipped ([GBrain: projects-qwen36-bastet-lora, July 2026]). Skips are not waste when they document refusal to manufacture a bad record.

Full fine-tune versus LoRA is shape-only guidance here. A full run generally changes more weights and carries a larger compute and operational commitment; LoRA/QLoRA changes the adaptation shape and can be easier to iterate. Our internal plan mentioned QLoRA, but its execution is unverified. Choose neither until the data passes. The cost shape is: GPU-hours × (dataset size × epochs) + your attention on eval loops. We have no completed training-cost receipt, so any claim that one clean small run beats one large dirty run is unverified as an outcome; it is a sensible hypothesis to test, not a result to advertise.

4. Where the dataset failed before it had the right to train

The most useful receipts were not the successful exports. They were the defects that made “just start a run” indefensible.

Failure mode	What the receipt says	Operator impact	Gate response
Hash dedup defect	Only 287 records kept	Silent data loss	Audit count deltas
Artifact duplication	8,005 rows / 302 artifacts; max 53 duplicates	Repetition masquerades as evidence	Dedup by artifact and prompt
Positive-only split	13 splits had no negatives	Over-flagging gets trained	Add clean, fixed, trap cases
Truncation	32% of train rows cut off	Broken target behavior	Shorten or raise context
Grounding failure	Reports missed source localization	Fluent, uncheckable findings	Halt below 70% recall

The artifact duplication receipt was especially ugly: 8,005 training rows across 302 unique artifacts, with as many as 53 duplicates per artifact ([GBrain: projects-qwen36-bastet-lora, July 2026]). That is not 8,005 independent pieces of evidence. It can overweight a small source subset until the model learns the dataset’s repetition pattern instead of the domain.

Then the pipeline itself collapsed at scale. 1,477 failures were attributed to local-model degradation under GPU load: tool calls stopped working and the model returned prose instead ([GBrain: projects/bastet-pipeline, July 2026]). Do not turn that into “local models cannot generate SFT data.” The documented root cause had two fixable parts: GPU-load degradation and a SOUL.md instruction to “use file operations,” which made the model describe file work instead of call tools. The successful 21 cases were the ones where it happened to call read_file; the fix explicitly required tool calls such as write_file and read_file, after which all 63 regression tests passed ([GBrain: projects/bastet-pipeline, July 2026]).

That is a data-quality lesson too. Your producer pipeline is part of the dataset. If it changes mode under load, your “examples” can be a mixture of real work, prose about work, and malformed fallbacks. Log the failure class before you count output rows.

5. When not to fine-tune: make the $0 call first

Skip training and fix the cheapest broken boundary first when any of these are true:

• Your held-out repository evaluation shows the base model plus prompt/RAG already passes your bar. Training is then a new risk, not a repair.
• The prompt leaks the label, finding title, or answer. Treat this as a zero-tolerance hard block, not a score to average.
• Candidate recall for source localization is below 70%. Our grounding job stopped at this boundary ([GBrain: projects-qwen36-bastet-lora, July 2026]).
• You cannot state how many examples are duplicates, negative, truncated, or label-mappable. The dataset is still an investigation.
• Your generator becomes unreliable under concurrency. Reduce load, fix tool-use prompts, and prove the pipeline with regressions before launching a multi-day run.

Our observed throughput estimates show why this restraint matters: eight workers were estimated at roughly 48 cases/hour; a 1,500-case run was estimated at 30–48 hours; selector work at 200–360 seconds/case and target writing at 80–160 seconds/case ([GBrain: projects/bastet-pipeline, July 2026]). Those are pipeline-time receipts, not training-time costs. They are enough to show the asymmetry: gates cost hours; a contaminated production run consumes hours and creates misleading evidence.

6. Run a 30-day SFT dataset test before buying more compute

Record a baseline on day zero, then review the same ledger at day 30. Do not substitute a vibe check or a loss curve for this table.

Metric	Baseline	Day 30	Decision use
Gates 1–7 failed	Count failures by gate	Pass/fail each gate	Find the blocking defect
Rows after cleaning	Before vs after dedup	Updated retained rows	Expose false volume
Candidate recall	Source-localization rate	Measured percentage	Must reach 70%
Halt decision	Hold / scrap / train	Re-decide with ledger	Spend GPU-hours only on green

The rule is deliberately blunt: if leakage blocks = 0, candidate recall is at least 70%, negatives meet your declared target ratio, and truncation is below 10%, you may spend the GPU-hours. Otherwise, the dataset is the project, not the training run. The 70% threshold is our documented recovery gate; the below-10% truncation target is a recommended operating threshold, not a verified outcome from this pipeline.

At day 30, compare base-model performance against any candidate adapter only after the gate ledger is green. Use the same held-out repositories, the same source-localization rubric, and the same failure accounting. If you do not yet have that paired evaluation, the honest decision is hold, not “probably improved.”

Bottom line

Fine-tuning is not the hard part when your SFT dataset is quietly losing records, repeating artifacts, leaking labels, omitting negatives, cutting off targets, or inventing source citations. Our internal audit-data work found all of those failure modes before it had a verified trained-model result, and that is the win this article can actually support.

Run prompt + RAG first. Then make the dataset pass dedup, labels, negative coverage, truncation, leakage, localization, and held-out-repository gates. Until leakage is zero, recall is at least 70%, negatives meet your target, and truncation is below 10%, keep the GPU-hours in your pocket.

---

More on this decision, three ways to look at it:

Sources

• GBrain: projects/bastet-pipeline, eight-stage pipeline, v0 dataset receipts, held-out repositories, dedup failure, tool-call failure/root cause, regression pass, throughput, and input constraints; facts recorded July 2026.
• GBrain: projects-qwen36-bastet-lora, synthetic and curated dataset lineage, negative ratio, duplicate-artifact receipt, grounding-pilot halt, candidate-recall gate, leakage constraints, and prompt export; facts recorded July 2026.
• GBrain: qwen36-bastet-lora-data, 137/39 train-validation split, over-4K rates, 32% train truncation, high-severity bias, and unmapped labels; facts recorded July 2026.
• Recovery-plan path checked but not available: /Users/kit/projects/qwen36-bastet-lora/docs/2026-06-05-production-sft-dataset-quality-recovery-plan.md (not found on 2026-08-21). The article relies on the GBrain receipts above.