In this piece
Can you train MiniMind from scratch in two hours on an RTX 3090?
MiniMind puts the part you never see in an LLM API response on your own machine. Start with random weights, teach the model to predict the next token, then teach it which side of a conversation to answer. You can save the weights after each stage and ask the two checkpoints the same question.
What does MiniMind’s two-hour estimate cover?
For the dense 64M MiniMind-3 Zero recipe, the authors estimate 2.31 hours on one 24 GB RTX 3090 for one pass over each of two prepared datasets. That is about 2 hours 19 minutes. The README’s opening footnote describes a separate measurement of around two hours for one supervised fine-tuning epoch alone. The authors give no reconciliation. Use their itemized figures when planning the two-stage run. The project’s original timing note
| Stage | GPU time | Rental estimate |
|---|---|---|
| Pretrain on the mini text corpus | 1.21 h | CNY 1.57 |
| Supervised fine-tuning on mini dialogues | 1.10 h | CNY 1.43 |
| Both stages, one epoch each | 2.31 h | CNY 3.00 |
An epoch is one pass over a dataset. Downloading the files and setting up PyTorch take additional time. The table gives tool-call training and reinforcement learning their own entries. If your goal is to see a model go from random weights to a first conversation, you can stop after the second row. MiniMind’s stage-by-stage GPU estimates
How do you run the one-epoch recipe from random weights?
Use the two _mini.jsonl files, the bundled tokenizer and a fresh checkout. Pass --epochs 1 to both trainers. Their defaults are two epochs each, so launching the scripts gives you a different experiment. The pretrainer starts with random weights when --from_weight none is set. SFT loads the checkpoint named pretrain. Pretraining arguments · SFT arguments
Clone https://github.com/jingyaogong/minimind.git and open the repository. In a fresh Python environment with a CUDA-enabled PyTorch build for your GPU, run the setup commands below. The CUDA check should print True before you budget for a GPU run. The trainers implement the optimization loop in PyTorch. They also use Hugging Face Transformers for the tokenizer and Datasets to read training data. Pinned Python dependencies
git checkout 233d00fa068ec2b8c0d4f07b703934d6391a5a73
python -m pip install -r requirements.txt
python -c "import torch; print(torch.cuda.is_available())"Put pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl in the repository’s dataset/ directory. The project lists them at about 1.2 GB and 1.6 GB. The files without _mini are the larger, 10 GB and 14 GB corpora. Cleaning and deduplication belong to corpus preparation. Zero starts with prepared files and an existing vocabulary, so you can spend the session watching the weights change. MiniMind training data downloads
cd trainer
python train_pretrain.py --epochs 1 --from_weight none
python train_full_sft.py --epochs 1 --from_weight pretrain
cd ..
python eval_llm.py --weight pretrain --max_new_tokens 80
python eval_llm.py --weight full_sft --max_new_tokens 80The default outputs are out/pretrain_768.pth and out/full_sft_768.pth. Exit the first interactive evaluation before starting the second. Try a short passage for the base model to continue, then ask the chat model to follow a simple instruction. Save the prompt and both outputs. The evaluation script treats the base checkpoint as a text completer and formats chat turns for the tuned one. Checkpoint loading and evaluation prompts
We checked the handoff on CPU using the pinned scripts, one layer of width 64 and two synthetic examples per stage. Both training commands wrote checkpoints. The evaluator loaded each and generated four tokens. This verifies that the commands connect end to end. The 64M GPU timing above comes from the project.
Why can a valid training example teach MiniMind nothing?
A dialogue can be valid JSON and still supply zero answer tokens for training. MiniMind’s SFT loader renders the conversation, truncates it, then marks the assistant’s reply as the part to learn. If a long user message fills all 768 positions before the assistant marker appears, the entire retained example is ignored by the loss function. Chat formatting, truncation and label masking
We ran the project’s tokenizer and SFT data loader offline on two synthetic exchanges with the same system message. In both, the assistant answers OK. In the short exchange the label generator kept four targets, including the end marker and a newline. After we repeated status 500 times in the user message, it kept zero. The longer prompt had pushed the entire answer past the cutoff.
Download the MiniMind SFT label check. From the pinned MiniMind repository root, run python /path/to/minimind-sft-target-check.py in the same environment. No model weights are needed. Then try the same check on your longest support transcripts. Count the labels other than -100 after truncation. If you find a zero, shorten or split that conversation so the answer survives, then inspect the decoded target text. A falling average loss can hide a batch of examples that contributed nothing.
Which trainer should you try after supervised fine-tuning?
Choose the next trainer by the evidence you can supply. A folder of good answers suits supervised tuning. Pairs in which a reviewer prefers one answer suit DPO. A task with an outcome you can score, such as choosing the right calculator arguments, gives you a reason to try reinforcement learning. The repository exposes these as separate scripts. MiniMind training implementations
| Method | What you supply or inspect |
|---|---|
| LoRA | Train small adapter weights on examples from your domain while keeping the base weights fixed. |
| DPO | Supply a preferred and a rejected answer for the same prompt. The pairing encodes the preference. |
| PPO and GRPO | Generate answers and score them. PPO learns a value estimate. GRPO compares rewards among answers sampled for the same prompt. |
| Tool use and agentic RL | Tool-call examples teach the format. In the agent trainer, a tool returns a result and the model can use it on a later turn. Its default loss is CISPO, with GRPO also available. |
| Distillation | Teacher-written answers can become SFT examples. The separate distillation trainer can also learn from a teacher’s distribution over next tokens. |
Ask the model to add two prices, then inspect the calculator call. The tool takes one string argument, expression. Change the prices and try again. You can check the operator, the two numbers in the expression and the returned sum separately. Agent trainer and built-in tools
When should you train MiniMind and when should you start from a checkpoint?
I’d train MiniMind from scratch to learn what a tokenizer, a loss mask and an optimizer each contribute. If the team’s goal is to classify incoming tickets or match a product-specific writing style, I’d first test an existing model against labeled examples. That gives the team a baseline before it invests in a new corpus.
Dense MiniMind-3 has eight decoder layers and a hidden width of 768. Counting the model tensors on PyTorch’s meta device gave us 63,912,192 parameters, which the project rounds to 64 million. The largest original GPT-3 has 175 billion, roughly 2,700 times as many. That ratio compares parameter counts. To decide whether the small model can handle a ticket, test it on tickets. MiniMind-3 model configuration · Original GPT-3 paper
Our Gemma fine-tuning walkthrough starts with pretrained weights. Our analysis of Airbnb’s small-model experiment looks at the separate decision of whether a specialist model earns its place in production.
MiniMind training questions
Do I need the eight GPUs listed in the project’s setup?
Do I need to train a tokenizer first?
Can I try MiniMind before training it?
Does the two-hour recipe produce the published full model?
Where are the vision, multimodal and diffusion versions?
MiniMind-V vision project · MiniMind-O multimodal project · Experimental diffusion language model
