Can you train MiniMind from scratch in two hours on an RTX 3090?

Back
Kevin Riedl

6 min read · 16 November 2026
Last reviewed

Next
Made on your device, with no Instagram connection. We copy the post link for Instagram’s Link sticker.

MiniMind puts the part you never see in an LLM API response on your own machine. Start with random weights, teach the model to predict the next token, then teach it which side of a conversation to answer. You can save the weights after each stage and ask the two checkpoints the same question.

What does MiniMind’s two-hour estimate cover?

For the dense 64M MiniMind-3 Zero recipe, the authors estimate 2.31 hours on one 24 GB RTX 3090 for one pass over each of two prepared datasets. That is about 2 hours 19 minutes. The README’s opening footnote describes a separate measurement of around two hours for one supervised fine-tuning epoch alone. The authors give no reconciliation. Use their itemized figures when planning the two-stage run. The project’s original timing note

MiniMind-3 Zero on a single RTX 3090, using the project’s estimated rental rate of CNY 1.30 per hour
StageGPU timeRental estimate
Pretrain on the mini text corpus1.21 hCNY 1.57
Supervised fine-tuning on mini dialogues1.10 hCNY 1.43
Both stages, one epoch each2.31 hCNY 3.00

An epoch is one pass over a dataset. Downloading the files and setting up PyTorch take additional time. The table gives tool-call training and reinforcement learning their own entries. If your goal is to see a model go from random weights to a first conversation, you can stop after the second row. MiniMind’s stage-by-stage GPU estimates

How do you run the one-epoch recipe from random weights?

Use the two _mini.jsonl files, the bundled tokenizer and a fresh checkout. Pass --epochs 1 to both trainers. Their defaults are two epochs each, so launching the scripts gives you a different experiment. The pretrainer starts with random weights when --from_weight none is set. SFT loads the checkpoint named pretrain. Pretraining arguments · SFT arguments

Clone https://github.com/jingyaogong/minimind.git and open the repository. In a fresh Python environment with a CUDA-enabled PyTorch build for your GPU, run the setup commands below. The CUDA check should print True before you budget for a GPU run. The trainers implement the optimization loop in PyTorch. They also use Hugging Face Transformers for the tokenizer and Datasets to read training data. Pinned Python dependencies

git checkout 233d00fa068ec2b8c0d4f07b703934d6391a5a73
python -m pip install -r requirements.txt
python -c "import torch; print(torch.cuda.is_available())"

Put pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl in the repository’s dataset/ directory. The project lists them at about 1.2 GB and 1.6 GB. The files without _mini are the larger, 10 GB and 14 GB corpora. Cleaning and deduplication belong to corpus preparation. Zero starts with prepared files and an existing vocabulary, so you can spend the session watching the weights change. MiniMind training data downloads

cd trainer
python train_pretrain.py --epochs 1 --from_weight none
python train_full_sft.py --epochs 1 --from_weight pretrain
cd ..
python eval_llm.py --weight pretrain --max_new_tokens 80
python eval_llm.py --weight full_sft --max_new_tokens 80

The default outputs are out/pretrain_768.pth and out/full_sft_768.pth. Exit the first interactive evaluation before starting the second. Try a short passage for the base model to continue, then ask the chat model to follow a simple instruction. Save the prompt and both outputs. The evaluation script treats the base checkpoint as a text completer and formats chat turns for the tuned one. Checkpoint loading and evaluation prompts

We checked the handoff on CPU using the pinned scripts, one layer of width 64 and two synthetic examples per stage. Both training commands wrote checkpoints. The evaluator loaded each and generated four tokens. This verifies that the commands connect end to end. The 64M GPU timing above comes from the project.

Why can a valid training example teach MiniMind nothing?

A dialogue can be valid JSON and still supply zero answer tokens for training. MiniMind’s SFT loader renders the conversation, truncates it, then marks the assistant’s reply as the part to learn. If a long user message fills all 768 positions before the assistant marker appears, the entire retained example is ignored by the loss function. Chat formatting, truncation and label masking

We ran the project’s tokenizer and SFT data loader offline on two synthetic exchanges with the same system message. In both, the assistant answers OK. In the short exchange the label generator kept four targets, including the end marker and a newline. After we repeated status 500 times in the user message, it kept zero. The longer prompt had pushed the entire answer past the cutoff.

Download the MiniMind SFT label check. From the pinned MiniMind repository root, run python /path/to/minimind-sft-target-check.py in the same environment. No model weights are needed. Then try the same check on your longest support transcripts. Count the labels other than -100 after truncation. If you find a zero, shorten or split that conversation so the answer survives, then inspect the decoded target text. A falling average loss can hide a batch of examples that contributed nothing.

Which trainer should you try after supervised fine-tuning?

Choose the next trainer by the evidence you can supply. A folder of good answers suits supervised tuning. Pairs in which a reviewer prefers one answer suit DPO. A task with an outcome you can score, such as choosing the right calculator arguments, gives you a reason to try reinforcement learning. The repository exposes these as separate scripts. MiniMind training implementations

Pick a follow-up experiment by what you want the model to learn
MethodWhat you supply or inspect
LoRATrain small adapter weights on examples from your domain while keeping the base weights fixed.
DPOSupply a preferred and a rejected answer for the same prompt. The pairing encodes the preference.
PPO and GRPOGenerate answers and score them. PPO learns a value estimate. GRPO compares rewards among answers sampled for the same prompt.
Tool use and agentic RLTool-call examples teach the format. In the agent trainer, a tool returns a result and the model can use it on a later turn. Its default loss is CISPO, with GRPO also available.
DistillationTeacher-written answers can become SFT examples. The separate distillation trainer can also learn from a teacher’s distribution over next tokens.

Ask the model to add two prices, then inspect the calculator call. The tool takes one string argument, expression. Change the prices and try again. You can check the operator, the two numbers in the expression and the returned sum separately. Agent trainer and built-in tools

When should you train MiniMind and when should you start from a checkpoint?

I’d train MiniMind from scratch to learn what a tokenizer, a loss mask and an optimizer each contribute. If the team’s goal is to classify incoming tickets or match a product-specific writing style, I’d first test an existing model against labeled examples. That gives the team a baseline before it invests in a new corpus.

Dense MiniMind-3 has eight decoder layers and a hidden width of 768. Counting the model tensors on PyTorch’s meta device gave us 63,912,192 parameters, which the project rounds to 64 million. The largest original GPT-3 has 175 billion, roughly 2,700 times as many. That ratio compares parameter counts. To decide whether the small model can handle a ticket, test it on tickets. MiniMind-3 model configuration · Original GPT-3 paper

Our Gemma fine-tuning walkthrough starts with pretrained weights. Our analysis of Airbnb’s small-model experiment looks at the separate decision of whether a specialist model earns its place in production.

MiniMind training questions

Do I need the eight GPUs listed in the project’s setup?
The maintainers list an eight-card workstation as their reference environment. Their Zero estimate uses one RTX 3090 with 24 GB of memory. For a different GPU, record the actual batch size, sequence length and time per step before estimating a full epoch.
Do I need to train a tokenizer first?
The Zero recipe uses the tokenizer in the repository. Training from scratch here refers to the language-model weights. Keeping the supplied tokenizer lets you compare checkpoints against the same vocabulary and chat markers.
Can I try MiniMind before training it?
Yes. The maintainers also publish pretrained weights. Loading them is a quick way to inspect the model’s answers. If you want to see what pretraining itself contributes, keep a separate run that begins from random weights.
Does the two-hour recipe produce the published full model?
Zero is the quick dialogue recipe using the two mini files. The project also describes larger corpora, longer runs and additional training objectives. Record the exact starting weights, datasets and stages when comparing a downloaded checkpoint with one you trained.
Where are the vision, multimodal and diffusion versions?
MiniMind-V is the vision-language project. MiniMind-O is the separate Omni multimodal project. The maintainers describe the diffusion language model as an experimental extension. Each has a different training setup from the two-stage text exercise.

MiniMind-V vision project · MiniMind-O multimodal project · Experimental diffusion language model

Make the pilot decision with evidence

Bring one workflow, representative examples and the outcome you need to accept. Define an evaluation set, failure limits and the evidence required before scaling an existing AI system.

Relevant service:

Inbox, without the noise

Follow the work that matters to you

Get a short email when we publish something new. Follow the whole blog or only the problems you care about.

What would you like to receive?
Choose your topics

Free, double opt-in, no tracking pixels.

Back
Kevin Riedl

6 min read · 16 November 2026
Last reviewed

Next

Get the next AI and agents field note

One concise email when we publish. No tracking pixels, and no inbox filler.

Free, double opt-in, no tracking pixels.