What Actually Fits on a 12 GB GPU

Sizing rules learned by hitting them: params times two bytes, why 4-bit quantization does not shrink the download, and the 22 GB text encoder hiding inside a “1.3B” video model.
Author

Benedict Thekkel

Published

July 12, 2026

Almost everything written about running models assumes you rented the GPU. This is the other case: one RTX 3060 with 12 GB of VRAM, in a container with 20 GB of system RAM, running every task in DL_tasks.

Working through 53 task notebooks on that box taught me that the question “will this model fit” has three different answers, and the one everybody quotes is the least likely to stop you.

Three budgets, not one

Budget Size here What blows it
VRAM 12 GB Model weights plus activations
System RAM 20 GB, minus ~2 GB for OS and JupyterLab CPU offload, and un-freed intermediates
Download / disk ~15-40 GB free Repo size, which is not param count

The habit is to check the first one and assume the rest follow. They do not. Most of the times I got stuck, VRAM was fine and one of the other two was the problem.

The easy rule: params times two bytes

In fp16 every parameter is 2 bytes, so weights alone are params x 2 bytes. Activations, KV cache and CUDA context sit on top, so in practice budget the weights at roughly two thirds of VRAM and leave the rest.

On 12 GB that means anything past about 3B parameters in fp16 does not fit, and you either pick a smaller checkpoint or quantize.

The rule that actually bites: download size is not param count

This is the one that cost me the most time, and it has nothing to do with whether the model runs.

Generative video pipelines ship a text encoder alongside the model doing the interesting work, and the encoder is frequently stored in fp32 while everything else is fp16. The encoder then dwarfs the thing it serves.

Wan 2.1 is marketed as a 1.3B video model. Its UMT5-XXL text encoder is 5.6B parameters stored in fp32, which is 22.7 GB, roughly 90% of a 29 GB download. The DiT you actually came for is a rounding error next to it.

Two things that do not save you

load_in_4bit does not shrink the download. bitsandbytes quantizes after the full fp32 weights have been fetched and read. It halves what you need at run time, which is real and useful, but the disk and the wait are unchanged. To download less you need a repo that stores fp16 or bf16.

variant="fp16" helps, but only where an fp16 variant exists. Image pipelines like SD 1.5 and SDXL publish 40-75 GB repos stuffed with format duplicates, and from_pretrained(..., variant="fp16") correctly fetches just the fp16 shards, a few GB. A text encoder that is only published in fp32 has no fp16 variant to select, so it arrives in full regardless.

Put those together and you get the failure mode: you pick a small model, you pass every memory-saving flag you know, and you still pull 29 GB.

Check before you commit a runnable cell

Metadata is free. There is no reason to discover a 29 GB download by starting it.

from huggingface_hub import HfApi

info = HfApi().repo_info(repo, files_metadata=True)
gb = sum((s.size or 0) for s in info.siblings) / 1e9

Then look at where the bytes are. A text_encoder/ folder of 20 GB is the tell, and it tells you whether the size is avoidable (fp16 variant available) or not (fp32 only).

The rule I settled on: a runnable cell downloads at most about 8 GB. Anything bigger goes in a comparison table with prose, or behind a RUN_HEAVY = False flag that prints what it would cost. Nobody opening a notebook should trigger a 29 GB pull by pressing shift-enter.

Datasets have the same trap, worse

load_dataset("repo", split="val") downloads every data file in the repo, then slices. Asking for a 0.9 GB validation split can pull 24 GB of training shards and fill the disk before it gives you anything.

val = load_dataset(
    "parquet",
    data_files={"val": "hf://datasets/<repo>/data/val-*.parquet"},
    split="val", cache_dir=HF_CACHE,
)

Also worth knowing: raw Hub blobs land in ~/.cache/huggingface/hub regardless of cache_dir, which only controls the built Arrow cache. So a runaway download escapes whatever directory you thought you had confined it to.

The working checklist

flowchart TD
  A[Candidate model] --> B{Params x 2 bytes<br/>under ~8 GB?}
  B -- no --> Q[Quantize or pick smaller]
  B -- yes --> C{repo_info size<br/>under ~8 GB?}
  C -- no --> D{Big files fp32-only?}
  D -- yes --> T[Table + prose, not a runnable cell]
  D -- no --> V[variant='fp16']
  C -- yes --> R[Runnable cell]
  V --> R

Takeaway

The parameter count tells you whether a model will run. It tells you almost nothing about whether you can get it. On a small box those are separate questions and the second one is the one that ruins an afternoon.

Three habits fixed it for me: size the weights before writing the cell, call repo_info before writing a download, and treat anything over ~8 GB as documentation rather than something a reader can execute. The full task-by-task working is in DL_tasks, and the memory side of the same problem is in the post on malloc_trim.

Back to top