LLM Memory Calculator

VRAM Calculator for LLMs

Estimate how much GPU memory a large language model needs to run or train. See the split between weights, KV cache, activations and optimizer states for inference, full fine-tuning and LoRA/QLoRA.

Formulas Shown
GPUs That Fit
10 Model Presets
FP32 to 2-bit
Calculate VRAM Now
No Sign-up
Instant Results
Free to Use
Use Case

Model Configuration

| |
Weight Precision / Quantization
FP16: 2 bytes per weight (50% smaller than FP32)
2048
128 32K
1
1 64

Optimization Techniques

Complete Guide to VRAM Requirements for Large Language Models

Estimate the GPU memory needed to run or fine-tune a large language model before you buy hardware or rent a cloud instance. The calculator above uses the standard memory accounting for transformer models (weights, KV cache, activations, gradients and optimizer states) and shows each part, so you can see what drives the total. Results are estimates: real usage depends on your framework, runtime settings and driver, so leave some headroom.

Understanding VRAM Requirements for AI Models

VRAM (Video RAM) is the dedicated memory on your graphics card that stores model weights, activations, and intermediate computations during AI inference or training. Unlike system RAM, VRAM sits next to the GPU cores and has much higher bandwidth, which matrix-heavy neural networks need. If a model does not fit in VRAM, it fails with an out-of-memory error or has to offload layers to slower system RAM. Running a 7 billion parameter model in FP16 precision needs about 14 GB of VRAM just for the model weights, plus memory for the KV cache, which grows with context length and batch size. Understanding these requirements prevents expensive hardware mistakes and deployment failures. If you are also building the PC around the GPU, the Bottleneck Calculator checks CPU/GPU balance.

VRAM Calculation Components:

Memory Breakdown
  • • Model Weights: Parameters × precision bits ÷ 8 (plus ~20% runtime overhead, shown separately)
  • • KV Cache: 2 × layers × KV heads × head size × tokens × batch × 2 bytes. Grouped-query attention (fewer KV heads) shrinks it a lot.
  • • Activations: Intermediate layer outputs. Small for inference; large for training, where they are kept for the backward pass.
  • • Optimizer States: Adam optimizer keeps two FP32 moments (8 bytes per parameter), plus a 4-byte FP32 master copy of the weights in mixed-precision training
  • • Gradients: One value per trainable parameter, usually in the same precision as the weights
Quick Rules of Thumb
  • • Inference FP16/BF16: ~2 GB per billion parameters for weights, plus KV cache and overhead
  • • Inference INT8: ~1 GB per billion parameters for weights (INT8 quantization)
  • • Inference 4-bit: ~0.5 GB per billion parameters for weights (GPTQ paper)
  • • Full fine-tuning (mixed precision, Adam): ~16 GB per billion parameters before activations
  • • LoRA / QLoRA: roughly the frozen weights at their precision, plus activations and a small amount for the adapters

Four Use Case Modes:

⚡ Inference
Running a model to generate text. Memory is mostly weights plus the KV cache
🎓 Training
Training every weight with optimizer states and gradients (same memory math as full fine-tuning)
🔧 Fine-tuning
Adapting pre-trained models for specific tasks (HuggingFace guide)
🎯 LoRA
Parameter-efficient training with low-rank adapters

Why Estimate VRAM First:

  • ✓
    Hardware Planning: Check whether a GPU has enough VRAM before you buy
  • ✓
    Avoid OOM Errors: Spot configurations that are likely to run out of memory
  • ✓
    Cost Planning: Narrow down which cloud GPU sizes are worth trying
  • ✓
    Batch Size Planning: Maximize throughput within memory constraints
  • ✓
    Quantization Decisions: Balance quality vs memory tradeoffs intelligently

Quantization Guide: Memory vs Quality Tradeoffs

FP32 (32-bit Floating Point)

Full single precision (IEEE 754 standard). Twice the memory of FP16 and rarely needed for inference

Memory per Param: 4 bytes
7B Weights Only: ~28 GB
Use Case: Rarely used for inference

FP16 / BF16 (16-bit Floating Point)

The usual precision for running and training LLMs (see PyTorch AMP). Half the memory of FP32 with essentially the same output quality

Memory per Param: 2 bytes
7B Weights Only: ~14 GB
Quality Loss: Usually negligible

INT8 (8-bit Integer)

Weights stored as 8-bit integers using post-training quantization. 75% smaller than FP32; quality loss is usually small

Memory per Param: 1 byte
7B Weights Only: ~7 GB
Quality Loss: Usually small

INT4 / 4-bit (GPTQ, AWQ)

4-bit methods such as GPTQ algorithm, AWQ and GGUF Q4 make large models fit on consumer GPUs. Quality loss depends on the model and method

Memory per Param: 0.5 bytes
7B Weights Only: ~3.5 GB
Quality Loss: Model-dependent

2-bit (Extreme Quantization)

Experimental ultra-low precision for resource-constrained devices

Memory per Param: 0.25 bytes
7B Weights Only: ~1.75 GB
Quality Loss: Significant

VRAM Requirements for Popular LLMs

These figures come from the calculator above. Inference: 4,096-token context, batch size 1, flash attention on. Fine-tuning: 2,048 tokens, batch size 1, gradient checkpointing on, LoRA rank 16. They include the runtime overhead, so they are higher than the weights-only sizes you often see quoted.

Llama 2 7B

Meta's Llama 2 7B uses standard multi-head attention (32 KV heads), so its KV cache is large for its size: about 0.5 MB per token at FP16.

FP16 Inference
19.0 GB
24 GB cards (RTX 3090, RTX 4090)
INT8 Inference
10.9 GB
12 GB cards (RTX 3060 12GB)
4-bit Inference
6.9 GB
8 GB cards
QLoRA (4-bit base)
5.5 GB
8 GB cards

Llama 2 70B

Uses grouped-query attention with 8 KV heads, so its KV cache per token (~0.33 MB at FP16) is smaller than Llama 2 7B's. The weights are what make it large.

FP16 Inference
168 GB
3× 80 GB GPUs, or 2× H200
INT8 Inference
85 GB
2× 80 GB GPUs, or 1× H200
4-bit Inference
44 GB
One 48 GB GPU (RTX 6000 Ada, L40S)
QLoRA (4-bit base)
46 GB
One 48 GB GPU

Mistral 7B

Mistral 7B has slightly more parameters than Llama 2 7B but only 8 KV heads, so long contexts cost about a quarter of the KV cache memory.

FP16 Inference
18.6 GB
24 GB cards
INT8 Inference
10.0 GB
12 GB cards
4-bit Inference
5.7 GB
8 GB cards
QLoRA (4-bit base)
5.8 GB
8 GB cards

Mixtral 8x7B

A mixture-of-experts model with 46.7B total parameters. Only about 13B are used per token, but every expert must be in memory, so VRAM is sized for the full 46.7B.

FP16 Inference
113 GB
2× 80 GB GPUs
INT8 Inference
57 GB
One 80 GB GPU
4-bit Inference
29 GB
One 32 GB GPU (RTX 5090) or 2× 24 GB
QLoRA (4-bit base)
30 GB
One 32 GB+ GPU

GPT-3 175B (size reference)

GPT-3 weights are not publicly released, but its published architecture is a useful reference for what a 175B dense model needs.

FP16 Inference
441 GB
6× 80 GB GPUs or more
INT8 Inference
231 GB
3× 80 GB GPUs
4-bit Inference
126 GB
2× 80 GB GPUs, or 1× H200
Full Fine-tuning
~2.9 TB
Multi-node cluster with sharding

Phi-2 2.7B

Microsoft's Phi-2 is small enough to run on most recent gaming GPUs, and even full fine-tuning fits on a single 48 GB card.

FP16 Inference
8.5 GB
12 GB cards
INT8 Inference
5.3 GB
6–8 GB cards
4-bit Inference
3.7 GB
4–6 GB cards
Full Fine-tuning
45 GB
One 48 GB GPU

GPU Recommendations by VRAM Capacity

Consumer GPUs (8-16 GB VRAM)

Enough for 7B–13B models with quantization. Check exact specs on NVIDIA GeForce and AMD Radeon

NVIDIA RTX 3060
• VRAM: 12 GB GDDR6
• Fits: 7B models at INT8 or 4-bit, Phi-2 at FP16
AMD RX 7600 XT
• VRAM: 16 GB GDDR6
• Fits: 7B at INT8, 13B at 4-bit (uses ROCm rather than CUDA)
NVIDIA RTX 4060 Ti
• VRAM: 16 GB GDDR6
• Fits: 7B at INT8, 7B QLoRA; FP16 7B is too tight

Enthusiast GPUs (20-24 GB VRAM)

The common choice for running 7B models at FP16 and 13B–34B models with quantization

NVIDIA RTX 3090
• VRAM: 24 GB GDDR6X
• Fits: 7B at FP16, 13B at INT8, 34B at 4-bit, 7B–13B QLoRA
NVIDIA RTX 4090
• VRAM: 24 GB GDDR6X
• Fits: Same model sizes as the RTX 3090, with faster compute
NVIDIA RTX A5000
• VRAM: 24 GB GDDR6
• Fits: Same model sizes as other 24 GB cards

Professional GPUs (40-48 GB VRAM)

Large enough for 70B models at 4-bit and QLoRA fine-tuning of 70B models

NVIDIA A40
• VRAM: 48 GB GDDR6
• Fits: 70B at 4-bit, Mixtral 8x7B at 4-bit, 34B at INT8
NVIDIA L40S
• VRAM: 48 GB GDDR6
• Fits: Same model sizes as other 48 GB cards
NVIDIA RTX 6000 Ada
• VRAM: 48 GB GDDR6
• Fits: Same model sizes as other 48 GB cards

Enterprise GPUs (80+ GB VRAM)

Top-tier NVIDIA data center accelerators for the largest models and production scale deployments

NVIDIA A100 80GB
• VRAM: 80 GB HBM2e
• Fits: 13B at FP16, 34B and Mixtral at INT8; 70B at INT8 is just over 80 GB
NVIDIA H100 80GB
• VRAM: 80 GB HBM3
• Fits: Same model sizes as the A100 80GB, with faster compute
NVIDIA H100 NVL
• VRAM: 94 GB HBM3
• Fits: 70B at INT8 with room for context

Memory Optimization Techniques

1. Flash Attention - Attention Memory

Avoids storing the full attention score matrix by computing attention in tiles with fused attention kernels. It does not shrink the KV cache.

Standard Attention:
• Materializes a heads × seq × seq score matrix
• Grows with the square of context length
Memory intensive
Flash Attention:
• Fused operations, less materialization
• Attention memory grows linearly with context
Biggest gains at long context

2. Gradient Checkpointing - Training Memory Reduction

Store only each layer's input and recompute the rest during the backward pass. Costs roughly one extra forward pass of compute

Without Checkpointing:
• Store all layer activations
• Memory scales with model depth
High memory usage
With Checkpointing:
• Keep only each layer's input
• Recompute others during backprop
Llama 2 7B, 2K tokens: ~9.1 GB → ~0.8 GB

3. LoRA - Parameter Efficient Fine-Tuning

Freeze the base model and train small low-rank adapters as detailed in Microsoft's research on LoRA implementation. QLoRA goes further by keeping the frozen base model in 4-bit.

Full Fine-tuning:
• Update all model parameters
• Requires optimizer states for billions of params
Llama 2 7B (BF16): ~111 GB
Multi-GPU territory
LoRA Fine-tuning:
• Freeze base model, train adapters
• Trainable params: <1% of total
Llama 2 7B: ~17.5 GB (BF16 base), ~5.5 GB (4-bit QLoRA)
Fits on one consumer GPU

4. Model Parallelism - Split Across Multiple GPUs

Distribute model layers using PyTorch model parallelism across GPUs when one GPU is not enough. Also see HuggingFace multi-GPU guide

Tensor Parallelism: Split individual layer computations across GPUs
Pipeline Parallelism: Place different layers on different GPUs
Example: Llama 2 70B at FP16 (~168 GB estimated) split across 3× 80 GB GPUs
Splitting adds some communication buffers, so plan a little extra per GPU

5. Mixed Precision Training

Use Automatic Mixed Precision (AMP) for FP16 computation while keeping FP32 master weights per NVIDIA's research

• Weights & Gradients: FP16/BF16 (2 bytes per parameter each)
• Master Weights: Kept in FP32 for numerical stability
• Optimizer: Adam moments plus master weights take 12 bytes per parameter, so weight-related memory is about the same as FP32 (16 bytes per parameter); the saving is in activations
• Speed: Usually faster on GPUs with Tensor Cores
Standard practice for modern training

6. Sequence Length Optimization - Context Window Management

KV cache grows linearly with context length. Estimated totals with flash attention, batch 1:

2K Context:
Llama 2 7B FP16
~17.8 GB
8K Context:
Llama 2 7B FP16
~21.4 GB
32K Context:
Llama 2 7B FP16
~35.7 GB
Use shortest context needed for your task

Common VRAM Calculation Mistakes to Avoid

❌ Forgetting KV Cache Memory

Many calculate only model weights, but KV cache grows with context and batch size. For Llama 2 7B at FP16 it is about 2 GB at 4K tokens but about 17 GB at 32K tokens, more than the weights themselves.

❌ Ignoring CUDA Overhead

PyTorch and CUDA need memory for the CUDA context, kernels and the caching allocator. This calculator adds 0.5 GB plus 20% of the weight size as a rule of thumb.

❌ Confusing Training vs Inference

Full fine-tuning with Adam needs about 16 bytes per parameter before activations, versus 2 bytes for FP16 inference weights. A GPU that runs a model may be far too small to fully fine-tune it; LoRA or QLoRA is often the practical option.

❌ Not Accounting for Batch Size

Batch size multiplies the KV cache and activations, not the weights. Llama 2 7B at FP16 with 4K context goes from ~19 GB at batch 1 to ~36 GB at batch 8.

❌ Overestimating Quantization Quality

4-bit quantization saves a lot of memory, but the quality cost depends on the model and method, and grows at 3-bit and 2-bit. Test on your own prompts before committing.

❌ Mixing Up MoE Architecture Memory

Mixtral 8x7B has 46.7B total parameters (see the Mixtral paper). Only about 13B are active per token, but all experts must be loaded, so MoE reduces compute, not VRAM.

Related Hardware Planning Tools

Other tools that help when planning an LLM setup.

PSU Calculator

Estimate the power supply wattage for a workstation with one or more high-power GPUs.

Bottleneck Calculator

Check whether your CPU and GPU are a balanced pairing when building a PC around your GPU.

AI Token Counter

Count the tokens in your prompts and documents to pick a realistic context length for the calculator.