Total FLOPs
6.30e+24
Math & numbers · Frontier
AI compute
AI compute
calculator
Plan a training run the way labs sketch one: parameters and tokens in, total FLOPs, GPU-hours, wall-clock time and cost out — first-order estimates, computed live in the page.
6NDtraining-run estimator
B
B
TFLOPS
%
$/GPU-hr
6 × 70B × 15,000B tokens · 1,024 GPUs
$11.06M
GPU-hours
4,423,660
Wall clock
180 days
FLOPs ≈ 6 · N · D · · · GPU-hours = FLOPs / (peak × MFU × 3600)
The arithmetic every training run starts with
Before any cluster is booked, a language-model training run is sketched with one line of arithmetic: total compute ≈ 6·N·D, where N is parameter count and D is tokens seen. The 6 covers the forward pass (≈2ND) plus the backward pass (≈4ND). This tool takes that number and converts it into the quantities that actually get budgeted — GPU-hours, wall-clock days at your cluster size, and dollars at your hourly rate.
The chain
FLOPs = 6·N·D
GPU-hours = FLOPs ÷ (peak × MFU × 3,600)
days = GPU-hours ÷ GPUs ÷ 24 · · · cost = GPU-hours × rate
GPU-hours = FLOPs ÷ (peak × MFU × 3,600)
days = GPU-hours ÷ GPUs ÷ 24 · · · cost = GPU-hours × rate
MFU — model FLOPs utilisation — is the honesty factor. No real run sustains a GPU's datasheet peak; well-engineered large-scale training typically lands in the 30–50% range once communication, memory stalls and stragglers are paid for. The default of 40% is a sensible planning figure; drop it to 30% for a pessimistic bound or push toward 50% for a highly tuned stack.
Worked default
70B params × 15T tokens → 6.3×10²⁴ FLOPs
at 989 TFLOPS · 40% MFU · 1,024 GPUs · $2.50/hr
at 989 TFLOPS · 40% MFU · 1,024 GPUs · $2.50/hr
Honest limits: 6ND is a first-order rule. It omits attention's context-length term, activation recomputation, embedding maths and any restarts — real bills usually land somewhat above it, and Chinchilla-style scaling questions of what N and D should be are a separate decision this tool deliberately stays out of. Treat the output as the right order of magnitude and a fair comparison between plans, not a quote. The inference line of thinking — roughly 2N FLOPs per generated token — lives in the FAQ below.
Common questions
AI compute FAQ
Each token in training costs about 2N FLOPs on the forward pass and about 4N on the backward pass, so ≈6N per token, times D tokens. It is the standard first-order estimate used across scaling-law work; real runs add overheads on top, but 6ND sets the order of magnitude every plan starts from.
Model FLOPs utilisation: the fraction of the GPUs' theoretical peak your training loop actually sustains. Communication, memory bandwidth and stragglers all eat into it. Large well-tuned runs typically report 30–50%; 40% is a reasonable planning default, and using 100% is the classic way to underestimate a budget by 2–3×.
The dense (non-sparse) throughput for the precision you train in — for example an H100 SXM is roughly 989 TFLOPS at BF16 dense. Marketing sheets often quote the 2× sparsity figure; training does not get that, so use the dense number or your estimate will be optimistic by half.
A decent rule of thumb is ≈2N FLOPs per generated token. Divide your GPU's effective throughput (peak × an inference MFU, often 20–40%) by 2N for tokens per second per GPU, then scale by fleet size and utilisation. Batch size, KV-cache memory and context length move the real number substantially.
Because 6ND is the floor, not the invoice: attention adds a context-length term, activation checkpointing recomputes work, evaluation and checkpoint saves burn time, and few runs finish without a restart. Teams commonly pad the estimate by 20–50%. The tool's job is the defensible baseline those pads are added to.
Keep going