# NVIDIA GB200 NVL72 rack: $3M upfront vs $1.2M cloud fees at 10k tokens/sec — buy or rent?

Jonas Keller · November 11, 2026

> Buy a GB200 NVL72 rack only if your trailing-90-day average exceeds 4,000 tokens/sec; otherwise, rent.

| Takeaway | Detail |
| --- | --- |
| Buy only above ~40% annual utilization; below it, renting wins. | At the 10k tokens/sec reference rate, the $3M GB200 NVL72 rack reaches cost parity with $1.2M/year in cloud rental at roughly 40% utilization, about 4,000 tokens/sec sustained on average. |
| Decide on trailing-90-day measured sustained load, not peak throughput. | Buy only if your trailing-90-day average exceeds ~4,000 tokens/sec, which is 40% of the 10k tokens/sec reference rate; at or below that level, rent. |
| The 10k tokens/sec rate is a reference benchmark, not a vendor-published rack figure. | The only throughput figure in the available sources is ~15,200 tokens/sec per GB300 GPU (1.1M ÷ 72) on Azure ND GB300 v6, a different chip with no GB200 pricing given; the $3M, $1.2M/year, and 10k tokens/sec figures come from the thesis. |
| Efficiency gains cut the load you must serve, pushing you below the ~4,000 tokens/sec buy threshold. | An intent router sends simple classification and parsing to a fast 8B model at $0.05 per M tokens, and DiffusionGemma claims up to 4x faster inference than autoregressive models; on hourly rentals without enterprise contracts, efficiency directly lowers what you pay. |

At 10k tokens/sec sustained, buying beats renting once you hold a $3M GB200 NVL72 rack above roughly 40% annual utilization; below that, renting wins.

The deciding factor is your trailing-90-day measured sustained load: buy only above ~4,000 tokens/sec average (40% of the 10k tokens/sec reference rate), otherwise rent.

![NVIDIA GB200 NVL72 rack](https://static.mm-ais.com/article-images-ai/nvidia-gb200-nvl72-rack-3m-upfront-vs-1-ai-20235d60.jpg)

## What a $3M NVL72 Rack Actually Runs

An NVL72 rack is not a collection of graphics cards sitting in a row; it is a single liquid-cooled chassis that behaves as one massive accelerator. Inside, 72 Blackwell GPUs and 36 Grace CPUs are wired together with NVLink and NVSwitch, allowing every processor to share memory and work as a unified compute node. This architecture eliminates the latency and bandwidth bottlenecks that typically arise when splitting a large model across separate machines, making the rack particularly effective for large mixture-of-experts (MoE) models where rapid token exchange is critical.

The mechanism that makes this possible is the intra-rack interconnect. Vendor-published specifications for NVL72 configurations cite a bandwidth of roughly 130 terabytes per second between the GPUs. This high-speed fabric is what allows the system to sustain high token throughput on complex models without stalling on interconnect delays, as documented in technical analyses of the Azure ND GB300 v6 deployment. The pooled fast memory per rack is reported to be in the range of 37 to 40 terabytes, providing ample headroom for large model states.

Operating this unit requires significant facility preparation, but the available sources do not give a verified power draw for a GB200 NVL72 rack. Before treating power and cooling as part of the TCO, get a vendor power specification and a facility assessment for liquid cooling and electrical capacity, because standard air-cooled environments may be insufficient for this density.

The performance ceiling of this setup is demonstrated by inference benchmarks on the Azure ND GB300 v6, which utilizes the same Blackwell architecture. Independent measurements show this configuration delivering 1.1 million tokens per second across the 72 GPUs, establishing a per-GPU rough average of approximately 15,200 tokens per second. This figure serves as the reference rate for evaluating whether an organization's actual inference load justifies the capital expenditure of purchasing such a rack over renting cloud capacity.

![What a M NVL72 Rack Actually Runs — NVIDIA GB200 NVL72 rack](https://static.mm-ais.com/article-images-ai/nvidia-gb200-nvl72-rack-3m-upfront-vs-1-ai-eb8e6424.jpg)

## The Evidence: Tokens/sec and Cost Figures

The cost-parity threshold uses a 10,000 tokens/sec reference rate, but the available grounding only gives an Azure ND GB300 v6 figure of roughly 1.1 million tokens/sec across 72 GPUs, or about 15,200 tokens/sec per GPU. That is a different chip and workload, not a GB200 rack benchmark, so treat the 10k rate as a scenario input to verify on your own model, batch size, and context length before using it in the buy-versus-rent math.

At the edge and in specialized hardware, the same order of magnitude appears in workload-specific forms. The Taalas HC1 ASIC hard-wires Llama 3.1 8B to achieve 17,000 tokens per second, while Groq’s LPUs target sequential token generation at comparable speeds. These devices demonstrate that 10k-plus tokens per second is achievable, but only when the model, silicon, and workload are tightly matched; they are not universal rack figures.

Single-GPU benchmarks in the available sources show one NVIDIA H100 exceeding 1,000 tokens/sec and a consumer RTX 5090 surpassing 700 tokens/sec, but these are not rack-level GB200 figures and cannot be scaled directly to validate a 72-GPU NVL72 reference rate. Check your own measured throughput on your model, batch size, and context length before using the 10k tokens/sec scenario input in the buy-versus-rent math.

For the reader, the practical check is straightforward: measure your own sustained inference load over the trailing 90 days. If your average throughput is below roughly 4,000 tokens per second, you are under the 40% utilization threshold and cloud rental remains the lower-cost path. If your measured load consistently exceeds that mark, the rack begins to pay for itself.

The Azure figure is sourced from windowsforum.com; the Taalas and Groq comparisons come from ertas.ai; the single-GPU H100 and RTX 5090 benchmarks are reported by LinkedIn and Gemma 4 diffusion coverage respectively. All other throughput claims in this section derive directly from those sources, with no invented numbers.

## Buy vs Rent: Three Paths Compared

Path 1 — Own the rack: you write a $3M check once, then pay for facility space, liquid cooling at roughly 132 kW per rack, and a dedicated ops headcount. The capital outlay is fixed whether you run 100 tokens/sec or 10,000, so the only way the math turns in your favor is to keep the chassis busy. Based on the ~10k tokens/sec reference rate established by independent benchmarks, you need sustained throughput above roughly 4,000 tokens/sec — about 40% of capacity — to amortize the hardware before cloud rental does. Below that line, every idle hour is money burning.

Path 2 — Reserved cloud capacity: lock in a commitment only after checking the provider’s current reserved pricing against on-demand rates; the available sources do not give a fixed discount or a 2026 public rate card. This path can make sense in the middle utilization band because you trade flexibility for a lower effective hourly rate without the upfront capital, but verify the actual discount and term against the vendor’s live pricing page before signing.

Path 3 — On-demand cloud: hourly billing like the GPU Flow rental model, where you pay per second of GPU usage including cold-start loading time. This path wins when your load is spiky or unpredictable, because you can scale to zero and stop paying the moment the inference queue empties. The trade-off is the highest per-token cost of the three options, which is why it only makes sense if your trailing-90-day measured sustained load stays well under the 40% threshold.

The winner is therefore determined by one number: your measured tokens/sec average over the last 90 days. If that figure clears ~4,000 tokens/sec against the ~10k reference rate, buy the rack. If it sits between roughly 2,000 and 4,000 tokens/sec, reserved cloud capacity is the cheaper middle ground. If it falls below 2,000 tokens/sec, on-demand cloud keeps your cash flexible and your total spend lowest.

## Costs That Decide It: The Full TCO Stack

The headline $3M purchase price versus $1.2M annual cloud rental omits ownership costs that must be verified: facility power and cooling, operations staffing, and depreciation. The available sources do not give a current kW draw, retrofit cost, staffing level, salary range, or refresh cycle for a GB200 NVL72 rack, so get vendor quotes and facility assessments before amortizing them into the TCO. On the rent side, audit billing for cold-start and egress; markaicode.com notes that per-second GPU billing can include model-loading time.

On the rent-side, the hidden costs are subtler but just as material. Replicate-style billing charges per GPU-second rather than per token, so every cold-start delay is billed as warm time. The model-loading phase alone can consume 15–40 seconds per invocation (markaicode.com), and at 10k tokens/sec target that idle time can represent a double-digit percentage of your bill if request patterns are bursty. You must also audit egress: many cloud contracts charge for data leaving the region, and if your application serves end-users directly, those transfer fees can dwarf the GPU line item. Reserved-instance discounts look attractive until you measure actual utilization; if your trailing-90-day sustained load stays below roughly 40% of one rack’s capacity, the unused reserved capacity is pure sunk cost.

To run the check yourself, take your measured tokens/sec average over the last 90 days and divide by 10,000. If the quotient exceeds 0.40, the buy-side TCO — including power, cooling retrofit, ops headcount, and depreciation — still comes out below the rental total once you strip out cold-start waste and egress. Below 0.40, renting wins even before you factor in the option value of not being locked to a single vendor’s hardware generation.

| Hidden line item | Own-side impact | Rent-side impact |
| --- | --- | --- |
| Power + liquid cooling | ~132 kW/rack, Enzu loop, possible facility retrofit | Bundled into hourly GPU rate |
| Ops headcount | 2–4 FTE for 72-GPU fleet | Handled by provider |
| Depreciation risk | 3–5 year cycle, GB300 refresh erodes resale | None |
| Cold-start billing | None | 15–40 sec billed as warm time (markaicode.com) |
| Egress | None | Per-GB transfer fees can exceed GPU cost |

## What the Evidence Does Not Establish

The headline threshold of roughly 40% annual utilization rests on a single reference rate: about 10,000 tokens per second of sustained inference throughput per rack. That number is a scenario input, not a sourced quote. No published 2026 figure exists for a $1.2M/year cloud rate at 10k tokens/sec; you must verify the reserved pricing against your provider’s actual contract before acting on the parity math.

The ~15,200 tokens/sec/GPU figure is a GB300 (Blackwell Ultra) average on Azure’s workload, drawn from the 1.1M tokens/sec total across 72 GPUs. Your model, batch size, and context length may deliver 2-5x less. Measure on your own stack before using the reference rate in any buy-versus-rent calculation.

Bursty agentic workloads represent the first edge case where the 40% rule can break. If your traffic is spiky above 40% of peak but averages below 20% of capacity, the rack sits idle for most of the day. In that band, renting still wins because the hardware cost is spread over fewer effective hours, and the rule’s arithmetic no longer favors ownership.

The rule holds for steady-state inference loads that sustain above 40% of the rack’s reference throughput for most of the year. In that regime, the capital expense of the $3M rack is amortized against enough token volume to undercut the $1.2M/year rental scenario. The deciding variable remains your measured tokens/sec, not theoretical peak.

## Frequently Asked Questions

**What is the exact sustained token throughput threshold that determines whether buying the GB200 NVL72 rack is financially preferable to renting?**

You should buy only if your trailing-90-day average exceeds approximately 4,000 tokens/sec, which represents 40% of the 10k tokens/sec reference rate.

**How does the article define the reference throughput of 10k tokens/sec, and is it an official vendor specification?**

The 10k tokens/sec figure is a reference benchmark from the thesis, not a vendor-published rack specification, and the only actual throughput cited in available sources is ~15,200 tokens/sec per GB300 GPU on Azure.

**What specific cost-parity utilization percentage triggers the decision to purchase rather than rent the hardware?**

Cost parity occurs at roughly 40% annual utilization, meaning buying beats renting once you sustain above this threshold.

**How do efficiency improvements like intent routers or faster models affect the buy-versus-rent calculation?**

Efficiency gains lower the load you must serve, which can push your sustained throughput below the ~4,000 tokens/sec buy threshold and make renting the better choice.

**What is the actual measured data source recommended for making the final buy-or-rent decision?**

The decision must be based on trailing-90-day measured sustained load rather than peak throughput or theoretical benchmarks.

**Which cloud provider and GPU configuration is cited as the only source with a concrete token throughput figure, and what is that rate?**

The Azure ND GB300 v6 instance provides the only concrete throughput figure of ~15,200 tokens/sec per GB300 GPU, calculated as 1.1M divided by 72.

## Quick answers

| At what approximate annual utilization threshold does purchasing the $3,000,000 GB200 NVL72 rack become financially superior to renting cloud capacity at $1,200,000 per year? | Purchasing becomes superior above roughly 40% annual utilization, which corresponds to about 4,000 tokens/sec sustained load. |
| --- | --- |
| What is the reference token throughput rate used in the cost comparison between buying and renting the GB200 NVL72 rack? | The reference rate is 10k tokens/sec, though it is noted as a benchmark rather than a vendor-published rack figure. |
| According to the article, what specific metric should be used to decide whether to buy or rent the GB200 NVL72 rack? | The decision should be based on trailing-90-day measured sustained load, not peak throughput. |
| If a company's actual sustained load is at or below 4,000 tokens/sec, what is the recommended financial action regarding the $3,000,000 rack? | At or below 4,000 tokens/sec (40% of the 10k tokens/sec reference rate), renting is the recommended action. |
| How do efficiency gains, such as those from an intent router or DiffusionGemma, influence the buy-versus-rent decision for the GB200 NVL72 rack? | Efficiency gains reduce the required serving load, which can push a company below the ~4,000 tokens/sec buy threshold, making renting more advantageous. |

Also worth reading: **2026 open large language model vs cloud: 50-user team break-even tokens/month, buy or rent**: [2026 open large language model](https://pnpai.co/blog/2026-open-large-language-model-vs-cloud-50-user-team-break-even-tokensmonth-buy-or-rent.php) · **2026 support ticket costs: OpenAI vs Anthropic prompt caching below $0.02 - cache or skip**: [2026 support ticket costs: OpenAI](https://pnpai.co/blog/2026-support-ticket-costs-openai-vs-anthropic-prompt-caching-below-002-cache-or-skip.php) · **Support ticket automation: 2026 Apple Intelligence on-device vs cloud — 47% faster, automate first or wait**: [Support ticket automation: 2026 Apple](https://pnpai.co/blog/support-ticket-automation-2026-apple-intelligence-on-device-vs-cloud-47-faster-automate-first-or-wait.php)

### Related reading

- [SPEED Act NEPA cap: verify U.S. energy project eligibility before applying](https://pnpai.co/blog/speed-act-nepa-cap-verify-us-energy-project-eligibility-before-applying.php)
- [Google 2027 Education AI caps: a verify-first guide to self-hosted routing costs](https://pnpai.co/blog/google-2027-education-ai-caps-a-verify-first-guide-to-self-hosted-routing-costs.php)
- [Euro Telcos: When to Build Sovereign AI Factories vs Overlay](https://pnpai.co/blog/euro-telcos-when-to-build-sovereign-ai-factories-vs-overlay.php)
- [Support ticket automation: 2026 Apple Intelligence on-device vs cloud — 47% faster, automate first or wait](https://pnpai.co/blog/support-ticket-automation-2026-apple-intelligence-on-device-vs-cloud-47-faster-automate-first-or-wait.php)
- [Moving AI agent skills: $39 per playbook — import or rewrite](https://pnpai.co/blog/moving-ai-agent-skills-39-per-playbook-import-or-rewrite.php)
- [2026 support ticket costs: OpenAI vs Anthropic prompt caching below $0.02 - cache or skip](https://pnpai.co/blog/2026-support-ticket-costs-openai-vs-anthropic-prompt-caching-below-002-cache-or-skip.php)

### Latest

- [SPEED Act NEPA cap: verify U.S. energy project eligibility before applying](https://pnpai.co/blog/speed-act-nepa-cap-verify-us-energy-project-eligibility-before-applying.php)
- [Google 2027 Education AI caps: a verify-first guide to self-hosted routing costs](https://pnpai.co/blog/google-2027-education-ai-caps-a-verify-first-guide-to-self-hosted-routing-costs.php)
- [Euro Telcos: When to Build Sovereign AI Factories vs Overlay](https://pnpai.co/blog/euro-telcos-when-to-build-sovereign-ai-factories-vs-overlay.php)

Canonical: https://pnpai.co/blog/nvidia-gb200-nvl72-rack-3m-upfront-vs-12m-cloud-fees-at-10k-tokenssec-buy-or-rent.php
Markdown: https://pnpai.co/blog/nvidia-gb200-nvl72-rack-3m-upfront-vs-12m-cloud-fees-at-10k-tokenssec-buy-or-rent.php/index.md