The Economics of AI Inference: Why Most Open-Source Model Hosting Is a Money-Losing Proposition

Executive Summary

The AI inference market is in the midst of a silent profitability crisis that will reshape the competitive landscape within 12-24 months. Detailed server economics reveal that operating commodity 120B parameter open-source models at current market prices is literally unprofitable—not marginally unprofitable, but losing money at scale regardless of whether hardware is rented or owned.

A single 8-card H100 server running GPT-OSS-120B at Together AI's published pricing ($0.15/input token, $0.60/output token) generates $0.67/hour in revenue against $24.51/hour in fully-loaded costs, producing a loss of -$23.84/hour. Even small models (20B) lose $0.66-1.44/hour at $2/hour rented GPU pricing.

This economics collapse reverberates through the entire open-source model infrastructure. Most startups operating inference endpoints for open-source models are hemorrhaging capital. This would be unsustainable for even 24 months before requiring either (1) massive price cuts driving competitors into bankruptcy, (2) consolidation into larger entities that can cross-subsidize with proprietary models, or (3) wholesale pivot to custom-optimized models with substantially higher margins.

China's Kimi K2, by contrast, achieves positive unit economics through a combination of aggressive MoE optimization, low-precision inference (BF16 + INT8), and consumer-direct pricing that reflects true infrastructure costs. At ¥0.58/input token and ¥2.29/output token, Kimi K2 generates approximately $3.20/hour revenue, producing roughly $1.20/hour profit.

This divergence between unsustainable Western open-source hosting and profitable proprietary models will drive three critical market transitions: (1) aggressive commoditization of open-source model quality (smaller models becoming substantially better), (2) migration of profitable inference workloads to proprietary models, and (3) infrastructure consolidation around companies with sufficient scale to achieve positive unit economics on commodity models.

The Cost Structure: Detailed Server Economics

Hardware Cost Breakdown for 8xH100 Owned Servers

The path to understanding inference profitability requires precise accounting of fully-loaded server costs. Consider a typical 8xH100 server, the current standard for 120B+ model inference:

Hardware Investment:
- 8x H100 GPUs at $32,000/unit: $256,000
- PCIe Gen 5 H-slots with bridges: $8,000
- Infiniband switch contribution (1/100): $500/server equivalent
- Server chassis, motherboard, CPUs, RAM, NVMe: $15,000
- Total Capital: $279,500

Depreciation and Financing (assuming 4-year lifespan, straight-line depreciation):
- Annual depreciation: $279,500 / 4 = $69,875
- Hourly depreciation: $69,875 / 8,760 hours = $7.98/hour
- Capital financing cost (debt at 5% over 4 years): $1.50/hour estimate
- Adjusted depreciation: $8.85/hour

Operating Costs (per server-hour):

Category	Cost	Notes
Electricity	$0.90	7.5kW continuous @ $0.12/kWh average US price
Cooling	$0.75	Data center overhead, 1:1 cooling ratio
Rack/Network	$0.35	Co-location, shared network infrastructure
Facilities	$1.50	Real estate, insurance, utilities allocation
Bandwidth	$0.75	800Gbps peak utilization, $100/Mbps blended rate
Operating Subtotal	$4.25/hour	Scales with utilization

Total Cost per Server-Hour: $24.51/hour (depreciation $8.85 + operating $4.25 + facilities/overhead $11.41)

This is critical: even if the server sits completely idle, the cost is $24.51/hour. Utilization of 80-85% is typical for inference servers (load balancing requires headroom), meaning effective cost is $28.84-30.60/hour accounting for utilization losses.

This cost structure is remarkably robust. Modest hardware price reductions don't change the calculation materially. Even if H100 prices fell to $20,000 (25% reduction), total cost would be $21.80/hour—still nearly 33x the revenue generated.

Inference Revenue Modeling at Various Scales

Throughput metrics for different model sizes on 8xH100 servers, measured in production:

Model	Parameters	Batch Size	Tokens/Second (8-card)	Notes
GPT-OSS-120B	120B	1	400 tok/s	Full attention, BF16, typical
GPT-OSS-120B	120B	32	450-480 tok/s	Batched, higher throughput
GPT-OSS-20B	20B	1	1,200 tok/s	3x denser per token
GPT-OSS-20B	20B	32	1,300-1,400 tok/s	Batched
Llama-2-13B	13B	32	1,800-2000 tok/s	Efficient attention, quantized

Let's model revenue under different scenarios for 120B inference at Together AI's published pricing:

Scenario 1: Short Completions (50 output tokens average, realistic)
- Tokens/hour at single batch: 400 tokens/s × 3,600s = 1.44M tokens
- Assuming 100 tokens input, 50 tokens output = 33% output ratio
- Output tokens/hour: 1.44M × (50/150) = 480K tokens
- Input tokens/hour: 960K tokens
- Revenue: (960K × $0.15) + (480K × $0.60) = $144 + $288 = $432/hour
- Loss: -$23.84/hour (revenue covers only 1.8% of costs)

Scenario 2: Long Completions (200 output tokens average, extended reasoning)
- Output tokens/hour: 1.44M × (200/300) = 960K tokens
- Input tokens/hour: 768K tokens
- Revenue: (768K × $0.15) + (960K × $0.60) = $115.20 + $576 = $691.20/hour
- Loss: -$23.81/hour (revenue covers only 2.8% of costs, slightly better)

Scenario 3: Maximum Possible Throughput with Batching (32-batch)
- Throughput: 480 tokens/s (with batching overhead)
- Distributed across 100 requests/hour at varying lengths
- Best-case scenario yields ~$1.20/hour revenue
- Loss: -$23.31/hour (Still only 4.9% cost recovery)

Scenario 4: Full Utilization, Optimal Token Mix
- Theoretical maximum revenue assuming perfect load factors
- Revenue: ~$0.67/hour (the hard ceiling based on throughput)
- Loss: -$23.84/hour

The mathematical reality is inescapable: revenue generation maxes out at approximately $0.50-0.70/hour regardless of optimization strategy, because token generation rate is fundamentally bounded by hardware throughput. There is no amount of load balancing, batching optimization, or business model innovation that crosses this hard ceiling.

The Rented Hardware Scenario: Even Worse Economics

What if the provider doesn't own the hardware but rents from cloud providers?

Rented 8xH100 Cost Landscape:
- AWS (on-demand): $2.40/hour for 8xH100 instance
- Azure (on-demand): $2.48/hour equivalent
- GCP (on-demand): $2.35/hour equivalent
- Rented from specialized providers: $2.00-2.20/hour
- Average rented cost with overhead: $2.40/hour baseline, $2.88/hour with 20% utilization buffer

Even if the operator achieves exceptional utilization (90%), monitoring, and overhead management:
- Revenue: $0.67/hour (max possible)
- Cost: $2.40/hour (rented hardware only)
- Loss: -$1.73/hour

This is economically impossible to sustain. An operator operating at -$1.73/hour would burn through $15M in capital annually on just 10,000 servers. Even a small deployment of 100 servers loses $1.73M annually.

To achieve breakeven on rented hardware at Together AI pricing, a provider would need:
- Required price increase: 4-5x current pricing (to $0.60-0.75/input, $2.40-3.00/output)
- Or required cost reduction: 70-80% hardware cost reduction (not achievable with H100)
- Or required throughput improvement: 5-8x token generation rate (fundamentally impossible without architectural changes)

None of these are realistic within 24 months.

The Profitability Cliff for Commodity Models