大模型的推理成本，正在以一种离谱的速度下降。
9月5日，硅谷一家专注 AI 基础设施的服务商发布了最新的推理价格表：70B 参数级别的 MoE 模型推理，每百万 Token 只需要 0.08 美元。这个价格是什么概念？半年前，同样级别模型的推理成本大概是每百万 Token 2-3 美元。一年前，可能要 10 美元以上。
一年时间，推理成本降了 100 倍。
这个下降速度，比芯片性能提升的摩尔定律快得多，也比大多数人预期的快得多。很多人还在讨论"大模型推理太贵了用不起"，但成本曲线已经悄悄滑到了一个临界点——低到足够支撑一大批以前想都不敢想的应用。
为什么降得这么快？
推理成本的下降不是单一因素驱动的，是四层优化同时叠加的结果：
第一层：模型架构优化。 MoE 架构的普及是最大的推手。同样是 70B 级别，稠密模型每次推理要跑全部 70B 参数，MoE 模型每次只激活其中的 10-20B。激活参数少了，计算量自然就少了，成本直接按比例下降。
第二层：推理引擎优化。 vLLM、TensorRT-LLM、SGLang 这些推理框架在过去一年里进化速度极快。批处理、投机解码、PagedAttention、连续批处理……各种各样的优化技术层出不穷，同样的硬件能跑的吞吐量翻了好几倍。
第三层：硬件利用率提升。 以前 GPU 跑推理，利用率可能只有 20-30%，大部分时间在等数据、等内存、等下一个请求。现在通过更好的调度算法、更高效的内存管理，GPU 利用率能拉到 70-80% 甚至更高。同样一张卡，能做的事情多了两三倍。
第四层：专用硬件入场。 英伟达的 L40S、AMD 的 MI300、还有各种国产推理芯片，专门针对推理场景优化的硬件越来越多。这些卡的单精度算力可能不如训练卡，但推理性价比高得多。
四层优化叠在一起，成本不崩才怪。
成本降到这个水平，意味着什么？
当推理成本降到每百万 Token 0.1 美元以下，很多事情的经济模型就彻底变了。
举个例子：一个智能体每天和你交互 1000 轮，每轮消耗 2000 Token，一天就是 400 万 Token，成本是 0.32 美元。一个月不到 10 美元。这意味着"个人 AI 助理"这件事，从经济上已经完全可行了。以前大家觉得"AI 助理太贵了养不起"，现在一个月一杯咖啡钱。
再举个例子：一家中型公司，每天用 AI 处理 10 万份文档，每份文档 5000 Token，一天就是 5 亿 Token，成本是 400 美元。一个月一万多美元。这意味着企业级 AI 应用的门槛，已经低到任何公司都能用得起。以前需要算 ROI、需要层层审批的 AI 项目，现在可能一个部门经理就能拍板。
更重要的是：当推理成本足够低，你就不需要"省着用"了。以前用 AI 要考虑"这个问题值不值得调用模型"，要精心设计 Prompt、要尽量少用 Token。成本降到足够低以后，你可以让模型随便想、多试几次、自己跟自己辩论——从"省着用"变成"可劲儿用"，这才是真正的范式转变。
接下来会发生什么？
推理成本还会继续降吗？大概率会的。MoE 还在演进，推理框架还在优化，专用芯片还在出新一代。但下降速度可能会放缓——从 100 倍降到 1000 倍，肯定比从 1 倍降到 100 倍难。
但更值得关注的不是"还能降多少"，而是"降到这个水平以后，会催生什么新东西"。
历史上每次基础设施成本下降一个数量级，都会催生一整代之前不可能存在的应用。服务器成本下降催生了互联网和云计算，存储成本下降催生了大数据和短视频，带宽成本下降催生了流媒体和云游戏。推理成本下降 100 倍，会催生出什么？现在没人知道答案，但答案正在被写出来。
可以确定的是：那些还在纠结"AI 贵不贵"的公司，可能已经错过第一波浪潮了。真正的玩家已经在假设"推理几乎免费"的前提下，重新设计他们的产品和商业模式了。
明天见。
LLM inference costs are dropping at an absurd pace.
On September 5, a Silicon Valley AI infrastructure provider released its latest pricing sheet: inference for 70B-parameter MoE models costs just $0.08 per million tokens. What does that price mean? Six months ago, inference for the same level of model was roughly $2-3 per million tokens. A year ago, it might have been $10 or more.
In one year, inference cost dropped 100x.
This rate of decline is much faster than Moore's Law for chip performance, and much faster than most people expected. Many people are still debating "LLM inference is too expensive to use at scale," but the cost curve has quietly slid past a critical point — low enough to support a whole class of applications that nobody dared imagine before.
Why Is It Dropping So Fast?
The drop in inference costs isn't driven by a single factor. It's the simultaneous combination of four layers of optimization:
Layer 1: Model architecture optimization. The popularization of MoE architecture is the biggest driver. At the same 70B level, a dense model runs all 70B parameters per inference; a MoE model only activates 10-20B of them per token. Fewer activated parameters means less computation, and costs drop proportionally.
Layer 2: Inference engine optimization. Inference frameworks like vLLM, TensorRT-LLM, and SGLang have evolved extremely fast over the past year. Batching, speculative decoding, PagedAttention, continuous batching — optimization techniques of all kinds keep emerging, and throughput on the same hardware has multiplied several times over.
Layer 3: Hardware utilization improvement. In the early days, GPU utilization for inference might have been only 20-30%, with most of the time spent waiting for data, waiting for memory, waiting for the next request. Now, through better scheduling algorithms and more efficient memory management, GPU utilization can be pulled up to 70-80% or even higher. The same card can do two or three times more work.
Layer 4: Specialized hardware entering the market. Nvidia's L40S, AMD's MI300, and various domestic inference chips — more and more hardware specifically optimized for inference scenarios is coming to market. These cards may not have the FP32 compute power of training cards, but their inference price-performance ratio is much higher.
With four layers of optimization stacking together, it would be weird if costs didn't collapse.
Now That Costs Are This Low, What Does It Mean?
When inference cost drops below $0.1 per million tokens, the economic model of a lot of things completely changes.
Take an example: an agent that interacts with you 1,000 times a day, consuming 2,000 tokens per interaction — that's 4 million tokens a day, costing $0.32. Less than $10 a month. This means "personal AI assistant" is already economically fully viable. People used to think "AI assistants are too expensive to keep," but now it's a cup of coffee per month.
Another example: a mid-sized company using AI to process 100,000 documents a day, 5,000 tokens per document — that's 500 million tokens a day, costing $400. A bit over $10,000 a month. This means the barrier to enterprise AI applications has dropped so low any company can afford it. AI projects that used to require ROI calculations and layers of approval can now be decided by a department manager.
What's even more important: when inference cost is low enough, you don't need to "use it sparingly" anymore. Before, using AI meant thinking "is this problem worth calling the model for?" — you had to carefully craft prompts, minimize token usage. Once cost drops low enough, you can let the model think freely, try multiple times, debate with itself — going from "use sparingly" to "use freely" is the real paradigm shift.
What Happens Next?
Will inference costs keep dropping? Most likely yes. MoE is still evolving, inference frameworks are still improving, new generations of specialized chips are still coming. But the rate of decline may slow — going from 100x to 1000x is certainly harder than going from 1x to 100x.
But what's more worth paying attention to than "how much more can it drop" is "now that it's this low, what new things will it spawn?"
Throughout history, every time infrastructure costs drop by an order of magnitude, a whole generation of previously impossible applications is born. Server cost drops birthed the internet and cloud computing. Storage cost drops birthed big data and short video. Bandwidth cost drops birthed streaming and cloud gaming. Inference cost dropping 100x — what will it birth? Nobody knows the answer yet, but the answer is being written right now.
What's certain is this: companies still obsessing over "is AI expensive?" may have already missed the first wave. The real players are already redesigning their products and business models under the assumption that "inference is essentially free."
See you tomorrow.
From using sparingly to using freely — that's the real paradigm shift. Once cost drops low enough, products and business models all need to be redesigned.
— The Dawn Vision Editorial Desk
inference cost · 100x drop · MoE architecture · vLLM · inference optimization · hardware utilization · specialized chips · paradigm shift · economic model restructuring
Sources · 信源 Sources
本文基于 Dawn Vision 认知引擎处理的 11 个源信号生成，经编辑部人工审核。素材来源：硅谷AI基础设施服务商、Towards AI、Semianalysis。
This article was generated by the Dawn Vision cognitive engine processing 11 source signals, with human editorial review. Sources: Silicon Valley AI infrastructure provider, Towards AI, Semianalysis.
