DeepInfer: DeepSeek opens the engine behind its AI

DeepSeek released DeepInfer on October 20: the inference engine it uses to serve DeepSeek-V4.1-Flash at scale. The code is available under Apache 2.0 in its official repository. The company also published benchmarks and details of its deployment.

The announcement raises a useful question: how much does opening the engine help when running it efficiently still takes substantial infrastructure? To answer it, look at both the code you can download and the conditions attached to the numbers.

What is DeepInfer and what does it do?

Inference is the process through which an already trained model generates a response. DeepInfer organizes that work: it distributes computation, coordinates requests and uses optimized kernels, small routines that make effective use of the hardware. Its goal is to handle more requests while keeping users’ wait times low.

According to the README for this release, the supported model is DeepSeek-V4.1-Flash, running on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. The release lets developers study and adapt the engine that runs it; model capabilities and service performance are separate questions.

What does the DeepInfer open-source release include?

The release provides the core execution engine and offline inference examples. The README identifies the additional pieces needed to build a complete service:

A KV cache reuses earlier context computations; a load balancer distributes requests. Both matter when many people use a service at once. Having the engine available therefore does not mean downloading a complete replica of DeepSeek’s service.

DeepInfer benchmarks: what do the numbers show?

DeepSeek publishes initial context processing (prefill) benchmarks using 4 SM100 GPUs and response generation (decode) benchmarks using 64 SM100 GPUs. Each test runs within one high-speed scale-up domain, according to its methodology and reproduction script.

This is its original generation chart. The horizontal axis measures speed per user; the vertical axis measures aggregate throughput per GPU. Point labels indicate requests per data-parallel rank.

Open the full-size chart. The curve shows a practical trade-off: handling more work simultaneously can increase total output per GPU while individual users receive tokens more slowly. A throughput number alone does not describe the whole experience.

Try changing the DSpark load

DSpark generates candidate tokens and then verifies them. DeepSeek compares always verifying five candidates with choosing a dynamic number. Its technical Q&A publishes the following throughput estimates derived from EP64/DP64 data. This is a separate experiment from the chart above.

Choose a concurrency level. At 16 sequences per rank, the dynamic variant loses slightly; at 96, it gains 11%. Near the operating point the company describes, about 60 sequences per rank, the table estimates +5.4%. Load matters.

DSpark · EP64 / DP64

Load changes the result

Generated tokens per second per rank · DeepSeek estimates

Fixed verification (5)Dynamic length

Concurrency · sequences per rank

60 sequences/rank+5.4%+5.4%throughput change

13,730 → 14,466 tokens/s/rank switching from fixed to dynamic verification.

Estimates · tokens/s/rank

Seq./rank	Fixed (5)	Dynamic	Change
16	4,891	4,816	-1.5%
32	8,964	9,111	+1.6%
48	12,032	12,467	+3.6%
60	13,730	14,466	+5.4%
72	14,962	15,961	+6.7%
96	16,817	18,660	+11.0%

What hardware do you need to run DeepInfer?

The repository requires Linux, GCC 13 or later and Python 3.12, plus CUDA 13 for NVIDIA or CANN 9.2 for Ascend. This release is aimed at teams operating AI infrastructure.

In its Q&A, DeepSeek recommends 64 GPUs for prefill plus another 64 for decode as the smallest deployment unit it considers efficient in this production design. That is 128 GPUs in total. This is a recommendation about efficiency at scale: it does not establish a universal minimum for starting any example.

If you mainly use a chatbot or creative tool, the release helps explain how the model is served. Directly using the code takes a team capable of integrating and operating that infrastructure.

Is DeepInfer free? Will API prices fall?

Apache 2.0 allows use and modification of the code under its terms. Running it still involves hardware, energy and operating costs. The announcement expresses an intention to reduce compute costs and energy use; it does not announce a new API rate we can apply to your bill.

The immediate contribution is access to the engine, architectural decisions and results that other teams can try to reproduce. Lower prices for users will depend on how that work is used and the rates each provider publishes.

DeepSeek’s original announcement

DeepSeek · original October 20, 2026 post

Introducing DeepInfer, our in-house inference framework, now fully open source!

DeepSeek introduces the engine it uses with V4.1-Flash and links to the code. The README details this version’s scope and the components still requiring integration.

Should OpenAI and Anthropic open their production engines too? There is a concrete question to debate: how much value comes from being able to inspect and adapt the software, and how much reaches people who cannot operate a cluster.

Editorial review: October 20, 2026. Technical sources pinned to commit 774ce244b87cc0eb8d4b94ab9f5de2ab22049f01. Benchmarks and estimates are attributed to DeepSeek; we have not run our own tests. Cover: HeyMark editorial composition using DeepSeek’s official GitHub avatar. Interior chart: original repository image.