# DeepInfer: DeepSeek opens the engine behind its AI

> DeepSeek open-sources DeepInfer, its inference engine. What the release includes, how to read its benchmarks and who has the hardware to run it.


Published: 2026-10-20. Original source: [DeepSeek · DeepInfer announcement](https://x.com/deepseek_ai/status/2112476496283775135).

**DeepSeek released DeepInfer on October 20: the inference engine it uses to serve DeepSeek-V4.1-Flash at scale.** The code is available under Apache 2.0 in its [official repository](https://github.com/deepseek-ai/DeepInfer). The company also published benchmarks and details of its deployment.

The announcement raises a useful question: **how much does opening the engine help when running it efficiently still takes substantial infrastructure?** To answer it, look at both the code you can download and the conditions attached to the numbers.

## What is DeepInfer and what does it do?

Inference is the process through which an already trained model generates a response. DeepInfer organizes that work: it distributes computation, coordinates requests and uses optimized kernels, small routines that make effective use of the hardware. Its goal is to handle more requests while keeping users’ wait times low.

According to the [README for this release](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md), the supported model is **DeepSeek-V4.1-Flash**, running on **NVIDIA SM100 GPUs** or **Huawei Ascend 950 NPUs**. The release lets developers study and adapt the engine that runs it; model capabilities and service performance are separate questions.

## What does the DeepInfer open-source release include?

The release provides the core execution engine and offline inference examples. The README identifies the additional pieces needed to build a complete service:

| Area | Status in this release |
| --- | --- |
| Engine, kernels and distributed execution | **Included** |
| DSpark, asynchronous scheduling and structured outputs | **Included** |
| KV cache system | Requires additional implementation |
| Disaggregated vision, prefill and decode | Requires integration; much of the engine support already exists |
| Load balancing across replicas | Requires additional implementation |

A KV cache reuses earlier context computations; a load balancer distributes requests. Both matter when many people use a service at once. Having the engine available therefore does not mean downloading a complete replica of DeepSeek’s service.

## DeepInfer benchmarks: what do the numbers show?

DeepSeek publishes initial context processing (*prefill*) benchmarks using **4 SM100 GPUs** and response generation (*decode*) benchmarks using **64 SM100 GPUs**. Each test runs within one high-speed scale-up domain, according to its [methodology and reproduction script](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md#performance-benchmarks).

This is its original generation chart. The horizontal axis measures speed per user; the vertical axis measures aggregate throughput per GPU. Point labels indicate requests per data-parallel rank.

![Official DeepSeek chart: increasing requests per rank raises aggregate throughput while reducing per-user speed. Configuration: EP64/DP64, 131,072-token context and 64 SM100 GPUs.](/news/deepinfer/decode-pareto.png)

*Complete official DeepSeek image. Decode with 64 SM100 GPUs; EP64/DP64, 131,072-token context and fixed nextn=3. This is not a HeyMark test.*

[Open the full-size chart](/news/deepinfer/decode-pareto.png). The curve shows a practical trade-off: handling more work simultaneously can increase total output per GPU while individual users receive tokens more slowly. **A throughput number alone does not describe the whole experience.**

### Try changing the DSpark load

DSpark generates candidate tokens and then verifies them. DeepSeek compares always verifying five candidates with choosing a dynamic number. Its [technical Q&A](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference_deployment_qa.md) publishes the following **throughput estimates derived from EP64/DP64 data**. This is a separate experiment from the chart above.

Choose a concurrency level. At 16 sequences per rank, the dynamic variant loses slightly; at 96, it gains 11%. Near the operating point the company describes, about 60 sequences per rank, the table estimates **+5.4%**. Load matters.

### DSpark: load changes the result

DeepSeek estimates derived from EP64/DP64 data. Unit: tokens/s/rank. Concurrency: sequences/rank. Fixed verification of 5 tokens versus dynamic length.

| Concurrency | Fixed (5) | Dynamic | Change |
| --- | --- | --- | --- |
| 16 | **4,891** | 4,816 | -1.5% |
| 32 | 8,964 | **9,111** | +1.6% |
| 48 | 12,032 | **12,467** | +3.6% |
| 60 | 13,730 | **14,466** | +5.4% |
| 72 | 14,962 | **15,961** | +6.7% |
| 96 | 16,817 | **18,660** | +11.0% |

The change ranges from −1.5% to +11.0% with load. This is not a HeyMark measurement, a price cut or a model-quality score.

[DeepSeek · Q&A](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference_deployment_qa.md)

## What hardware do you need to run DeepInfer?

The repository requires Linux, GCC 13 or later and Python 3.12, plus CUDA 13 for NVIDIA or CANN 9.2 for Ascend. This release is aimed at teams operating AI infrastructure.

In its Q&A, DeepSeek recommends **64 GPUs for prefill plus another 64 for decode** as the smallest deployment unit it considers **efficient** in this production design. That is **128 GPUs in total**. This is a recommendation about efficiency at scale: **it does not establish a universal minimum for starting any example**.

If you mainly use a chatbot or creative tool, the release helps explain how the model is served. Directly using the code takes a team capable of integrating and operating that infrastructure.

## Is DeepInfer free? Will API prices fall?

Apache 2.0 allows use and modification of the code under its terms. **Running it still involves hardware, energy and operating costs.** The announcement expresses an intention to reduce compute costs and energy use; it does not announce a new API rate we can apply to your bill.

The immediate contribution is access to the engine, architectural decisions and results that other teams can try to reproduce. Lower prices for users will depend on how that work is used and the rates each provider publishes.

## DeepSeek’s original announcement

[DeepSeek · original October 20, 2026 post](https://x.com/deepseek_ai/status/2112476496283775135)

> Introducing DeepInfer, our in-house inference framework, now fully open source!

DeepSeek introduces the engine it uses with V4.1-Flash and links to the code. The README details this version’s scope and the components still requiring integration.

**Should OpenAI and Anthropic open their production engines too?** There is a concrete question to debate: how much value comes from being able to inspect and adapt the software, and how much reaches people who cannot operate a cluster.

<small>Editorial review: October 20, 2026. Technical sources pinned to commit 774ce244b87cc0eb8d4b94ab9f5de2ab22049f01. Benchmarks and estimates are attributed to DeepSeek; we have not run our own tests. Cover: HeyMark editorial composition using DeepSeek’s official GitHub avatar. Interior chart: original repository image.</small>

## Plan your next post in HeyMark.

Keep the idea, review the draft with your team, and see how it performed in the accounts you connected.

[Start free](https://app.heymark.ai) · [See how it works](https://heymark.ai/en/#product)

## Developer and agent resources

- [HeyMark MCP documentation](https://heymark.ai/en/mcp/)
- [llms.txt](https://heymark.ai/llms.txt)
- [Full site content for language models](https://heymark.ai/llms-full.txt)
- MCP protocol endpoint: `POST https://mcp.heymark.ai`
- OAuth protected-resource metadata: [/.well-known/oauth-protected-resource](https://mcp.heymark.ai/.well-known/oauth-protected-resource)
- MCP server card: [/server-card](https://mcp.heymark.ai/server-card)