Summary
DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with optimisations for scheduling, speculative decoding and large-model execution. The release does not include DeepSeek’s full cache-management, prefill/decode-disaggregation or load-balancing stack.
Why it matters
Opening part of the serving stack gives researchers and infrastructure operators more visibility into how a major open-model provider handles high-volume inference while highlighting the limits of the release.
Related coverage
- Google and Meta open-source native PyTorch support for TPUs
- Anthropic’s draft IPO filing reveals extraordinary future compute commitments
- SpaceX says its AI business became its largest revenue segment