Sean Woo
Key takeaways
- On September 4, 2026, NVIDIA published a guide and its own measurements for running reasoning models locally on Jetson.
- NVFP4 quantization uses lower-precision numbers to reduce memory traffic and computation.
- Speculative decoding lets a smaller draft model propose several tokens, which the larger model verifies together.
- NVIDIA reported up to 6.28x higher decode throughput for Qwen3.8-27B versus BF16, under a specific Jetson, checkpoint, and serving configuration.
- Higher token throughput does not automatically improve robot accuracy or safety; teams still need application-level and worst-case latency tests.
What if a robot could decide without asking the cloud?
Imagine an inspection robot finding fallen rock in an underground tunnel. If it must send camera data to a distant data center and wait for an answer, a weak connection can delay or interrupt the response. If an onboard computer can read the sensors and local maintenance instructions, it can continue a basic stop, reroute, and report sequence even when the network is unavailable. That is the practical appeal of edge inference. A safety stop, however, should not depend only on a sentence from a generative model. Verified control logic and a separate safety layer must retain final authority.
On September 4, 2026, NVIDIA published a technical guide to running compact reasoning models on Jetson AGX Thor and AGX Orin. Its examples were Nemotron 3.5 Lightning, a mixture-of-experts model with 30 billion total parameters and 3 billion activated per token, and Qwen3.8-27B, a dense model that activates all 27 billion parameters. NVIDIA suggested that the first may suit response-heavy agent loops, while the second may suit fewer but harder decisions. That is a workload recommendation, not a universal ranking verified across all robots. [NVIDIA technical post]
Acceleration tool one: smaller numbers with NVFP4
An AI model repeatedly moves weights and intermediate values from memory, then multiplies and adds them. A format such as BF16 provides a relatively generous range and precision, but it also means more data must be moved and processed. Quantization represents those numbers with fewer bits. NVFP4 is NVIDIA’s four-bit floating-point family. Like packing cargo more tightly into the same truck, it can reduce pressure on memory bandwidth and speed up calculation.
Four bits does not mean that every model file and runtime allocation becomes exactly one quarter of BF16. Some values remain at higher precision, while scaling data, the KV cache, and temporary runtime buffers still consume memory. Coarser numbers can also change the answer on sensitive tasks. NVIDIA therefore recommends starting with available quantized checkpoints, then considering quantization-aware training or distillation if quality drops on representative prompts. “Smaller” and “equally capable” are two separate claims that require separate measurements.
Acceleration tool two: draft first, verify together
A conventional language model generates one next token, feeds it back into the model, and repeats. Producing ten words therefore requires many sequential passes through the large model, and the delay accumulates. Speculative decoding asks a smaller draft model to propose several future tokens. The large model verifies the block in parallel, accepts the valid prefix, and resumes from the first rejected position. The large model retains the final decision. With the exact acceptance procedure described in the original research, this can reduce wall-clock time while preserving the target model’s output distribution.
The benefit depends on how often draft tokens are accepted. If the small model proposes poor continuations, drafting and verification can add cost without much progress. The ICML 2023 paper reported a 2x to 3x speedup for T5-XXL over the standard T5X implementation, but that result cannot simply be transferred to Jetson or another model. NVIDIA’s 2026 measurements also found different winners: DSpark for Nemotron 3.5 Lightning and DFlash2 for Qwen3.8-27B. The model, draft checkpoint, and real workload must be tested together. [ICML 2023 paper] [vLLM documentation]
Up to 6.28x does not mean the whole robot became 6.28x faster
NVIDIA used BF16 as a 1x baseline, then compared NVFP4 and NVFP4 combined with the fastest speculative-decoding configuration it tested for each model. In the published chart, Nemotron 3.5 Lightning reached 2.20x and 3.37x, while Qwen3.8-27B reached 2.33x and 6.28x decode-throughput improvement. These are NVIDIA’s relative generation-stage measurements using the listed model, vLLM container, memory settings, and draft method on Jetson AGX Thor or AGX Orin. They are not an independent reproduction, and they do not measure the complete delay from sensor input to motor action.
Absolute throughput also varied by task. With the fastest fixed configuration for each model, NVIDIA tested four SpeedBench categories: writing, reasoning, summarization, and retrieval-augmented generation. Nemotron ranged from 123.01 to 138.02 output tokens per second, while Qwen ranged from 27.69 to 34.44. Both models benefited most on RAG and writing and less on summarization. A buyer should therefore not select a platform from the 6.28x peak alone. The length of robot decisions, tool-call frequency, sensor preprocessing, and control-cycle timing all matter.
What should a robotics team validate?
First, measure time to first response and worst-case latency, not only average tokens per second. A robot that is normally fast but occasionally stalls in a hazardous situation is unsuitable for a safety loop. Second, compare quality on field prompts. Maintenance retrieval, anomaly explanation, and task planning need representative questions, including cases where a wrong answer is expensive. Third, limit which local data and tools the model can access when the network is down. Running at the edge does not automatically solve privacy, cybersecurity, or authorization.
Fourth, separate the reasoning model from the real-time controller. The model may summarize a situation or propose a plan, while verified safety logic enforces collision avoidance, speed limits, and emergency stops. Fifth, measure sustained performance with power and temperature included. A short benchmark may look impressive, but a sealed robot can reduce clock speed as it heats up. The real value of on-device reasoning is not one peak number. It is whether the system can repeat the required decisions within a defined time and power budget when connectivity is poor.
Primary sources reviewed
- NVIDIA Technical Blog · Published September 4, 2026; Jetson models, configurations, and throughput comparisons
- Fast Inference from Transformers via Speculative Decoding · ICML 2023; draft-and-verify method and T5-XXL evaluation
- vLLM Speculative Decoding Documentation · Supported methods and deployment conditions
Sources checked: September 15, 2026
About Sean Woo
I have spent more than 15 years working on robotics technology and business direction. Drawing on public technical documents, research papers, and company announcements, I explain changes in robotics and AI in accessible language. The analysis in this publication does not represent the official position of any company or institution.