TensorRT 10 × Blackwell Migration Guide (Part 2): A Passing Build Does Not Mean Correct Output — Five Cases of Silent Degradation

Hello!

In the previous article, "TensorRT 10 × Blackwell Migration Guide (Part 1): Inference Assets Won't Run on RTX 50 — The Basics and the First Walls," we covered why existing inference assets stop working in the move to the Blackwell generation, and how to get a minimal TensorRT 10 conversion through. The problems that appeared in Part 1 actually have one thing in common.

Every one of them had the decency to stop with an error.

What is truly frightening lies beyond that. In a TensorRT migration pipeline, there exists a way to fail where:

• the build passes
• execution passes too
• the speed is properly there
• the output's shape and values look plausible at a glance
• and yet the contents are wrong

.

In this article — covering not only TensorRT's own behavior but also export mistakes, precision settings, and engine deployment mistakes — we take failures that produce wrong output without stopping on an error and, for convenience, call them "silent degradation".

It is exceptionally troublesome, because it comes to light only after the tests pass, the benchmarks post good numbers, and you have already reported "we made it N times faster."

In this article we share five instances of silent degradation we actually stepped on, in the order of symptom → why it happens → how to find it → workaround.

At the end, we consolidate a verification design for detecting these systematically. First, in the interest of honesty, here is how far each case has been substantiated.

Part 1: Mixing up input/output dtypes

Symptom

We built an engine with FP16 tactics allowed and ran it by passing PyTorch half tensors directly
(the naive implementation, from before we added the "two dtype-matching lines" included in Part 1's TRTRunner).

No errors. Good speed.But the output was completely different from PyTorch's.

How different? On a model whose output range is a bit under 0.5, the maximum absolute error was about 13.8. That is not on the level of an fp16 precision issue. It is a different animal entirely.

Why it happens

config.set_flag(trt.BuilderFlag.FP16) means "allow FP16 implementation candidates," so FP16 tactics become selectable for internal computation (some layers may still get fp32). However, the network's input and output tensor types remain as defined in the ONNX. If you exported the ONNX in fp32, the engine's inputs and outputs stay fp32.

Now, what happens when you hand it the address of a PyTorch half tensor (data_ptr())? TensorRT treats that region as an input buffer laid out with fp32 elements. Since fp16 is 2 bytes per element and fp32 is 4, not only does the interpretation of the values change — TensorRT may also read past the region PyTorch allocated. The behavior is undefined: beyond corrupted output, some environments may see CUDA errors or crashes.

And the worst part is that in our measurements, even in this state `execute_async_v3` returned `True` (success), produced no NaN, and threw no exception.

Workaround

Query the dtype the engine requires and explicitly convert before passing. The two lines we put into Part 1's TRTRunner are exactly this.

want = _TRT2TORCH[self.engine.get_tensor_dtype(name)]   # query the dtype the engine requires
t = t.to(device=self.device, dtype=want).contiguous()   # match it, then pass
if not self.ctx.set_tensor_address(name, t.data_ptr()):
    raise RuntimeError(f"set_tensor_address failed: {name}")

"I built with fp16, so the input must be fp16" does not hold.Always ask the engine.

Part 2: mode='linear' was silently being executed as nearest

Of the five, this was the hardest to find and the most surprising trap.

Symptom

With the 5D grid_sample covered in Part 1, TensorRT refuses to build (which is, in its way, kind). So what about 4D `grid_sample`? There, both the build and the execution succeed without a problem。

But compare the output against PyTorch and it is uniformly off. It is off by the same amount in fp16 and fp32. So this is not a precision issue — something is structurally different. And yet, at a glance, the output image looks perfectly plausible.

First, pin down what is happening

The minimal reproduction is just this (build_engine / TRTRunner are used unchanged from Part 1).

class PlanarWarp(nn.Module):
    """Minimal module containing a 4D grid_sample"""
    def forward(self, im, grid):
        return F.grid_sample(im, grid, align_corners=False)

torch.manual_seed(0)
im = torch.randn(1, 8, 128, 128, device="cuda")
grid = torch.rand(1, 128, 128, 2, device="cuda") * 2 - 1
model = PlanarWarp().eval().cuda()

torch.onnx.export(model, (im, grid), "planar_warp.onnx",
                  input_names=["im", "grid"], output_names=["y"],
                  opset_version=20, dynamo=True)
build_engine("planar_warp.onnx", "planar_warp.plan", fp16=False)
runner = TRTRunner("planar_warp.plan")
trt_out = runner.run({"im": im, "grid": grid})["y"]

To identify the cause, we cross-checked TensorRT's output exhaustively against every interpretation of PyTorch's `F.grid_sample`。mode (bilinear / nearest) × align_corners (True / False) × padding_mode (zeros / border / reflection) — all 12 combinations.

for mode, align, pad in itertools.product(
        ["bilinear", "nearest"], [False, True], ["zeros", "border", "reflection"]):
    ref = F.grid_sample(im, grid, mode=mode, align_corners=align, padding_mode=pad)
    print(mode, align, pad,
          (trt_out - ref).abs().max().item(),   # maximum absolute error
          torch.equal(trt_out, ref))            # bitwise equality check

Here are the results.

TensorRT's output matched the result of `mode='nearest', align_corners=False` with `torch.equal` returning True — a bitwise exact match. Meanwhile, the difference from the intended bilinear is more than 3.1. When bilinear interpolation is swapped for nearest-neighbor, the output changes only to the extent of "looking somewhat jaggy." Buried inside a warping pipeline, you will almost never catch it by eye.

Why it happens — isolating with opset 16 versus 20

Inspecting the node attributes in the ONNX file, this is what we found.

ONNX GridSample attributes: {'align_corners': 0, 'mode': 'linear', 'padding_mode': 'zeros'}

mode is, per the opset 20 specification, correctly written as linear. And here an important fact comes in.In ONNX's GridSample, the interpolation mode name was `bilinear` in opset 16 and was renamed to `linear` in opset 20. And according to NVIDIA's official documentation, linear interpolation for GridSample is supported by TensorRT. So it is not that "TensorRT does not support linear." What is suspect is how the parser interprets the new opset 20 attribute name.

So we exported the same model with the same inputs at both opset 16 and opset 20 and compared.

Same TensorRT, same model: opset 16 is correct, opset 20 comes out nearest. From this measurement we can infer with high confidence that the ONNX ingestion path of TensorRT 10.16 in our environment fails to map the mode='linear' introduced in opset 20 to linear interpolation and falls back to nearest (in both the ONNX specification and TensorRT's documentation, linear is supported and is the default-side value, so this is not a case of "nearest is the default"). We have not inspected the post-parse layer attributes or the parser implementation, so we are not asserting the exact location, and a future version may fix it.

Workarounds — three, in order of ease

Workaround 1: export at opset 16 (easiest; verified by measurement)

As the table shows, exporting at opset 16 yields correct bilinear behavior in our environment. Unless you need 5D (which requires opset 20), this is sufficient.

Workaround 2: decompose `grid_sample` into primitive operations (verified through TensorRT execution)

If you have reasons to use opset 20, or would rather not entrust the operation to TensorRT, another option is to rebuild the operation as a mathematically equivalent composition of primitive operations with proven numerics in TensorRT. Bilinear interpolation is, at heart, just "fetch the 4 neighbors and take a weighted average," so it decomposes into Floor / Clamp / Gather and basic arithmetic.

Workaround 3: a custom plugin(covered in the final installment)

import torch
import torch.nn.functional as F

def bilinear_grid_sample(im, grid, align_corners=False):
    """Reimplements the forward inference of F.grid_sample(im, grid, mode='bilinear', padding_mode='zeros').

    Swap this in only at export time. Assumes finite-valued grids;
    NaN/Inf, backward, border/reflection, and 5D inputs are out of scope.
      im:   (N, C, H, W)
      grid: (N, Hg, Wg, 2)  range [-1, 1] (x, y order)
    """
    n, c, h, w = im.shape
    gx, gy = grid[..., 0], grid[..., 1]

    # normalized coordinates [-1, 1] → pixel coordinates
    if align_corners:
        x = (gx + 1) / 2 * (w - 1)
        y = (gy + 1) / 2 * (h - 1)
    else:
        x = ((gx + 1) * w - 1) / 2
        y = ((gy + 1) * h - 1) / 2

    x0, y0 = torch.floor(x), torch.floor(y)
    x1, y1 = x0 + 1, y0 + 1

    # weights for the 4 neighbors
    wa = (x1 - x) * (y1 - y)
    wb = (x1 - x) * (y - y0)
    wc = (x - x0) * (y1 - y)
    wd = (x - x0) * (y - y0)

    # Reproducing padding_mode='zeros':
    # add 1px of zero padding, clamp indices to [-1, size], then add 1.
    # However far out of range a coordinate goes, it ends up referencing a zero-valued padding pixel.
    im_p = F.pad(im, (1, 1, 1, 1), mode="constant", value=0.0)
    x0i = (x0.clamp(-1.0, float(w)) + 1).long()
    x1i = (x1.clamp(-1.0, float(w)) + 1).long()
    y0i = (y0.clamp(-1.0, float(h)) + 1).long()
    y1i = (y1.clamp(-1.0, float(h)) + 1).long()

    wp = w + 2
    im_flat = im_p.reshape(n, c, -1)          # flatten to (N, C, (H+2)*(W+2))

    def gather(yi, xi):
        # shape it as GatherElements(dim=2) to guarantee the ONNX / TensorRT correspondence
        idx = (yi * wp + xi).reshape(n, 1, -1).expand(-1, c, -1)
        return torch.gather(im_flat, 2, idx)

    va, vb, vc, vd = gather(y0i, x0i), gather(y1i, x0i), gather(y0i, x1i), gather(y1i, x1i)

    hg, wg = grid.shape[1], grid.shape[2]
    out = (va * wa.reshape(n, 1, -1) + vb * wb.reshape(n, 1, -1)
           + vc * wc.reshape(n, 1, -1) + vd * wd.reshape(n, 1, -1))
    return out.reshape(n, c, hg, wg)

This decomposed version was verified not only with self-tests in PyTorch but all the way through ONNX export → TensorRT build → execution. The three-way comparison measured as follows.

We have confirmed that it stays correctly bilinear when running on TensorRT. It is also worth writing a self-test that includes out-of-range grids alongside it.

torch.manual_seed(0)
dev = "cuda:0"
src = torch.randn(4, 3, 256, 256, device=dev)

cases = {
    "uniform_pm1":   torch.rand(4, 256, 256, 2, device=dev) * 2 - 1,
    "outside_pm1.5": torch.rand(4, 256, 256, 2, device=dev) * 3 - 1.5,   # includes out-of-range values
}
ys, xs = torch.meshgrid(torch.linspace(-1, 1, 256, device=dev),
                        torch.linspace(-1, 1, 256, device=dev), indexing="ij")
ident = torch.stack([xs, ys], dim=-1).expand(4, -1, -1, -1)
cases["identity_plus_flow"] = ident + torch.randn(4, 256, 256, 2, device=dev) * 0.05

for name, grid in cases.items():
    ref = F.grid_sample(src, grid)          # bilinear / zeros / align_corners=False
    got = bilinear_grid_sample(src, grid)
    ma = float((ref - got).abs().max())
    print(f"[selftest] {name}: max_abs={ma:.3e} -> {'PASS' if ma < 1e-5 else 'FAIL'}")

The point is to perform this swap only inside the export wrapper. Your production model code stays completely untouched.

Part 3: Even within the same generation, engines can break across GPU models

Symptom

We took an engine built on one GPU and ran it on a different model of the same Blackwell generation (sm_120). It runs. It is fast.

But the output was corrupted.

TensorRT's log said this (quoting the latter sentence verbatim).

[TRT] [WARNING] Using an engine plan file across different models of devices is not
supported and is likely to affect performance or even cause errors or deadlock.

Before it, one more warning appears, to the effect of "this engine requires more SMs (Streaming Multiprocessors) than this device has; a deadlock is likely". Even so, `deserialize` succeeds, `execute_async_v3` returns `True`, and processing does not stop.

Why it happens

As described in Part 1, TensorRT benchmarks kernels on the actual machine at build time and picks the fastest implementations. A default-configuration engine bases its execution plan not only on Compute Capability but on multiple hardware characteristics of the GPU it was built on (SM count, shared memory, L2 cache, and so on). A different model differs in these, which can lead not only to performance loss but to execution problems — and TensorRT itself detects the mismatch and warns.

"Same Compute Capability means compatible" is wrong. Even between two sm_120 GPUs, different models must be treated as different targets.

The genuinely nasty part

We also ran an experiment reproducing this situation: build the engine on a model with more SMs, then load and run it on one with fewer. The result was this.

Both warnings appeared. Yet the output matched the result on the build machine (the measured maximum absolute error was exactly 0.0, though we did not verify as far as `torch.equal`).

In other words, with a small, simple model, ignoring the warnings still "works". That is the worst possible outcome. Once you have seen it "work" during development, the warning starts to look like mere noise. Then, when you switch to the complex production model, it bares its teeth — spewing CUDA errors while continuing to stream corrupted output. What we actually experienced was the latter.

The official documentation does note that an engine may function on a different GPU of the same architecture with only a small performance loss. Even so, for production we treat this warning as a fail-fast condition (stop when it appears). As long as the breaking conditions remain unidentified, our judgment is that one should not bet on "it works now, so it is fine."

Workarounds

1. Manage and build engines per GPU model, not per Compute Capability — splitting the storage directory by sm_120/ alone is not enough; separate by model name as well
2. At build time, save "GPU product name, Compute Capability, SM count, TensorRT / CUDA versions" as metadata (a sidecar JSON, for example); at startup, check it against the runtime environment and refuse to deserialize on mismatch — this is the effective seawall. Note that checking the return value of `execute_async_v3` cannot detect this trap — in our experiment, True was returned even under the model mismatch
3. Treat TensorRT warnings as errors in production — do not swallow the logger's WARNING; combine it with the check above and fail fast
4. If distribution to multiple models is truly unavoidable, build with hardware compatibility mode (`SAME_COMPUTE_CAPABILITY`, etc.) — the official mechanism for use across GPUs with the same Compute Capability. It constrains performance and features, however, so verify both speed and numerics separately before adopting it

Part 4: Building in fp32 still lets TF32 into the candidate pool

Symptom

A module whose accuracy fell short under FP16-allowed settings was rebuilt with an FP32 configuration that disallows FP16 tactics. Yet its agreement with PyTorch did not improve as expected. What is more, the agreement fluctuated slightly with every rebuild — sometimes clearing the acceptance bar, sometimes not.

Why it happens

For convolutions and matrix multiplications on fp32 tensors, TensorRT by default "permits" the use of TF32 tactics. TF32 is a fast format with the mantissa trimmed to 10 bits, used by Tensor Cores on Ampere and later. Not every fp32 operation becomes TF32 — ordinary fp32 implementations may still be chosen — but the assumption "built in fp32, therefore everything runs in fp32" does not hold.

A plausible cause of the per-build fluctuation is measurement noise at build time. Because TensorRT's build relies on on-device measurement, different tactics may be chosen on different builds from candidates with similar performance (the official documentation describes this as well). That said, we did not go as far as comparing which tactics were actually selected.

The workaround — and a surprising side effect

For engines that strictly verify fidelity, disable TF32 explicitly.

config = builder.create_builder_config()
config.clear_flag(trt.BuilderFlag.TF32)     # ★ remove TF32 tactics from the candidate pool

Here are the measured results.

The maximum absolute error improved by three orders of magnitude, from 2.55e-04 to 6.26e-07.

And interestingly, in this one pair of builds, the TF32-disabled side came out faster (0.874 ms → 0.772 ms). A different tactic may have been selected, but we did not isolate the cause (no comparison of the selected tactics). The difference is within reach of build-time measurement noise, so do not read this roughly 12% as a general trend. Still, it would be a shame to skip the experiment out of the assumption that "disabling TF32 always slows things down," so measure speed in that exact configuration as well. We should add that even with TF32 disabled, numerical differences from PyTorch arising from operation ordering and tactic choices do not disappear entirely.

Part 5: Swapped arguments in the ONNX export (the accident that hurt the most)

The last one is not a TensorRT problem but our own mistake. Yet it took longer to find than anything else in this series. The same shape of accident can happen to anyone, so we are sharing it. Note that this case occurred in a production model that confidentiality prevents us from publishing, and we have not run a reproduction experiment with a dummy model for this article.

Symptom

After converting a certain module to TensorRT, we hit a strange symptom: how the output broke depended on the input. Under some conditions it matched perfectly; under others it clearly degraded. Moreover, the amount of degradation grew in proportion to the "magnitude of motion" in the input.

fp16 or fp32, TF32 on or off — the degradation did not change.At that point we strongly suspected a structural problem rather than a simple precision difference.

Why it happened

torch.onnx.export had received its arguments in an order that disagreed with the parameter order of the model's forward.

# forward definition:   forward(self, feature, kp_driving, kp_source)
# what export was given:      (feature, kp_source, kp_driving)   ← 2nd and 3rd swapped
torch.onnx.export(model, (feature, kp_source, kp_driving), "m.onnx", ...)

These are positional arguments, so Python raises no complaint. The export succeeds, the ONNX file is generated, the TensorRT build passes, and it runs. Except what you end up with is an ONNX in which the roles of two inputs are swapped and frozen that way inside the model.

Because the internal computation took a difference form along the lines of "reference − input A + input B," the error comes out to exactly 2 × (the difference between the two inputs). Hence the symptom: inputs with a small difference match, and the larger the difference, the more spectacularly it breaks.

Why discovery took so long — the "self-reenactment" verification pitfall

This is the biggest problem. Our verification at the time was performed by feeding the same data to both inputs. Under that condition the two inputs are equal, so even with them swapped, the results match perfectly. The tests passed with full marks.

We call this the "self-reenactment parity" trap.If you verify equivalence only under conditions where the inputs coincide with the reference, structural bugs like swapped arguments are rendered completely harmless-looking.

How to find it — determining whether the culprit is TensorRT or the ONNX

What proved effective here was isolation using onnxruntime.

1. First, split the model into stages and binary-search for where the numbers start to diverge (comparing intermediate tensors)
2. Once the diverging subgraph is identified, run the same ONNX on onnxruntime's CPU execution
3. If onnxruntime diverges in the same direction, suspect the ONNX graph, the input binding at export, or operator specification differences first. If PyTorch and onnxruntime agree and only TensorRT diverges, prioritize TensorRT's parser, engine, and runtime-side configuration

This is not a procedure that logically convicts the culprit; it is a triage that decides where to look first (onnxruntime can have implementation differences of its own, and the issue can also be how input names or dtypes are passed on the TensorRT side). Even so, it dramatically changes the opening moves of an investigation. In our case, onnxruntime disagreed by exactly the same amount, so we examined the export side first and pinned down the positional-argument mix-up in the code.

For per-layer comparison, polygraphy's compare-all-layers mode (mark all) is convenient, but on large models it exhausts memory and dies. In practice, a binary search that carves out only the suspicious part was more dependable.

The fix — keyword binding via kwargs

Make the argument passing to export itself keyword-based. torch.onnx.export provides, separate from the positional args, an official `kwargs` parameter for passing inputs by keyword. Use it, and the very concept of tuple ordering disappears.

# Problem: args is a tuple, bound by position. Get the forward argument
#       order wrong and the export still succeeds, generating an ONNX with
#       the input roles left swapped (input_names merely attaches names;
#       it guarantees no semantic correspondence)
# Fix:  bind "name → tensor" explicitly via the kwargs parameter
torch.onnx.export(
    model,
    args=(),
    kwargs={
        "feature": feature,
        "kp_source": kp_source,
        "kp_driving": kp_driving,
    },
    f="m.onnx",
    input_names=["feature", "kp_source", "kp_driving"],
    output_names=["y"], opset_version=20, dynamo=True,
)

After the fix, the degradation vanished completely.

A verification design that catches silent degradation

Lay the five traps side by side, and shared countermeasures come into view.

1. Measure numerics before speed

Reverse the order, and the good news — "it got faster" — arrives first, reducing numerical verification to a formality.A speedup with wrong numbers has no value.

2. Do not trust agreement on random inputs — verify with real data

We were able to back this up with measurements as well. We built an engine from a block stacking LayerNorm and GELU with FP16 tactics allowed, and measured the error while varying only the input range. The results follow (the reference is PyTorch fp32; in both the "FP16 allowed / disallowed" columns, TF32 remains at its default, i.e., allowed).

As the input scale widened, the FP16 configuration's maximum absolute error grew substantially (though the growth is not simple proportionality to the input multiplier, because LayerNorm normalizes the scale). The nRMSE, meanwhile, actually fell from 8.9e-04 to 2.5e-04 — at the very least it does not worsen with input scale. In other words, absolute error and normalized error paint completely different pictures of the same result. What this tells us is that an acceptance bar of "absolute error below X" calibrated only on standard normal randoms is easily broken at real-data ranges.

In our real model, moreover, this went beyond mere error growth and appeared as every output turning NaN the moment real data went in (an intermediate value just before a LayerNorm exceeded fp16's representable range). The remedy was to disallow FP16 tactics for that module and build it with an FP32 configuration.

3. Do not verify equivalence by "self-reenactment"

As in Part 5, tests confined to conditions where the input coincides with the reference hide structural bugs completely.Make sure your verification data includes the large-difference cases that can occur in real operation.

4. Triangulate the culprit with PyTorch / onnxruntime / TensorRT

• PyTorch and onnxruntime agree, only TensorRT diverges → suspect TensorRT's semantics and kernels (Part 2)
• onnxruntime diverges as well → suspect the ONNX — that is, the export (Part 5)

Being able to make this cut changes investigation time by an order of magnitude.

5. Carve out the suspicious operation and test it alone

Chasing a vague "somehow off" across a whole model is inefficient. Part 2 was settled in a single stroke by carving out one grid_sample and cross-checking it exhaustively.

6. Toggle precision and watch whether the symptom changes

If switching among fp16 / fp32 / TF32 barely changes the amount of degradation, prioritize structural problems — input binding, operator semantics — over rounding error. It is not a proof, but as a signal for setting investigation priorities it served us extremely well.

7. Build in mechanisms that refuse to swallow failure

execute_async_v3 return-value checks, engine-to-device consistency checks, and a design that refuses to start when a component is missing.Never allowing a state that "looks normal while partly broken" is the last seawall against silent degradation.

What we have not yet confirmed

• The root location of `mode='linear'` becoming nearest — from the opset 16/20 comparison we strongly infer interpretation at the parser stage, but we have not examined the parser source code or the layer attributes immediately after parsing. Verification also covered a single version, TensorRT 10.16.1.11, and a future version may fix it
• The conditions under which cross-model engines break — with this article's dummy model, the warnings appeared yet the outputs matched. We have not identified at what scale or structure breakage begins
• The same class of problems under int8 quantization — this article covers fp16 / fp32 / TF32 only

Summary

Summed up in one sentence: "in a TensorRT migration pipeline, neither 'the build passed' nor 'it ran' nor 'it got faster' guarantees in any way that the output is correct.".

That is exactly why we measure numerics before speed, measure with real data rather than randoms, triangulate the culprit when things diverge, and build in mechanisms that refuse to swallow failure. We consider these the minimum conditions for putting TensorRT into production.

Next time (the final installment) takes on the biggest homework left over from Part 1: getting 5D grid_sample, which TensorRT does not natively support, through with a custom plugin. Building it against the new CUDA / TensorRT generation involves quite a few hurdles: what to do when your local nvcc cannot generate code for Blackwell; the problem that pip-installed TensorRT ships without the development headers; and the plugin's fp16 kernel breaking down numerically — once again, a story bordering on silent degradation.

See you next time!

References (primary sources)

• NVIDIA TensorRT documentation (Developer Guide / Release Notes)
• ONNX Operators: GridSample (specification of mode / align_corners / padding_mode)
• PyTorch: torch.nn.functional.grid_sample
• NVIDIA/TensorRT issue #3890: request for native 5D GridSample support
• NVIDIA: TensorFloat-32 (TF32) explained
• polygraphy (TensorRT debugging and comparison tool)
• Related article from Qualiteg: 2026 NVIDIA GPU quick-lookup tool