[{"data":1,"prerenderedAt":12},["ShallowReactive",2],{"stack-tr-en-tensorrt":3},{"tagline":4,"description":5,"strengths":6,"useCase":11},"NVIDIA's inference compiler that squeezes maximum performance out of a GPU in production.","TensorRT compiles a network, usually imported from ONNX, into an engine optimized for one specific NVIDIA GPU: layer and tensor fusion, automatic selection of the fastest kernels for the target architecture, and reduced precision in FP16, calibrated INT8 or FP8. The engine runs server-side through Triton or embedded on Jetson, and TensorRT-LLM applies the same approach to large language models with in-flight batching and a paged KV cache. It's our choice when the target is a fixed NVIDIA GPU and every millisecond of latency counts.",[7,8,9,10],"Layer fusion and kernels auto-tuned for the target GPU architecture.","FP16, calibrated INT8 and FP8 precision: lower latency and memory footprint.","TensorRT-LLM: in-flight batching, paged KV cache and quantization for LLM serving.","Same toolchain from data center to Jetson: integrated with Triton, DeepStream and Torch-TensorRT.","High-frame-rate video detection, high-throughput LLM serving, embedded inference on Jetson.",1792675979977]