Year
2025
Role
Benchmarking methodology, profiling, analysis
Stack
- TensorRT
- ONNX Runtime
- TorchScript
- torch.compile
- Nsight
- CUDA
Systems performance
Optimization for AI Inference Engines on GPUs
Optimising GPU inference for ResNet50 and DistilBERT with FP16, INT8, TorchScript/torch.compile and ONNX.
- The project focuses on optimizing AI inference on GPUs for ResNet50 and DistilBERT through FP16, INT8, TorchScript/torch.compile, and ONNX.
- Results show that FP16 provides significant improvement, especially for DistilBERT: throughput increased from 305 → 1202 samples/s at batch size 8 (≈4×).
- INT8 (TensorRT) achieves 2–5× speedup but requires careful calibration to maintain accuracy.
- Graph optimization yields a 1.2–1.8× speedup, while CUDA profiling identifies the main bottlenecks in GEMM and ReLU kernels.
- The project demonstrates that mixed-precision and quantization are the most effective approaches for large-scale inference optimization.
- FP16 throughput
- ≈4×
- DistilBERT, 305 → 1202 samples/s at batch size 8
- INT8 speedup
- 2–5×
- TensorRT, needs careful calibration
- Graph optimisation
- 1.2–1.8×
