Distributed inference

Classroom Trash Detection

A computer-vision system split across a network. A stateful client handles the camera and the dashboard, and a stateless FastAPI service runs YOLOv8 on a GPU.

• Refactored a single-process app into a stateful client and a stateless GPU inference service, connected by one authenticated HTTP contract.
• Cut network and GPU load by downscaling frames to 640 px and rescaling the returned coordinates, and by running inference on sampled frames with cached results in between.
• Added a local fallback for when the remote service is down, and a status API that reports which backend served each request and how long it took.
• Exported the model to half-precision ONNX, loaded it at startup, and deployed the same FastAPI app to a notebook GPU, a Docker image and a serverless GPU.