A retail brand doesn’t ask how many people walked into a store. It asks why a store with high footfall isn’t selling, which stores deserve more stock before the festive season, and where to open the next one. Answering those questions from a camera feed takes far more than a model that can detect people.
I had already shipped multiple features with AI-assisted workflows, and in earlier projects the work usually started and ended with a technical problem. Camera Analytics for retail was different. I was building an ML pipeline for computer vision, and getting the model to work turned out to be the smaller part of the job.
The real product was everything around it: understanding what retailers needed, making it work in stores that were all set up differently, and keeping it reliable as features and data grew.
Retailers didn’t want a footfall counter
Retail brands wanted more than a count of how many people walked in. They wanted to understand customer behaviour: how long people browsed, whether they were greeted, and whether they came in alone or as part of a group.
Zones turn a camera view into behaviour: greeting at the entrance, browsing on the floor, and conversion at the billing counter. AI-generated image.
Those details fed much bigger decisions. Brands could compare footfall with sales, understand festive shopping trends, and use store performance to plan inventory and new store openings.
I spoke with retail clients in India and Poland to understand how they would use this. What surprised me was how far these granular details travelled, well beyond the store floor into decisions across the business. The needs also varied from one brand and store to the next, so understanding the use cases early shaped a product that could support more than one setup.
That early focus on real use cases shows in the results: the product converts more than 90% of proofs of concept (POCs) into customers.
Every store is a different problem
No two stores were set up the same way, and each one brought its own problems. Even a single camera angle changing could affect the setup and the results.
If every adjustment needed an engineer, onboarding more stores would become the bottleneck. So I built admin-level controls and an Electron app that handle much of the onboarding work. Someone who understands the system and the model can use it to reconfigure a store remotely, including when a camera angle changes.
Documenting the use cases and working through these requirements before building made the biggest difference. The work stopped being about implementing a feature and became about designing how the product would actually be used and maintained.
The infrastructure is part of the product
Every new pipeline stage or downstream task added load to the system. Before releasing a feature, I needed to know how it affected worker throughput, GPU load and power usage. The goal was to process as much data as possible without throttling.
With large volumes of data, each new feature meant benchmarking, training the pipeline, reviewing results and stress-testing the server under high load. Grafana alerts helped me monitor it, but I still wanted to see how the system behaved under real test conditions before trusting it.
The next step is moving parts of the pipeline from Python to C++ to reduce the impact of the GIL on parallel execution. Early tests show it is already reducing that bottleneck, though completing the migration will take time.
Where the textbook got it wrong
Most optimisation advice for GPU inference pipelines is sound in general. On our hardware and under our load, a couple of the obvious moves didn’t do what the textbook promised. Each time, the lesson was the same: measure what is actually happening before deciding what to optimise.
For context, the detection model runs on the GPU, while most of the preprocessing happens on the CPU. That split between a fast GPU and a busy host shapes everything below.
Detection and tracking on a busy store floor, where every frame has to be processed at scale. AI-generated image.
What was the GPU actually doing?
The first surprise was that the GPU wasn’t running flat out, even though it was at its power ceiling. This is what nvidia-smi showed under load:
| Metric | Value |
|---|---|
| Power draw | 63–72 W, against a 70 W limit |
| Power limit / max limit | 70 W / 70 W (it can’t be raised) |
| Throttle reason | SW Power Cap active; HW slowdown and thermal slowdown not active |
| SM clock | 1275–1335 MHz, against a 1590 MHz maximum (about 16–20% below boost) |
| GPU utilisation | 68–69% |
| Temperature | 77 °C |
The card was hitting its software power cap, not a thermal limit, so the driver was holding clocks 16–20% below boost. At the same time, utilisation sat at 68–69%. The GPU was spending its whole power budget without being fully busy, which pointed my investigation back at the host: what was the CPU doing, and was it feeding the GPU fast enough?
Was preprocessing really the bottleneck?
The textbook move is to offload letterbox resizing, normalisation, BGR-to-RGB conversion and the NCHW transpose to the GPU, where they are cheap. I had been tracking a profiling bucket labelled “preprocessing” that looked large enough to justify it.
When I broke it down, most of that bucket turned out to be the session.run call itself. The true preprocessing steps (resize, astype and divide by 255, cvtColor, transpose) were only a small share of host CPU time.
The lesson: profile at the granularity of the call, not the stage. One mislabelled bucket had been steering the whole roadmap.
Why was the CPU busy while the GPU did the work?
The next clue was strange: session.run used more CPU time than wall-clock time. By default, CUDA synchronisation busy-polls a CPU core while it waits for the GPU to finish, so a thread that should be idle is spinning.
Setting ONNX Runtime’s session.intra_op.allow_spinning to 0 fixed only part of it. The full fix was calling cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync) before the CUDA context is created, which makes the waiting thread sleep instead of spin. In isolation, host CPU per call dropped by about 40%.
What did these lessons have in common?
| Optimisation | Expected | What I found | Lesson |
|---|---|---|---|
| Move preprocessing to the GPU | Resize, normalise and transpose are cheap on the GPU, so offload them | The “preprocessing” bucket was mostly session.run; true preprocessing was a small share of CPU | Profile at the granularity of the call, not the stage |
| Default CUDA synchronisation | Waiting on the GPU costs the CPU nothing | CUDA busy-polled a CPU core; session.run used more CPU time than wall-clock time | Blocking sync before context creation cut host CPU per call by about 40% |
In both cases, the GPU wasn’t the real constraint. The host was. That is also what makes the move from Python to C++ worthwhile: the more work comes off the CPU and out from under the GIL, the more of the GPU’s limited power budget turns into useful throughput.
Manual first, automated later
My rule was simple: keep a process hands-on while it is still unproven, and only make it repeatable once it works reliably in our environment.
I applied that to benchmarking the pipeline, stress-testing it, monitoring it, creating camera zones and testing accuracy. Doing each one by hand first taught me how it actually behaved. That understanding is what made it possible to automate it safely and hand it off, instead of automating something I didn’t yet trust.
What AI changed, and what it didn’t
This wasn’t the project where I started using AI. I had already shipped features with AI-assisted workflows. What changed was where the saved time went, and how much of the product I took responsibility for.
AI assistance gave me room to spend more time on the questions around the code. What is the retailer actually trying to achieve? What happens when a camera setup changes? How do we test a new feature at scale? What needs to be configurable so onboarding doesn’t become a bottleneck?
What it didn’t change was the judgment. Deciding what to build, checking whether it worked in our environment, and deciding when a process was stable enough to automate stayed with me.
What still needs me
Not everything is solved, and it’s worth being honest about where the gaps are.
The Electron app removed the need for an engineer in most onboarding work, but it still needs someone who understands both the system and the model. That’s a big step forward, but it isn’t fully self-serve yet.
The C++ migration is still in progress, so the throughput gains are early results, not the final picture. And a strong POC conversion rate shows the product delivers value to new clients. Proving it holds up as more stores, features and data come online is exactly what the stress-testing and migration work are for.
Building the product is only the beginning
I started out thinking about features and technical problems. Over time, my work moved toward owning the product end to end, from use cases to infrastructure.
The shift was from asking, “How do I build this?” to asking, “What does this product need to do, how will it behave in the real world, and how can we make it work reliably as it grows?”
