Amer Sheeny breaks down the performance optimization of a document-scanning app powered by LLM pipelines, detailing latency metrics, model version shifts, and timeout handling.
The app scans photographed pages and returns the recognized text via Firebase. It previously used Google Cloud Vision to read the text and a separate OpenAI service for titles and corrections. The newer version uses an artificial intelligence model for the entire process. It returns the language, the original and corrected text, as well as a title and, optionally, a summary and additional information.
The model that delayed scans
With gemini-2.0-flash-lite, the median daily scan time was 6,6–7,4 seconds. 95% of scans completed in 14–17 seconds. After switching to gemini-2.5-flash-lite, the respective times increased to 8–9,5 and 20–29 seconds. The slowest 1% reached the 60-second mark daily.
Over a two-week period, at least 200 scans failed after 59,99 seconds. The retry process could result in up to six model calls for a single scan.
Sheeny tested the same model from a different connection point to the service, but it was just as slow. This showed that the region from which the connection was being made was not the cause and suggested that the model was at fault.
↳
READ ALSO
HACKS.GR / RELATED
Tech Guides
THORChain: exchange Bitcoin for Ethereum without an account or identification
↗
Switching to gemini-3.1-flash-lite roughly halved the times. On a tax document scan, the time dropped from 13,6–15,5 to 7,3–12,9 seconds. There were no failures in 30 tests.
What did the measurements show?
The first change was to increase the time limit of the function that processes scans from 60 to 300 seconds. Each attempt has its own time limit, and a new attempt is only started if there are at least 12 seconds left.
Measuring only successful scans hid the problem. Of the last 1.000 successful scans, the slowest 1% completed in 26,9 seconds. If failed attempts were also counted, the corresponding percentage reached 60 seconds. So the slowest cases were left out of the picture when only successes were recorded.
Sending lower resolution images reduced the input received by the model by 25%, but did not change the waiting time. In contrast, changing the model significantly affected the times, even though the rest of the code remained the same.
Each instance of the processing function can handle up to 80 requests at a time by default. You also need to take into account the different wait times of the application and the service. The default settings are 70 seconds for Android and 60 for Flutter. In Sheeny's example, the service had a timeout of 290 seconds.
Sheeny suggests recording the times along with the failures and comparing them to the application's change history. He also suggests testing one change at a time and re-evaluating each model change. In this case, the metrics showed that the problem was associated with the slower version of the model.
>Stay connected with Hacks.gr
Set us as a preferred source on Google and find us on social media.
Preferred source
↗
end of article

