Disclosure: I work on ClearWhirl, the iPhone photo cleaner this post is about. It is on the App Store: https://apps.apple.com/us/app/phone-cleaner-clearwhirl-ai/id6835310950
Short version: everything that looks at pixels runs through Apple's Vision framework on downscaled copies, on a capped dispatch queue, with results saved to a local index. Apple's on-device Foundation Models never see a photo. They do one job: turn "payment receipts from 2025" into a typed search filter. When the model is unavailable, search falls back to plain rules. Nothing leaves the phone, and the app works in airplane mode.
Two stages: metadata first, pixels second
Stage A reads only PHAsset metadata and fills several folders in seconds:
if a.mediaSubtypes.contains(.photoScreenshot) { f[.screenshots, default: []].append(id); continue } if a.mediaType == .video { if i.bytes > 100_000_000 || a.duration > 120 { f[.largeVideos, default: []].append(id) } continue } if a.representsBurst || a.mediaSubtypes.contains(.photoLive) && i.bytes > 6_000_000 { f[.bursts, default: []].append(id) } Favourites go to a "Keepers" folder and are protected from every other folder by default.
Stage B reads pixels, once per photo:
- a 256 px copy for
VNGenerateImageFeaturePrintRequest,VNClassifyImageRequestandVNDetectBarcodesRequest(QR, accepted at confidence ≥ 0.9 with a payload), - a 900 px copy for
VNRecognizeTextRequestwhen the labels suggest a receipt, ticket, document or similar, and for every screenshot. Small print is unreadable at thumbnail size. - a 128×128 grayscale Laplacian variance for sharpness, plus Vision's aesthetics score on iOS 18+.
Similar shots: neighbours in time, not all pairs
Comparing every photo with every other photo is quadratic. Most near-duplicates are taken seconds apart, so the scan walks photos in capture order and only compares each one with the previous one:
// Similar: compare with the previous photo if taken within 3 minutes. if let print, let pp = prevPrint, let p = prev, info.date.timeIntervalSince(p.date) < 180 { var d: Float = 99 try? print.computeDistance(&d, to: pp) if d < 0.42 { if currentGroup.isEmpty { currentGroup = [p.id] } currentGroup.append(info.id) } else { if currentGroup.count > 1 { groups.append(currentGroup) } currentGroup = [] } } The trade-off is explicit: two similar photos taken a day apart are not grouped. The best shot in a group is the highest aesthetics score where available, with sharpness as tie-break, and only the extras count towards "space you could free".
Lesson 1: do not block the Swift task pool
Our first version ran image loads and Vision requests inside Task.detached. The scan froze after the first batch. Synchronous, blocking work starves the cooperative thread pool. The fix was a plain dispatch queue with a semaphore, bridged with a continuation:
nonisolated static let workQueue = DispatchQueue(label: "cw.analysis", qos: .utility, attributes: .concurrent) nonisolated static let workLimit = DispatchSemaphore(value: 4) nonisolated static func onWorkQueue<T>(_ body: @escaping () -> T) async -> T { await withCheckedContinuation { (c: CheckedContinuation<T, Never>) in workQueue.async { workLimit.wait() let r = body() workLimit.signal() c.resume(returning: r) } } } Photos go through in batches of six with a TaskGroup, and results are put back in order so the "compare with the previous photo" rule still holds.
Lesson 2: background time is about 26 seconds unless you ask early
On the device, without a continued-processing task, iOS gave the app roughly 26 seconds after it left the foreground, then suspended it. We were submitting BGContinuedProcessingTaskRequest only after a 68-second phase that measures file sizes for a 10,624-asset library, so anyone who left the app during that phase froze the scan. Requests made from the background were also refused. Now the request is submitted before the size phase, from the foreground, with a strategy that fails rather than queues:
let request = BGContinuedProcessingTaskRequest(identifier: identifier, title: title, subtitle: subtitle(done, total, phase)) // Run now or not at all: a queued request could start long after the person left, with nothing on screen. request.strategy = .fail try BGTaskScheduler.shared.submit(request) A user pausing or the system expiring the task now completes it with success: true (a pause is not a failure), and one counter feeds the system banner, the Live Activity and the widget.
Lesson 3: a partial index must never replace a complete one
The facts per asset (labels, text, scores) are saved to a JSON index so later launches are instant. Saves happen in batches and replace the file atomically. A rule we unit-test: a partial pass never overwrites a complete index. An interrupted scan resumes only the assets that are not in the index.
Where Foundation Models come in (and where they do not)
The language model is text-only. It never receives an image. It is used to turn a search sentence into a typed filter with guided generation:
@Generable struct PhotoQuery { @Guide(description: "Folders to search. Allowed values: similar, screenshots, blurry, largeVideos, bursts, receipts, memes, unwanted. Empty if none fit.", .maximumCount(4)) var folders: [String] @Guide(description: "Words to find in text inside images, lowercase. Example: boarding pass gives boarding, gate, flight.", .maximumCount(5)) var textWords: [String] @Guide(description: "Exact four digit year if the person names one or says this year or last year, otherwise 0.") var year: Int @Guide(description: "True only if the person asks for videos.") var videosOnly: Bool @Guide(description: "Minimum size in megabytes if the person asks for big or large items, otherwise 0.") var minMegabytes: Int // ...scene labels, before/after year } The output is clamped before use: folder names must parse into the app's enum, a year must be between 1900 and this year, and a range with only one bound is dropped rather than guessed. A second schema maps a word to labels from the library's own label list only, with the instruction to never pick a broader label ("for flight choose airplane, never vehicle"). Broad Vision labels such as outdoor, building and vehicle matched too much to be useful in search, so they are excluded anyway.
Three practical details made it reliable:
- A fresh session per request keeps the 4,096-token context small. Overflow on a long query retries once with the first 120 characters.
- One request at a time. Overlapping requests failed with
concurrentRequests, so a tiny actor serialises them, and a newer query makes older waiting ones give up. - One retry after a second on
rateLimited,concurrentRequestsorassetsUnavailable. Any other error returnsnil, and the caller falls back.
let s = LanguageModelSession(instructions: instructions) return try await s.respond(to: prompt, generating: T.self).content The session is prewarmed when the search field gets focus, since the request is then at least a second away.
Fallback is the default, not the exception
SystemLanguageModel.default.availability has several unavailable reasons. We map them to a small pure enum so the wording and the fallback rule are unit-tested without the framework:
enum AIAvailability: Equatable { case available, forcedOff, unsupportedLanguage, deviceNotEligible, notEnabled, notReady, unknown var usesBasicSearch: Bool { self != .available } var worthShortRetry: Bool { self == .notReady } } Only notReady (the model preparing after a reboot or update) is worth a three-second wait. Everything else goes straight to basic search: labels, recognised text and rules, on the device. Settings shows plain words ("Turn on Apple Intelligence in Settings", "Not supported on this iPhone"), never a bare "unavailable".
Testing what you cannot run
Our development Mac is Intel, so Foundation Models cannot run on it at all, and Vision is slow (each model takes 1 to 2.5 minutes to load the first time; performing several requests together crashed, so they run one at a time there). What we do:
- Rule unit tests compiled with
swiftcin seconds. Every wrong result seen on a phone becomes a case ("adobe is not a DOB", "boardingpass means boarding pass"). - A labelled synthetic image set (314 samples across 11 labels) run through the same Vision requests and rules. Synthetic images are cleaner than real ones, so any score there is an upper bound.
- An on-device UI test that drives the app and saves screenshots, plus device logs.
- Widget snapshot tests on the simulator.
Results, with their limits
On one iPhone 15 Pro Max, the first scan of a 10,480-item library (about 8,000 photos and 770 screenshots) took roughly 10 minutes and grouped about 1,485 similar photos using 6.0 GB. One device, one library, not a benchmark. We have not published per-category accuracy or search precision on real photos, because we have not measured them in a way we are willing to publish. No battery or thermal numbers either.
If you are building something similar: keep blocking work off the task pool, ask for background time before the slow phase, treat the language model as an optional filter-builder with a typed schema, and make the no-model path a first-class product.
More on the approach in plain language: https://clearwhirl.appikorn.com/articles/how-on-device-vision-works-plain-language/ · The test write-up: https://clearwhirl.appikorn.com/case-studies/case-study-10480-item-test-library/

