It started with a question that was probably not worth spending two days on:
Could I use my fingers to control the lights in my room from a distance?
I didn't know much about computer vision. I had barely touched AVFoundation, Vision, or Core Media. I had never built a gesture recognition system.
Which, naturally, made me want to try it.
The idea behind HandKit was simple: point my iPhone's front camera at myself, recognize a few hand gestures, and translate them into actions in my HomeKit home.
The first two interactions I wanted were: pinch my thumb and index finger together to toggle a light. Move my index finger horizontally to control brightness from 0 to 100%.
The final interaction looks almost trivial. A pinch. A lamp turns on. But getting from a raw camera feed to an intentional, reliable gesture turned out to involve camera pipelines, pixel buffers, coordinate systems, noisy measurements, a little geometry, signal filtering, state machines, hysteresis, Swift Concurrency, and finally HomeKit.
This article is a walkthrough of how I built it, but more importantly, why the architecture and algorithms evolved the way they did.
Starting with the camera
Before detecting a hand, I needed access to something much simpler: images.
On iOS, that starts with AVFoundation.
The capture pipeline can be simplified to:
AVCaptureDevice ↓ AVCaptureDeviceInput ↓ AVCaptureSession │ ├────────→ AVCaptureVideoPreviewLayer │ ↓ │ Preview │ └────────→ AVCaptureVideoDataOutput ↓ Video frames Each component has a different responsibility.
AVCaptureDevice represents the physical capture device. In my case, the iPhone's front-facing camera.
AVCaptureDeviceInput turns that device into an input that can be attached to a capture session.
AVCaptureSession coordinates the capture pipeline.
From there, I needed the video in two different places. The first path goes to an AVCaptureVideoPreviewLayer, so I can see the camera feed on screen. The second goes to an AVCaptureVideoDataOutput, because displaying the video isn't enough. I need access to the individual frames if I want to analyze what my hand is doing.
That distinction became the foundation of HandKit:
Camera │ ├──→ human sees the preview │ └──→ application analyzes the frames Managing the capture session with Swift Concurrency
I didn't want camera lifecycle operations scattered throughout the UI, so I introduced a CameraManager. It owns the capture session and is responsible for configuring, starting, and stopping it.
I implemented it as an actor.
actor CameraManager { let session = AVCaptureSession() // ... } Actors are useful here because they provide isolation around mutable state.
One thing I had to properly understand while building this was that an actor is not a thread.
An actor defines an isolation domain. Its executor determines where jobs isolated to that actor execute.
For the capture session, I wanted serialized execution on a dedicated queue, so I gave CameraManager a custom serial executor.
Conceptually:
CameraManager actor ↓ Custom SerialExecutor ↓ camera.session queue ↓ AVCaptureSession A simplified version looks like this:
actor CameraManager { private let sessionQueue = DispatchSerialQueue(label: "camera.session") nonisolated var unownedExecutor: UnownedSerialExecutor { sessionQueue.asUnownedSerialExecutor() } // ... } This gave the camera pipeline a clear execution context without turning the MainActor into the place where capture work happens.
I also kept configuration inside the manager.
From outside, starting the camera shouldn't require knowing:
- configure first
- then start
- but don't configure twice
That is an internal invariant.
So CameraManager tracks whether the session has already been configured and exposes a higher-level start() operation.
This was one of the first architectural principles that kept appearing throughout the project:
The caller should express an intention, not reproduce the internal procedure required to achieve it.
Keeping session management and frame processing separate
The next step was receiving frames from AVCaptureVideoDataOutput.
AVFoundation provides them through AVCaptureVideoDataOutputSampleBufferDelegate.
I initially considered making CameraManager the delegate too, but there was both a technical and architectural reason not to.
The delegate participates in Objective-C's NSObjectProtocol, while my camera manager is an actor. More importantly, managing the lifecycle of the camera and processing every video frame are different responsibilities.
So I introduced VideoOutputDelegate.
CameraManager │ ├── configure / start / stop ├── AVCaptureSession │ └── VideoOutputDelegate ↓ frames ↓ Vision I also separated their execution queues.
camera.session ├── configure ├── start └── stop frame.session ├── frame ├── frame ├── frame └── Vision processing This matters because analyzing an image can take time. I don't want that work unnecessarily blocking operations on the capture session itself.
It also gave the code a much cleaner conceptual boundary:
CameraManager manages capture. VideoOutputDelegate interprets what was captured.
A video is just a lot of images arriving very quickly
The delegate eventually receives:
func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection ) The important value here is the CMSampleBuffer.
I initially thought of a camera frame as essentially an image, but the media pipeline is richer than that.
A CMSampleBuffer is a container for a media sample. For video, it carries the sample and associated information such as timing and format metadata.
For Vision, what I really needed was the image buffer inside it:
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } That gives me a CVPixelBuffer. The pipeline had now become:
Camera ↓ AVCaptureVideoDataOutput ↓ CMSampleBuffer ↓ CVPixelBuffer ↓ Vision There is no need to turn every frame into a UIImage. The pixel buffer can go directly into Vision.
Asking Vision to find a hand
Apple's Vision framework provides VNDetectHumanHandPoseRequest.
Instead of recreating the request for every frame, I keep one instance around and ask it to detect at most one hand.
private let handPoseRequest: VNDetectHumanHandPoseRequest = { let request = VNDetectHumanHandPoseRequest() request.maximumHandCount = 1 return request }() For each pixel buffer, I create a request handler:
let handler = VNImageRequestHandler( cvPixelBuffer: pixelBuffer, orientation: .right, options: [:] ) try handler.perform([handPoseRequest]) Once Vision detects a hand, it returns a VNHumanHandPoseObservation. From that observation, I can request specific landmarks:
let indexTip = try hand.recognizedPoint(.indexTip) let thumbTip = try hand.recognizedPoint(.thumbTip) Vision doesn't return screen pixels. The locations are normalized coordinates, which makes them independent of a specific image resolution. So now I had something incredibly useful:
index fingertip → (x, y) thumb fingertip → (x, y) That sounds like enough to build gestures. Well it wasn't lol
Vision gives estimates, not truth tho
Each recognized point also has a confidence value. Very quickly, I discovered why.
I printed the index coordinates while moving my finger around. When Vision was confident, I would get values like:
x: 0.578 y: 0.300 confidence: 0.872 x: 0.577 y: 0.311 confidence: 0.867 x: 0.575 y: 0.328 confidence: 0.855 But occasionally the tracking would produce a large jump with very low confidence.
If I interpreted every coordinate as equally trustworthy, a tracking mistake could look like an extremely fast gesture.
My first filter therefore became:
guard indexTip.confidence >= 0.7 else { return } 0.7 isn't a universal Vision constant. It was an experimental starting point based on the data I was observing.
That distinction matters throughout HandKit: most thresholds aren't presented as magic values that should work for everybody. They're values I measured and tuned while experimenting with my own setup.
But confidence filtering wasn't enough.
Even when Vision was confident and my finger looked completely still to me, its coordinates continued to move slightly.
That led to the first genuinely interesting algorithmic problem.
How do you measure movement when "not moving" still moves?
To know whether my finger is moving, I need more than its current position. I need its previous one. For two consecutive measurements:
previous = (x₁, y₁) current = (x₂, y₂) I calculate:
Δ x = x_2 - x_1 Δ y = y_2 - y_1 But I also wanted one value representing how much the point moved regardless of direction.
That's just the hypotenuse of the triangle formed by the two deltas:
d = sqrt((Δ x)^2 + (Δ y)^2) Swift already gives me exactly that operation:
let movementDistance = hypot(deltaX, deltaY) This was my first little return to the Pythagorean theorem for a project involving smart lights. The interesting discovery came when I kept my index as still as possible. movementDistance wasn't zero.
I saw values around:
0.001 0.003 0.002 0.005 0.001 ... and occasional larger spikes. The system needed to distinguish actual intentional movement from natural tracking jitter.
A single frame wasn't enough to make that decision.
Filtering the tracking signal
Instead of considering one movement measurement at a time, I started keeping a sliding window containing the five most recent distances.
[d1, d2, d3, d4, d5] When a sixth arrives:
[d1, d2, d3, d4, d5, d6] I remove the oldest:
[d2, d3, d4, d5, d6] In Swift:
recentMovementDistances.append(movementDistance) if recentMovementDistances.count > 5 { recentMovementDistances.removeFirst() } Then I calculate the median. For five sorted values:
[0.001, 0.002, 0.003, 0.005, 0.050] ↑ median That was useful because an isolated bad measurement such as 0.050 doesn't drag the result upwards as strongly as it would affect an average.
This is deliberately a very small and simple filter. HandKit isn't trying to be a sophisticated signal-processing library.
But it was enough to introduce an important idea: a noisy measurement becomes much more useful when you reason about it over time instead of treating every sample independently.
And that observation led directly to state machines.
Detecting movement with hysteresis
My first instinct could have been:
movement > threshold → moving movement < threshold → idle But imagine the signal oscillating around that threshold:
0.0069 0.0071 0.0068 0.0072 The state could rapidly become:
IDLE MOVING IDLE MOVING IDLE Instead, I introduced two different thresholds. From my experiments, I started with:
median > 0.01 → start moving median < 0.005 → become idle Between them, I keep the current state.
> 0.01 ┌───────────────────────┐ │ ↓ IDLE MOVING ↑ │ └───────────────────────┘ < 0.005 This is hysteresis. The gap between the two thresholds prevents the state from constantly flipping when the signal sits near a boundary.
In code, I represent the state explicitly:
enum MovementState { case idle case moving } That small choice became useful because I wasn't really trying to answer:
Is there movement in this exact frame?
I was answering:
Has an intentional movement started, and are we currently inside it?
That is a temporal question.
A gesture is something that happens over time
Originally, I started thinking about recognizing a swipe. A single position obviously can't represent a swipe.Even two consecutive positions aren't enough. A gesture has a beginning, a trajectory, and an end.
So when transitioning from .idle to .moving, I store:
startPosition startTime using a ContinuousClock for elapsed time measurement.
When movement eventually ends, I can compare:
startPosition → endPosition startTime → endTime and derive:
totalDeltaX = endX - startX totalDeltaY = endY - start plus the gesture duration.
This gave me the information required to reason about a swipe:
- Did it travel far enough horizontally?
- Did it remain reasonably horizontal?
- Did it happen quickly enough?
But testing this exposed a completely different bug.
The horizontal gesture that somehow moved vertically
I performed horizontal movements and looked at my data.
Instead of getting:
|Δx| >> |Δy| I repeatedly got large changes on Y. Then I performed an intentionally vertical gesture. This time X changed dramatically. My coordinate system was effectively rotated relative to what I was seeing.
The culprit was this:
orientation: .up I had initially told Vision to interpret the pixel buffer as if it were already oriented upright.
The preview looked fine, which made this easy to miss.
But the orientation of what the user sees and the orientation in which Vision interprets the raw image buffer aren't necessarily the same thing.
I tested .right. Then repeated the horizontal gestures. This time I got measurements such as:
Δx = -0.217 Δy = +0.013 Δx = +0.263 Δy = -0.040 Δx = -0.278 Δy = +0.018 Exactly what I wanted:
|Δx| >> |Δy| I also explicitly tested direction with the front-facing camera.
In my current configuration:
Δx > 0 → visual movement towards the left Δx < 0 → visual movement towards the right Again, I preferred measuring the behavior rather than assuming what mirroring would do.
Realizing I didn't actually want a swipe (yes, kinda late to think that)
While designing the swipe, I realized something else.
For brightness control, a swipe is the wrong interaction.
A swipe is a discrete gesture:
gesture ↓ recognized ↓ perform action once That's great for something like swipe → next track
Brightness is continuous.
What I actually wanted was a gesture-controlled slider:
finger starts moving ↓ keep tracking horizontal position ↓ brightness changes continuously ↓ finger stops So the same .idle / .moving state machine remained useful, but the meaning of the moving state changed.
Instead of waiting for the gesture to end before doing something, I calculate brightness continuously while movement is active.
There's another subtle decision here.
I could update brightness using each frame's delta:
brightness += currentDelta But then tracking errors can accumulate over time.
Instead, at the beginning of the gesture I store:
startPosition startBrightness Every subsequent value is calculated relative to those fixed references:
Δ x = currentX - startX brightness = startBrightness + f(Δ x) This means that if I move away and then return exactly to my starting position, I should return to the original brightness regardless of the intermediate path.
That avoids cumulative drift.
The current prototype maps the normalized horizontal displacement to a value between 0 and 100 and clamps the result:
let newBrightness = startBrightness + brightnessDelta brightness = min(max(newBrightness, 0), 100) Detecting a pinch with geometry
The second gesture was simpler conceptually. I already had the index fingertip. Vision can also give me the thumb tip.
let indexTip = try hand.recognizedPoint(.indexTip) let thumbTip = try hand.recognizedPoint(.thumbTip) After applying the same confidence filtering, I have two normalized points.
So detecting how close the fingers are is again a geometry problem:
Δ x = indexX - thumbX Δ y = indexY - thumbY distance = sqrt((Δ x)^2 + (Δ y)^2) or:
let distance = hypot(deltaX, deltaY) The confidence check needs to cover both points. A reliable index position doesn't help if the thumb position is uncertain:
guard indexTip.confidence >= 0.7, thumbTip.confidence >= 0.7 else { return } let deltaX = indexTip.location.x - thumbTip.location.x let deltaY = indexTip.location.y - thumbTip.location.y let distance = hypot(deltaX, deltaY) This time, the distance compares two landmarks in the same frame. Earlier, I used the same calculation to compare one landmark across consecutive frames. The geometry is identical; the question is different.
Before deciding what counted as a pinch, I measured the distance while opening and closing my fingers. In my setup, the observations were roughly:
Fingers apart: 0.15 to 0.21 Fingers touching: 0.005 to 0.015 Contact didn't produce a perfect zero. Vision estimates the locations of landmarks in a two-dimensional image, and those estimates still contain noise when the fingers touch.
These are distances in normalized image coordinates, not centimeters or measurements of physical contact. Normalizing the coordinates removes the dependence on pixel resolution; it doesn't make the measurement independent of hand size, camera distance, or image geometry. Moving my hand farther from the camera changes its apparent size and therefore the distance between the detected points.
The measurements were enough to establish a useful threshold for this prototype, but they weren't a general calibration for every hand and camera setup.
A pinch needs memory too
The movement detector had already taught me what happens when a noisy measurement sits near one threshold.
For the pinch, I reused hysteresis with a separate state machine:
enum PinchState { case open case pinched } The rules became:
OPEN -- distance < 0.015 --> PINCHED PINCHED -- distance > 0.03 --> OPEN Between the two thresholds, I preserve the current state. At either exact boundary, the strict comparisons also leave the state unchanged.
That gap matters. After recognizing a pinch, I don't want a small increase from 0.014 to 0.016 to mean that the hand has opened again. The fingers have to separate more clearly before another pinch can begin.
But there was one more distinction to make: the state and the event are different things.
If I called the toggle action on every frame where the distance was small enough, holding my fingers together would repeatedly switch the lamp on and off.
The event I care about is the transition:
OPEN -> PINCHED -> emit one toggle
The core logic can be expressed as:
switch pinchState { case .open: if distance < 0.015 { pinchState = .pinched onToggle() } case .pinched: if distance > 0.03 { pinchState = .open } } Once the state becomes .pinched, subsequent frames stay in that branch until the fingers reopen. Releasing the pinch rearms the interaction; it doesn't toggle the light.
This gave me a specific behavior to test: pinch, hold, release, pinch again. Holding should produce no additional action, and the second pinch should produce exactly one new action.
Hysteresis stabilizes the transition, but it doesn't solve every tracking problem. For example, losing the hand while it is pinched raises a separate question about when to reset or rearm the recognizer. That is part of making the experiment more robust, beyond this first working interaction.
Finding the actual light with HomeKit
At this point, the gesture pipeline could produce a toggle event. I still needed to connect that event to something outside the phone.
HomeKit introduced another hierarchy to understand:
HMHomeManager | +-- HMHome | +-- HMAccessory | +-- HMService (Lightbulb) | +-- HMCharacteristic (PowerState) +-- HMCharacteristic (Brightness) HMHomeManager provides access to the configured homes. Each home contains accessories, and each accessory exposes services. The values I can inspect or change belong to the characteristics of those services.
Apple's HMCharacteristic documentation describes that final layer.
I began by inspecting what my own home actually exposed.
Within the service, the two characteristic types I needed were:
HMCharacteristicTypePowerStateHMCharacteristicTypeBrightness
PowerState represents whether the light is on. Brightness represents its brightness as an integer percentage of the maximum, as defined by Apple's brightness characteristic documentation.
Finding those characteristics didn't yet prove that I could control the lamp. I first asked HomeKit to read their values.
The actual output was:
BRIGHTNESS: 100 POWER: 1 Now I knew that I had found the intended accessory and could read its state: the lamp was on, with brightness at 100%.
Testing one layer at a time
It was tempting to connect the pinch immediately. Instead, I worked through four small steps:
Read state -> Manual write -> Temporary button -> Gesture
The first read established that discovery worked.
Giving HomeKit its own controller
I put HomeKit discovery and control in LightController.
Its job was to find Lamp, retain the characteristics the app needed, and expose an operation such as toggle() to the rest of the application.
The references begin as optional values because HomeKit discovery is asynchronous:
private var powerCharacteristic: HMCharacteristic? private var brightnessCharacteristic: HMCharacteristic? They refer to live HomeKit objects. They aren't preferences to persist in UserDefaults, and keeping them avoids traversing the entire home hierarchy on every gesture.
Once discovery had found the power characteristic, toggle() followed this sequence:
Power characteristic available? | v Read current value | v Convert to Bool and invert | v Write new value | v Check completion for an error I read the value before toggling because the lamp could also be changed from the Home app or another controller. A local Boolean would only describe what HandKit last believed, which might no longer match the accessory.
There was also an Objective-C interoperability detail: HMCharacteristic.value is exposed as Any?. In my tests, the power value arrived as an NSNumber, so I used its boolValue before inverting it:
guard let value = powerCharacteristic.value as? NSNumber else { return } let newState = !value.boolValue A read followed by a write still isn't an atomic toggle. Another command could arrive between them, and rapid gestures could overlap requests. Serializing device commands is a further robustness step; the working demo doesn't establish that all concurrent-control cases are handled.
At this snapshot, retaining the brightness characteristic prepared the second interaction. It did not mean that the gesture-controlled brightness value was already being written to the lamp.
Keeping recognition independent from the action
The frame-processing code knows about landmarks, distances, and gesture states. It doesn't need to know what a HomeKit accessory is.
VideoOutputDelegate communicates through two callbacks:
private let onToggle: @Sendable () -> Void private let onBrightnessChanged: @Sendable (CGFloat) -> Void One emits a discrete event. The other emits a continuously updated value.
CameraManager receives the toggle callback from the application and passes it to the delegate. The application decides that this event should call LightController.toggle().
That boundary means I can inspect or log the gesture output without a connected lamp. It also means the same pinch recognizer could eventually trigger something else without changing the geometry or thresholds.
There is still a deliberate prototype-level coupling here: onBrightnessChanged names the value in terms of the current interaction. It abstracts away HomeKit, but it isn't a completely generic gesture API. If I extracted this into a reusable component, a normalized slider value or displacement event might be a better contract.
For HandKit, the useful separation was already clear:
- Recognition: What did the hand do?
- Application: What should that mean?
- HomeKit: How do I ask the lamp to do it?
Crossing from the frame queue to the MainActor
These callbacks also exposed a concurrency boundary. Frames arrive on frameQueue. The UI-facing model and, in this project, LightController are isolated to the MainActor.
Calling lightController.toggle() directly from the synchronous toggle callback produced an isolation error.
The callback needed an explicit handoff:
onToggle: { Task { @MainActor in lightController.toggle() } } The brightness callback uses the same pattern:
onBrightnessChanged: { newBrightness in Task { @MainActor in viewModel.brightness = newBrightness } } @Sendable describes a closure's ability to cross concurrency boundaries with safe captures. It doesn't dispatch the closure onto the main actor. The Task { @MainActor in ... } body establishes that execution context explicitly.
The distinction follows Swift's concurrency and isolation model.
The values crossing this boundary are small: an event or a brightness number. The pixel buffers, Vision request, and recognition state remain in the frame-processing path.
Moving the callback onto the main actor also doesn't make the physical device operation synchronous. HomeKit still completes its read and write asynchronously. Actor isolation protects access to application state; it doesn't guarantee that the lamp has already reacted.
For this prototype, the callback handoff made the two paths explicit. If updates become more frequent than the UI or accessory can use, the next step is to coalesce them rather than create an ever-growing stream of pending work. Independent tasks also shouldn't be used as a guarantee of event ordering.
One shared model for the camera and the interface
The on-screen brightness indicator reads from a CameraViewModel that is both observable and main-actor isolated:
@MainActor @Observable final class CameraViewModel { var brightness: CGFloat = 0 } This reduced example shows the two separate responsibilities.
@Observablelets SwiftUI track changes to the state it reads. @MainActor defines where that mutable state is accessed. Observation alone doesn't establish concurrency isolation.
There was an equally important ownership detail: ContentView and CameraManager needed the same model instance.
If each created its own CameraViewModel, the camera callback could update one object while SwiftUI observed another. The gesture calculations would be correct, but the indicator would never reflect them.
In the view's initialization, the wiring therefore follows this shape:
init() { let viewModel = CameraViewModel() let lightController = LightController() self._viewModel = State(initialValue: viewModel) self.lightController = lightController self.cameraManager = CameraManager( viewModel: viewModel, onToggle: { Task { @MainActor in lightController.toggle() } } ) } This is an excerpt of the wiring, with the surrounding stored properties and view body omitted. The important part is that the local viewModel passed to CameraManager is the same reference retained for the view's state. Inside the manager, the brightness callback updates that reference on the main actor.
The resulting feedback path is:
Vision -> slider value -> callback -> MainActor | v CameraViewModel | v SwiftUI indicator That shared instance is what connects a correct calculation to something visible on screen.
The complete architecture
By the end of this iteration, the system looked like this:
ContentView creates and connects objects | v CameraManager actor | custom SerialExecutor | sessionQueue | AVCaptureSession | +----------------+------------------+ | | v v AVCaptureVideoPreviewLayer AVCaptureVideoDataOutput | | v v Camera preview frameQueue | v VideoOutputDelegate | v CMSampleBuffer -> CVPixelBuffer | v Vision hand-pose request | v Confidence-filtered points | +------------------+----------------+ | | v v Movement / slider Pinch state | | v v onBrightnessChanged(value) onToggle() | | v v Task { @MainActor in ... } Task { @MainActor in ... } | | v v CameraViewModel LightController shared with ContentView | | v v HomeKit PowerState SwiftUI 0-100 indicator | v Physical Lamp Next connection, not yet implemented at this snapshot: slider value -> HomeKit Brightness -> physical light intensity The boundaries each answer a practical question. The camera manager owns capture lifecycle. The delegate ...
