Tormod Ree, chief product and engineering officer at Neat, describes meeting rooms with 11 or more cameras and dozens of microphones. The system has to tell who is present, who is talking, and who is looking at whom, then choose what a remote participant sees and hears. That choice sits on the live media path. Where you run it is the design decision.

Why does live audio and video have to run on the device?

Ree ranks the case for edge AI as time sensitivity first, privacy second, and cost third. He places cost behind the other two. Processing on the path of seeing and hearing people has to run on the device. That covers detecting people, detecting who is speaking, picking a camera, and mixing microphones. Neat uses both edge and cloud. The split follows the job.

Yes. So we do a combination of edge and cloud. We sort of do it where it makes sense, but typically sort of arguments for doing it at the edge in probably an order of importance is kind of time sensitivity, privacy and cost.

Tormod Ree, chief product and engineering officer at Neat

Privacy is a training constraint as well as a runtime one. The devices sit in rooms where the conversations are private. Neat cannot simply collect customer meetings as training data. Ree describes internally sourced material and relevant public datasets, with relatively little synthetic data. He ties that to local processing: what happens locally stays locally and is processed locally.

Slower work belongs in another bucket. Fleet management can wait. In one mode, audio summaries run in the cloud because they are less time sensitive. Neat processes that audio during the meeting or as the meeting closes. Google Meet, Microsoft Teams, and Zoom remain cloud meeting services. That cloud path is the call. The framing that makes the room watchable stays on the hardware in the room.

What counts as too slow when the output is a framing decision?

Latency for a language model separates time-to-first-token from the time until the model finishes. A framing decision has a different output: which video stream, which crop, which microphone mix. If it lands late, the remote person is looking at the wrong face or hearing the wrong mic. The latency glossary notes that a 100-millisecond interaction and a ten-second one are different products even at identical quality. Ree does not state a millisecond cap for the camera cut. I would set the budget from the failure you can see. If the step must finish while that person is still in frame and still talking, run it on the device next to the sensors. If the user needs the result only after the meeting, cloud is allowed.

Keep the coordinating device light. Ree describes a captain, usually at the front of the room, with microphones, cameras, speakers, and the screen. Other devices, including a unit at the table, each run their own audio or video inference. They send metadata along with media. The captain selects and mixes from that metadata, and can skip a second inference pass on the incoming streams. Inference and orchestration both stay at the edge. When one chip is not enough, he adds devices. A camera unit or a microphone unit still has compute, so the room picks up more of it.

The rules that turn those signals into a layout are still mostly deterministic. Someone talks, or the headcount crosses a threshold, and a fixed rule picks the experience.

the decisions that you make to produce the end, the result experience, those algorithms are still to a large extent deterministic, right?

Tormod Ree, chief product and engineering officer at Neat

He expects that director to become a trained model later. The shift has not happened. It needs probably more of this kind of data than most companies have, and a decent amount of compute. The devices are typically Android, in the class of a phone: the mobile compute available when that hardware was designed. A five-year life, which customers often expect, means living with something like a new phone from five years earlier. A Mac Mini with a lot of memory is a richer box, and a cloud GPU is another class entirely. Models he still runs locally have to stay narrow and targeted. I would keep the director as explicit rules until a trained model beats those rules on devices already in rooms.

How do you test speech and vision apart?

Ree says detectors that find people, and detectors that tell who is talking, have moved a long way. The harder layer is nonverbal signal, such as someone standing or presenting, and combining cameras and microphones so the room acts as one system rather than one powerful device at the front. A speech miss and a bad crop show up in different tests. Score them apart.

For speech, use word error rate. Align the recognizer output with a human transcript, count insertions, deletions, and substitutions, and divide by the number of words in the reference. That glossary cites rough bands for one speech service: 5 to 10% as good quality, around 20% as acceptable though it may need more training, and 30% or more as a sign it needs customization. The bands depend on the audio and the task, so treat them as a starting point. I would read the error mix before I read the single percentage. Many deletions usually trace to a weak signal, insertions to noise or crosstalk, and substitutions to domain terms the model has not seen enough of. Word error rate scores words. It can hold steady while the camera frames the wrong person.

For vision, label the events the product acts on: a person present, a person looking at someone, a person standing, a person presenting. Report misses and false alarms for each event. Room size, acoustics, and how people interact all vary, which is why the test has to be a real room. Ree says meeting-room scenes differ from a security camera, a car, or the sets you typically find online, so the model has to be trained and tested on relevant data.

# Sketch only: keep the speech and vision reports separate. def speech_report(insertions, deletions, substitutions, n_ref): wer = (insertions + deletions + substitutions) / n_ref return {"wer": wer, "deletions": deletions, "insertions": insertions} def vision_report(events): # events: name -> (misses, false_alarms, count) return { name: (misses + false_alarms) / count for name, (misses, false_alarms, count) in events.items() } 

When word error rate moves and the vision rates hold, look at the microphones and the speech model. When a vision event rate moves and word error rate holds, look at that detector. Publish the two reports together.

Checklist

  • Run every live audio and video step on the device. Leave fleet management and post-meeting audio summaries on a slower path.
  • Place each step by time sensitivity, then privacy, then cost.
  • Let each device emit metadata, and let one captain select and mix.
  • Score speech with word error rate and the error mix. Score each vision event with its own misses and false alarms.
  • Keep layout rules deterministic until a trained director wins on hardware already shipped.

The full conversation, with the transcript, is on Chain of Thought.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.