FiftyOne multimodal viewer playing an Aria Digital Twin MCAP episode, with the semi-dense point cloud, 3D bounding boxes, and synchronized synthetic, depth, segmentation, and RGB camera tiles on one timeline.

FiftyOne multimodal viewer playing an Aria Digital Twin MCAP episode, with the semi-dense point cloud, 3D bounding boxes, and synchronized synthetic, depth, segmentation, and RGB camera tiles on one timeline.

Download one Aria Digital Twin episode, author every ground-truth stream it ships with into a single synchronized MCAP file, and load it into FiftyOne’s multimodal viewer, using Meta’s own capture pipeline and about 300 lines of Python.

Aria Digital Twin (ADT) is a research dataset from Meta Reality Labs: real people performing everyday activities in two fully digitized rooms, recorded with Project Aria smart glasses and paired with millimeter-accurate six degrees of freedom (6DoF) ground truth for every object, person, and device in the scene. It ships as a .vrs recording plus eight separate ground-truth zip files, each behind its own presigned URL. None of that is something FiftyOne (an open-source toolkit for visualizing and curating ML datasets; pip install fiftyone) opens directly.

What FiftyOne’s multimodal viewer wants is one file per episode: a single timeline where the RGB video, depth, segmentation, synthetic renders, 2D and 3D object boxes, human skeletons, eye gaze, the semi-dense point cloud, the device’s 6DoF trajectory, and the inertial measurement unit (IMU) stream all play back in sync. That file is an MCAP: a container format for timestamped, multi-channel binary data, the same idea as a ROS bag but not tied to ROS.

Here’s the whole path from Meta’s download page to a dataset you can scrub through in FiftyOne, for one Apartment-scene episode, with every modality ADT ships turned into its own MCAP channel.

Keep it honest. Aria Digital Twin (Meta Reality Labs Research, arXiv:2306.06362) is licensed for personal, non-commercial research only, and the license explicitly bars distributing or publicly displaying the dataset “or any materials or works derived from it.” You need to accept that license yourself through Meta’s Aria Dataset Explorer before you get download links. This post shows code, not a copy of the data.

Key takeaways

  • Aria Digital Twin (236 sequences as of this writing, up from the 200 in the original paper) ships as a raw .vrs recording plus ground-truth zips, served from presigned URLs gated by Meta’s license, six of which this pipeline actually reads.
  • One episode with every modality is a real download: 12.4 GB across seven files for a 145-second sequence, with the raw depth zip alone accounting for 7.8 GB of that.
  • AriaDigitalTwinDataProvider hands you per-frame images, boxes, skeletons, and gaze by timestamp; twelve foxglove channels turn those into one .mcap file FiftyOne plays natively, with no exporter or custom loader.
  • Two silent gotchas: ADT’s world frame is Y-up while Foxglove/ROS 3D viewers assume Z-up, and writing each modality in its own loop desyncs playback unless you re-sort every message by log_time afterward.
  • A third gotcha only 3D boxes hit: 304 of this episode’s 354 tracked objects never move. Redrawing all of them every frame made that one channel 77 MB; logging the static ones once and only redrawing the 50 that actually move is the fix.
  • The payoff: a 145-second episode becomes one 2.5 GB .mcap with 181,444 messages across 12 channels, synchronized, one fo.Sample(filepath=...) call away from FiftyOne’s multimodal viewer.

Installation

pip install foxglove-sdk projectaria-tools mcap pillow numpy pandas fiftyone

Every code block in this post ran against foxglove-sdk 0.26.0, projectaria-tools 2.2.0, mcap 1.3.1, pillow 11.3.0, numpy 2.4.6, pandas 3.0.0, and fiftyone 1.21.0. You’ll also need curl on your PATH for Step 1’s downloads.

Step 1: Download one Aria Digital Twin episode

Request access to ADT through the Aria Dataset Explorer or the dataset download page, and accept the license. Meta returns a single JSON file, commonly called a “CDN file,” with two top-level keys: sequences (every sequence name, each mapping component name to a presigned download URL, file size, and checksum) and sequence_config (which components exist). Save it locally; this post calls it adt_download_urls.json, but yours may be named differently. manifest["sequences"].keys() lists every sequence name available to pick from, including Apartment_release_multiskeleton_party_seq101_M1292, the one used throughout this post.

Meta ships an official downloader for this file: pip install projectaria-tools (already in this post’s Installation step) installs an aria_dataset_downloader command that reads a CDN file directly:

aria_dataset_downloader \ --cdn_file adt_download_urls.json \ --output_folder adt_raw \ --sequence_names Apartment_release_multiskeleton_party_seq101_M1292 \ --data_types main_vrs main_groundtruth mps_slam_trajectories mps_slam_points segmentation depth synthetic

That downloads the raw files. Every modality below needs its own component from the CDN file:

Component Backs Size (this episode)
main_vrs RGB/SLAM images, IMU, calibration 2,341 MB
main_groundtruth 2D/3D boxes, instances, skeleton files, raw eye gaze 97 MB
mps_slam_trajectories Device 6DoF pose 15 MB
mps_slam_points Semi-dense point cloud 624 MB
segmentation Instance segmentation video 573 MB
depth Per-pixel metric depth video 7,792 MB
synthetic Photo-realistic synthetic render 918 MB

The seven ADT components this pipeline downloads, what each one backs in the final MCAP, and its size for Apartment_release_multiskeleton_party_seq101_M1292

To see exactly what’s happening, and to go straight into the flat, extracted layout this pipeline expects, here’s the equivalent in plain Python instead of the CLI:

import json import subprocess import zipfile from pathlib import Path manifest = json.loads(Path("adt_download_urls.json").read_text()) entry = manifest["sequences"]["Apartment_release_multiskeleton_party_seq101_M1292"] # Only the components this pipeline actually reads. The manifest offers more # per sequence (mps_slam_calibration, mps_eye_gaze) -- see "Good to know" below. NEEDED = [ "main_vrs", "main_groundtruth", "mps_slam_trajectories", "mps_slam_points", "segmentation", "depth", "synthetic", ] raw_dir, flat_dir = Path("adt_raw"), Path("adt_flat") # raw_dir: downloaded files as-is; flat_dir: zip contents merged flat raw_dir.mkdir(exist_ok=True) flat_dir.mkdir(exist_ok=True) for key in NEEDED: info = entry[key] dst = raw_dir / info["filename"] subprocess.run(["curl", "-sS", "-L", "-o", str(dst), info["download_url"]], check=True) if key != "main_vrs": # main_vrs is a single .vrs file, not a zip -- nothing to extract  with zipfile.ZipFile(dst) as zf: zf.extractall(flat_dir) main_vrs_path = raw_dir / entry["main_vrs"]["filename"]

Good to know. Why not mps_slam_calibration or mps_eye_gaze? They’re on Meta’s own recommended download list, but nothing in this pipeline reads online_calibration.jsonl or the MPS copy of general_eye_gaze.csv: eye gaze comes from eyegaze.csv in main_groundtruth, and camera calibration comes from the VRS file’s own embedded calibration, not the MPS refinement pass. AriaDigitalTwinDataPathsProvider.get_datapaths() doesn’t reference either file, which is how to check this for any zip you’re tempted to skip.

AriaDigitalTwinDataProvider expects a specific directory layout: video.vrs plus an mps/slam/ subfolder. Symlink everything into that layout:

# AriaDigitalTwinDataProvider expects a fixed layout: video.vrs at the # sequence root, MPS SLAM outputs under mps/slam/, and every ground-truth # file (2D/3D boxes, instances.json, skeleton files) flat at the root. seq_dir = Path("adt_flat_structured") (seq_dir / "mps" / "slam").mkdir(parents=True, exist_ok=True) (seq_dir / "video.vrs").symlink_to(main_vrs_path.resolve()) for name in flat_dir.iterdir(): (seq_dir / name.name).symlink_to(name.resolve()) # 2d/3d boxes, instances.json, skeleton files  for name in ["closed_loop_trajectory.csv", "semidense_points.csv.gz"]: (seq_dir / "mps" / "slam" / name).symlink_to((flat_dir / name).resolve())

Step 2: Author the MCAP

This continues in the same script as Step 1: seq_dir is the directory you just built there. Everything below writes into one foxglove.open_mcap(...) block. Set up the providers, instance metadata, and camera geometry first, since several streams share them:

import io from math import tan import foxglove import numpy as np import pandas as pd from foxglove import messages as msgs from PIL import Image from projectaria_tools.core.stream_id import StreamId from projectaria_tools.core.sensor_data import TimeDomain from projectaria_tools.projects.adt import ( AriaDigitalTwinDataPathsProvider, AriaDigitalTwinDataProvider, AriaDigitalTwinSkeletonProvider, MotionType, bbox3d_to_line_coordinates, ) # Aria VRS stream IDs are fixed per device, not per episode: 214-1 is the # RGB camera, 1202-1 is the right IMU. RGB_STREAM = StreamId("214-1") IMU_STREAM = StreamId("1202-1") # paths_provider resolves every file path under seq_dir; gt_provider wraps # both the raw VRS (images, IMU) and every ground-truth file (boxes, # skeletons, gaze, poses). paths_provider = AriaDigitalTwinDataPathsProvider(str(seq_dir)) gt_provider = AriaDigitalTwinDataProvider(paths_provider.get_datapaths()) # The master per-frame clock: every other stream below gets matched against # these timestamps, one way or another. ts_rgb = gt_provider.get_aria_device_capture_timestamps_ns(RGB_STREAM) # The time window ADT's ground truth actually covers -- can be a frame or # two narrower than the RGB recording itself. adt_t_start, adt_t_end = gt_provider.get_start_time_ns(), gt_provider.get_end_time_ns() traj_df = pd.read_csv(seq_dir / "mps" / "slam" / "closed_loop_trajectory.csv") # Built once, reused for every 2D/3D box and skeleton entity below: # per-instance metadata (name, motion_type, human vs. object) never changes # mid-episode. instance_info_cache = { uid: gt_provider.get_instance_info_by_id(uid) for uid in gt_provider.get_instance_ids() } cam_calib_rgb = gt_provider.get_aria_camera_calibration(RGB_STREAM) raw_provider = gt_provider.raw_data_provider_ptr() # Needed to project 3D eye gaze into 2D pixel coordinates on the RGB image. transform_cpf_sensor = raw_provider.get_device_calibration().get_transform_cpf_sensor( cam_calib_rgb.get_label() ) # CPF = Aria's eye-tracking reference frame, centered between the two eyes skeleton_ids = gt_provider.get_skeleton_ids() # one id per tracked person joint_connections = AriaDigitalTwinSkeletonProvider.get_joint_connections() # joint-index pairs forming each limb

Good to know. Doesn’t MCAP need real calendar timestamps for log_time? No. log_time just needs to be in nanoseconds on a single consistent clock; MCAP doesn’t require wall-clock Unix epoch time. ts_rgb is nanoseconds since the Aria device booted, which puts this episode’s timestamps around 30 minutes past epoch. Any tool that prints log_time as a calendar date shows something in January 1970. That’s expected for this dataset, not a unit bug, and it doesn’t affect playback or sync, since every stream below shares the same clock.

Good to know. Where does the axis and rotation math live? Three small helpers do all the geometry correction, reused by every 3D and image stream below:

def to_world_up_z(x, y, z): """ADT's world frame is Y-up; Foxglove/ROS 3D viewers assume Z-up.""" return x, -z, y def to_world_up_z_quat(qx, qy, qz, qw): """Same +90 deg X-axis correction as to_world_up_z(), applied to orientation.""" s = 0.7071067811865476 return (qw + qx) * s, (qy - qz) * s, (qy + qz) * s, (qw - qx) * s def rotate_cw90(arr): """Aria's cameras are mounted rotated; raw frames come out sideways.""" return np.rot90(arr, k=3) def ts(ns: int) -> msgs.Timestamp: """Convert nanoseconds into the {sec, nsec} pair every Foxglove message timestamp uses.""" return msgs.Timestamp(sec=ns // 1_000_000_000, nsec=ns % 1_000_000_000)

Open the MCAP and declare one channel per stream. IMU has no built-in Foxglove or ROS schema, so it gets a small custom JSON one:

img_schema = msgs.CompressedImage.get_schema() tf_schema = msgs.FrameTransform.get_schema() ann_schema = msgs.ImageAnnotations.get_schema() pc_schema = msgs.PointCloud.get_schema() scene_schema = msgs.SceneUpdate.get_schema() IMU_SCHEMA = { "title": "imu_measurement", "type": "object", "properties": {k: {"type": "number"} for k in ["accel_x", "accel_y", "accel_z", "gyro_x", "gyro_y", "gyro_z"]}, } # One foxglove.Channel per MCAP topic. message_encoding must match how the # schema is actually serialized below: protobuf for every foxglove.* # schema, json for the custom IMU one. with foxglove.open_mcap("episode_raw.mcap", allow_overwrite=True): rgb_ch = foxglove.Channel("/camera/rgb", schema=img_schema, message_encoding="protobuf") depth_ch = foxglove.Channel("/camera/depth", schema=img_schema, message_encoding="protobuf") seg_ch = foxglove.Channel("/camera/segmentation", schema=img_schema, message_encoding="protobuf") syn_ch = foxglove.Channel("/camera/synthetic", schema=img_schema, message_encoding="protobuf") ann_ch = foxglove.Channel("/annotations/rgb", schema=ann_schema, message_encoding="protobuf") tf_ch = foxglove.Channel("/tf", schema=tf_schema, message_encoding="protobuf") pc_ch = foxglove.Channel("/pointcloud/semidense", schema=pc_schema, message_encoding="protobuf") scene_ch = foxglove.Channel("/scene/3d_boxes", schema=scene_schema, message_encoding="protobuf") static_scene_ch = foxglove.Channel("/scene/3d_boxes_static", schema=scene_schema, message_encoding="protobuf") skel_ch = foxglove.Channel("/skeleton/3d", schema=scene_schema, message_encoding="protobuf") gaze_ch = foxglove.Channel("/gaze/3d", schema=scene_schema, message_encoding="protobuf") imu_ch = foxglove.Channel("/imu", schema=IMU_SCHEMA, message_encoding="json")

The 3D boxes: split static from dynamic before you write a single frame

ADT tags every object instance’s motion_type. In this episode, 304 of 354 tracked objects (furniture, wall art, kitchenware) never move at all; only 50 (people, handled objects) do.

 dynamic_box_uids = { uid for uid in instance_info_cache if instance_info_cache[uid].motion_type != MotionType.STATIC } def bbox3d_scene_entity(uid, bbox3d, ts_msg, lifetime): """Build one 3D wireframe box entity for one object at one timestamp. bbox3d.aabb is an axis-aligned box in the object's own local frame; transform_scene_object maps its corners into world coordinates. """ T = bbox3d.transform_scene_object.to_matrix() world_pts = [to_world_up_z(*(T @ np.append(pt, 1.0))[:3]) for pt in bbox3d_to_line_coordinates(bbox3d.aabb)] is_human = instance_info_cache.get(uid) and instance_info_cache[uid].instance_type.name == "HUMAN" color = msgs.Color(r=1.0, g=0.2, b=0.2, a=0.8) if is_human else msgs.Color(r=0.2, g=0.8, b=0.2, a=0.8) return msgs.SceneEntity( timestamp=ts_msg, frame_id="world", id=f"bbox3d_{uid}", lifetime=lifetime, lines=[msgs.LinePrimitive(type=msgs.LinePrimitiveLineType.LineStrip, thickness=0.02, color=color, points=[msgs.Point3(x=float(p[0]), y=float(p[1]), z=float(p[2])) for p in world_pts])], ) # First RGB timestamp inside ADT's ground-truth window: the one snapshot  # that defines "the scene" for every object that never moves.  first_ts = next(int(t) for t in ts_rgb if adt_t_start <= t <= adt_t_end) first_boxes = gt_provider.get_object_3d_boundingboxes_by_timestamp_ns(first_ts).data() static_scene_ch.log( msgs.SceneUpdate(entities=[ bbox3d_scene_entity(uid, b, ts(first_ts), msgs.Duration(sec=0, nsec=0)) # nsec=0 means persist forever  for uid, b in first_boxes.items() if uid not in dynamic_box_uids ]).encode(), log_time=first_ts, )

Keep it honest. Why does this matter enough to be step one? Redrawing all 354 boxes as a full wipe-and-rebuild every frame at 30 Hz and /scene/3d_boxes alone comes out to around 600 MB, roug...