Inside X's For You Feed: Thunder, Phoenix, and the Recommendation Pipeline
A recommendation feed is not a single ranking model. It is a pipeline that has to find candidate posts, understand the viewer's history, remove content that should not appear, score the remaining candidates, and assemble a useful timeline within a request budget.
The xAI open-source repository makes a substantial part of the code behind X's For You feed inspectable. It is a valuable systems-design resource because it exposes the separation between retrieval, ranking, visibility filtering, and final feed blending.
The repository is not new: GitHub records its creation on January 19, 2026. It appeared at position 3 in the Rust daily Trending snapshot collected at 06:57 UTC on November 17, with 159 stars shown for that day. Its latest commit in the repository API was made at 03:20 UTC today, with the message "Open-source X Recommendation Algorithm". GitHub showed 34,271 stars at 10:51 UTC. This is renewed attention around an actively updated codebase, not a newly created project.
The feed starts with retrieval, not ranking
The README describes two main candidate sources.
- Thunder handles in-network retrieval. It keeps recent posts from accounts a viewer follows in memory.
- Phoenix retrieval and SimClusters find out-of-network candidates. A popular-post source can add a shared set of recent posts.
Those sources are combined before the ranking model scores them. That distinction is important. A ranking model cannot select a relevant post that the retrieval stage never returned.
The README also separates the post pipeline from the blending pipeline. The post pipeline finds, ranks, and filters posts. The blending pipeline adds other feed elements such as ads, Who to Follow recommendations, and prompts.
A simplified view
The following is a conceptual diagram based on the repository README. It is not a complete production deployment diagram.
flowchart TD A[Viewer request] --> B[Thunder: followed accounts] A --> C[Phoenix retrieval: out-of-network] A --> D[SimClusters and popular posts] B --> E[Candidate merge] C --> E D --> E E --> F[Phoenix ranking] F --> G[Visibility filtering and deduplication] G --> H[VMRanker diversity] H --> I[Feed blending] I --> J[For You timeline] The source also makes clear that visibility filtering is a separate concern from ranking. Blocks, mutes, and content or account labels can determine whether a candidate is eligible to appear at all. A high model score should not override a visibility rule.
Phoenix uses two different model stages
Phoenix contains both retrieval and ranking code.
Retrieval uses a two-tower model. One tower encodes a viewer's recent engagement history. The other represents candidate posts. The system can compare embeddings to narrow a large corpus to a smaller candidate set.
Ranking uses a transformer to score the retrieved candidates. The model predicts the likelihood of different viewer actions, and those predictions are combined into a score. The README describes a Rust serving layer around the JAX model code.
The repository also documents a VMRanker stage that applies a determinantal point process over candidate embeddings. In practical terms, this helps trade a little raw score for less similarity between neighboring posts. A feed with ten nearly identical high-scoring items is not necessarily a good feed.
The current README records a notable October update: Phoenix retrieval was trained to surface posts people read through, and the ranking model was improved at estimating time spent. That is a useful reminder that recommendation systems can optimize for more than immediate clicks.
Explore the runnable Phoenix path
The top-level repository contains more than a diagram. The Phoenix directory includes a quickstart that trains and serves small models with synthetic data.
The documented requirements are:
- Linux with an NVIDIA GPU and CUDA 12
- Python 3.11 or newer and uv
- A Rust toolchain
- protoc 3.15 or newer
From the Phoenix directory, the quickstart begins with:
uv sync --extra engine export PYTHONPATH=$PWD Then verify that the ranking serving path starts:
uv run python xrex/inference/oss_bench/bench.py --smoke --service_type ranking This smoke test uses random weights. It verifies that the local serving stack can boot, not that a model has learned useful recommendations.
The full quickstart goes further: it generates deterministic synthetic data, trains a small ranking model, saves and resumes checkpoints, serves the checkpoint through gRPC, and runs a retrieval-to-ranking request. The nano configurations are designed to make the path testable on a single GPU.
That is a good way to learn the architecture without pretending you can reproduce X's production environment on a laptop.
What is actually open, and what is missing?
The Phoenix README says the repository includes production model definitions, training code, checkpointing, and the Rust serving engine. It also explains that internal data feeds, cluster orchestration, telemetry, and other xAI-specific infrastructure are not included. Synthetic generators replace production datasets in the local quickstart.
That boundary matters. You can inspect real model code and run a proof-of-concept path, but you cannot reconstruct the exact production feed, its data, or every operational dependency from this repository alone.
The code is licensed under Apache 2.0. There are no tagged GitHub releases in the repository's releases endpoint at the time of this run, so pin a commit when experimenting.
How to use this repository as a developer
It is most useful for:
- Learning how a large recommendation system separates retrieval from ranking.
- Studying two-tower retrieval and transformer-based candidate scoring.
- Understanding why visibility filtering is not the same thing as relevance ranking.
- Exploring a Rust serving layer connected to JAX model code.
- Building a small synthetic recommendation pipeline for experiments.
It is not a ready-made recommendation API for your own product. If you are building a new recommender, compare it with TensorFlow Recommenders, TorchRec, and RecBole. Those projects are general-purpose building blocks; none is a drop-in copy of X's feed.
For a separate system-design walkthrough, this independent video covers the repository's major stages:
The video is from Debug with Dheeraj and was published in May 2026. It is not an official xAI presentation, so use the repository README and source files for current implementation details.
Sources and further reading
- xAI x-algorithm repository
- System architecture in the README
- Phoenix architecture and model code
- Phoenix local quickstart
- X's public explanation of For You recommendations
- Independent architecture walkthrough
If you were designing a feed for a smaller community app, which stage would you build first: candidate retrieval, ranking, or visibility filtering?
Written by Sh Raj. More engineering notes: LinkedIn · StackShade on YouTube.