Actions speak louder than words.
Here's why that broke how recommenders are built.

How recommenders work today (and why they hit a wall)

Every feed you scroll — Instagram, YouTube, TikTok, Amazon — is powered by a DLRMDeep Learning Recommendation Model — the industry-standard architecture: take thousands of features about a user and an item, feed them through neural network layers, output a score like "probability of click".: a Deep Learning Recommendation Model. As an engineer, think of it as a giant scoring function:

score = f(thousands_of_features_about_user_and_item)

The two-stage funnel

You can't score every item for every user — there are billions of items. So production systems run a funnel: retrieval (cheaply narrow billions down to thousands of candidates) then ranking (expensively score those thousands to pick the top handful). Two separate stages, traditionally two separate models.

The features problem

Where do those "thousands of features" come from? Humans. Engineers hand-build pipelines that compute things like user_clicks_on_sports_last_7d, item_ctr_by_country, cosine_sim(user_topic_vector, item_topic_vector). A mature system at Meta's scale has thousands of these, each with its own pipeline, backfill jobs, and on-call rotation.

The wall: compute doesn't help

In language models, a famous pattern holds: throw 10× more compute and data at the model, and quality keeps improving along a predictable curve (a "scaling law" — more in Module 6). DLRMs don't do this. Past a point, making a DLRM bigger barely helps. The model is bottlenecked by its diet of hand-made, lossy features — not by its size. Meta was sitting on tens of billions of user actions per day and couldn't convert that data advantage into model quality.

ML crash course: just enough to read the paper

Four concepts unlock the whole paper: embeddings, training, attention, and autoregressive generation. Each gets two minutes and an animation.

2.1 — Embeddings: hashing things into meaning-space

Computers can't compare "Inception" and "Interstellar" as strings. An embeddingA learned list of numbers (a vector, e.g. 256 floats) representing an item, user, or word. Trained so that similar things end up with nearby vectors. maps each item to a vector of numbers — like coordinates on a map — and training nudges those coordinates so that similar items end up close together. Similarity becomes geometry: just measure distance.

2.2 — Training: gradient descent in one paragraph

A model starts with random numbers (parameters). You show it an example, it makes a prediction, and a loss function scores how wrong it was. Calculus tells you which direction to nudge every parameter to be slightly less wrong. Repeat billions of times. That's it — training is while (wrong) { nudge(); } at absurd scale.

2.3 — Attention: every token looks at every other token

Given a sequence (words, or user actions), attentionA mechanism where each position in a sequence computes a relevance weight for every other position, then blends their information together according to those weights. The core of the Transformer architecture. lets each element ask: "which earlier elements matter for understanding me?" — and blend in their information, weighted by relevance. Stack layers of this and you get a Transformer, the architecture behind GPT and, with modifications, behind this paper.

2.4 — Autoregressive generation: predict the next token, forever

GPT works by repeatedly answering one question: given everything so far, what comes next? Each prediction is appended and the loop continues. This framing is the engine the paper borrows: instead of predicting the next word, predict the user's next action.

The big idea: Generative Recommenders

Here's the reframe that gives the paper its title. Stop treating recommendation as "score this user-item pair using 1,000 features". Start treating it as sequential transductionMapping an input sequence to an output sequence. Here: map a user's chronological action sequence to predictions about their next actions and responses.: the user's life is a chronological sequence of action tokens, and the model's job is to generate what comes next. The paper calls models built this way Generative Recommenders (GRs).

Where did all the features go?

The radical part: GRs delete dense features (all those hand-computed counters and ratios). The bet is that a sequence model can re-derive them from raw events — your click-through-rate over the last week is, after all, computable from your last week of events. Categorical features that carry real new information (like "user followed creator X") aren't deleted — they're sequentialized: converted into auxiliary tokens and merged into the main timeline, so everything the model knows arrives as part of one stream.

Why "actions speak louder than words"?

Language models learn from what people say. This paper's claim is that what people do — tens of billions of actions daily on a large platform — is an even richer, largely untapped modality, big enough to train GPT-3-scale models on behavior alone.

HSTU: a Transformer rebuilt for behavior data

Could Meta just run GPT over action sequences? They tried; vanilla Transformers struggled. Recommendation data breaks three Transformer assumptions, and HSTU — Hierarchical Sequential Transduction Units — is the redesign. Three problems, three fixes:

Problem 1: softmax erases intensity

Standard attention runs its relevance scores through softmaxA function that converts a list of scores into probabilities that sum to exactly 1. Great for "pick one word", bad for expressing absolute strength — doubling all scores changes nothing after normalization., which forces all weights to sum to 1. That answers "which token matters relatively most?" — perfect for grammar, terrible for preferences. If you binged 40 cooking videos, that absolute intensity is signal, but softmax normalizes it away. HSTU's fix: pointwise attention. Each relevance score goes through a soft on/off gate (SiLU) independently — no normalization, so total enthusiasm is preserved.

Problem 2: position alone isn't enough

In text, "3 tokens ago" means something. In behavior, 3 actions ago could be 3 seconds or 3 weeks ago, and a doom-scrolling session generates 500 tokens while a deliberate purchase generates 1. HSTU adds a relative attention bias built from both position and timestamps, so the model natively understands recency and session rhythm.

Problem 3: too slow at recommendation scale

Sequences here are long (~8,000 actions) and must be re-scored constantly. HSTU attacks cost from several angles: it simplifies each layer (fusing the usual attention + feed-forward blocks into one leaner unit where a gating multiply replaces the big MLP), exploits the fact that real user histories have ragged, varying lengths rather than padded uniform ones, and during training uses Stochastic LengthA training trick: randomly subsample long user histories into shorter sequences most of the time. Because attention cost grows quadratically with length, this slashes compute with negligible quality loss. to randomly shorten histories. Net result: 5.3–15.2× faster than a FlashAttention2-optimized Transformer on 8,192-length sequences.

Serving it: M-FALCON, or how to afford inference

Training a huge model is one bill; serving it to billions of users at feed-refresh speed is the bill that kills projects. Ranking means scoring thousands of candidates per request. Naively, that's thousands of forward passes through a giant model. Per user. Per scroll.

The trick: candidates share almost everything

Every candidate is scored against the same user history. So why recompute the history's representation thousands of times? M-FALCON processes candidates in micro-batches: compute the expensive user-history part once, cache it (like attention KV cachingKey-Value caching: storing the intermediate attention tensors for a fixed prefix so subsequent computations can reuse them instead of recomputing — the same trick that makes LLM chat responses fast. in LLM serving), then score batches of candidates against the cached state, amortizing the cost across all of them.

Results, scaling laws, and why this paper mattered

The numbers

On public benchmarks (MovieLens, Amazon Books), HSTU beat strong baselines by up to 65.8% on NDCGNormalized Discounted Cumulative Gain — a 0-to-1 metric for ranking quality that rewards placing the truly relevant items near the top of the list., a standard ranking-quality metric. But the result that made the industry pay attention came from production: a 1.5-trillion-parameter GR, deployed on surfaces serving billions of users, lifted online A/B testThe gold standard for measuring real impact: show the new system to a random slice of real users, the old one to the rest, and compare engagement metrics. metrics by 12.4% — an enormous swing against a DLRM baseline that hundreds of engineers had optimized for years.

The headline: scaling laws arrive in recsys

The deepest finding: GR quality improves as a smooth power law of training compute, holding across three orders of magnitude, up to GPT-3/LLaMa-2 scale. DLRMs flatline; GRs keep climbing. For the first time, a recommender team could trade money for quality on a predictable curve — the same property that fueled the LLM boom.

What you should remember

① Recommendation was reframed from feature-based scoring to next-action generation. ② HSTU rebuilt the Transformer for behavior data: pointwise (not softmax) attention, time-aware bias, 5.3–15.2× faster at length 8,192. ③ M-FALCON made serving affordable (285× complexity, same budget). ④ It worked in production: +12.4% online, and quality scales with compute — the property that changed everything in NLP, finally demonstrated in recsys.