The Relational Transformer: what survives when the schema changes

Chapter one

The model that can't leave home

Feed TinyLibrary to a schema-specific model like RelGT. The very first encoder, the one that stamps each token with its table, is a lookup with exactly three entries: customers, transactions, products. The word members isn't in it.

What we're actually asking for

In language, one pretrained model does your email, your code, your homework, and nobody retrains it per task; you describe the task in the input and the same frozen weights handle it. Notice what moved: the task used to live in the weights, and now it lives in the input. That single sentence is the most important design idea in the paper.

Databases are the hard case for this. A language model succeeds partly because it has read the internet: ask about Paris and it remembers Paris. Your company's database is proprietary. Alice's purchase history appears in no pretraining corpus. There is no Paris to remember. So if a pretrained model is going to help on your database, whatever it learned must be deeper than facts: some pattern that a shop, a Formula One season and a library all share. Does such a pattern even exist? That's the real scientific question, and chapter four settles it with evidence.

The goal, stated precisely: zero-shot relational learning: predict new targets, on a new database, with a new schema, with no weight updates. Everything the model knows about your database has to fit through one channel: the context window.

Three routes that fail

Route one: just retrain per database. The industry default, and it isn't stupid, just expensive, and it transfers nothing. Your rel-f1 model spent millions of steps learning things like recent behaviour predicts future behaviour, which is obviously not about Formula One. But it lives smeared across weights shaped like the rel-f1 schema, so the library relearns it from zero. The requirement this failure hands us is brutal and precise: no parameter may be tied to any table, column, or task. The paper's word is schema-agnostic, and the test is: could these exact frozen weights accept a database they've never seen and produce a prediction?

Route two: serialise the database to text and ask a language model. This one genuinely runs on any schema; text is text. So before looking at how well it does, we need a judge. Here is the pettiest baseline imaginable: to predict whether Alice churns, take Alice's own past churn labels and answer their average. It's called EntityMean and it has zero parameters.

R² vocabulary, since it recurs: 1 is perfect, 0 means you did no better than always guessing the global average, and negative means worse than that.

Route three: flatten the database into one table and use a tabular foundation model. Those do transfer: they dodge route one's trap by refusing to learn any column's identity, treating a table as anonymous streams of numbers. Hold onto that idea; the paper steals a piece of it. But they read exactly one table.

Three dead routes, four requirements:

1. No parameter tied to any table, column, or task.
2. The task is specified in the input, so one frozen model serves every question.
3. We need the meaning of names, that price of product and fee of book rhyme, without paying 27 billion parameters for it.
4. The pointer structure stays first-class. Never joined away.

Which leaves the design question. Every input representation we've used was built from schema-shaped pieces: whole rows, typed nodes. What is the smallest piece of any database that looks the same in every database?

Chapter two

Every cell is a token

The purple table, and one task to rule them all

For some questions the answer is already a cell: a missing age, a missing category. Hide it, guess it; that's autocomplete. But the questions businesses pay for, like will Alice churn next month, have answers in no table at all, because they're about the future. So build them a table. One row per question: which entity, as of which date, and what the answer turned out to be.

Yes, the label is computed from the future. That's legal: it's the answer sheet, not the input. The law we keep is that the model's input never crosses the question's date.

And the quiet consequence, which becomes chapter four's loaded gun: because the task table is just another table wired in by a foreign key, the model's input for Alice's February question naturally includes her older task rows. Her past answers ride along in the context.

The atom becomes a vector

A cell is a value and two names. To feed a transformer, that triple has to become one vector, and the paper builds it in two halves.

The value half. Cells come in four flavours (numbers, booleans, dates, text) and get four encoders, per datatype, not per column. That distinction is everything: a per-column encoder is welded to one schema; a per-datatype encoder works on any database that ever contained a number.

x = W_d · r + W · E_schema(c, t) W_d one small matrix per DATATYPE (4 of them) r the normalised value E_schema the frozen phrase embedding (384-dim) W ONE shared matrix masked cell: x = m_d + W · E_schema(c, t) (value half swapped for a learned mask vector)

The dog that didn't bark

A language model stamps each token with its position, because sentences have an order. The RelGT glued five stamps onto every token. This paper adds none.

Chapter three

Four masks and no positions

Attention, in one breath: every token asks a question with its query, offers an answer with its key, the match becomes a score, scores go through a softmax, and each token walks away with a weighted blend of everyone's values. The only new word is mask. Take the score table and, before the softmax, overwrite the forbidden pairs with −∞.

Language models use one famous mask: each word sees only the past. This paper's entire trick is to use four masks, and let the database schema write them. Every token already knows three things for free (its column, its table, its row) and those three facts do all the work positions ever did.

One block is: column attention, add and normalise; feature, add and normalise; neighbour, add and normalise; full, add and normalise; then a feed-forward network. Stack twelve. The authors tried all six orderings of the three relational masks, and running them in parallel: differences within noise. What matters is that each constraint exists, not when it runs.

12 blocks · d = 256 · 22M parameters · 1,024-cell context 50k steps · ~2 hours on 8 A100s one objective (fill the masked cell) for pretraining, fine-tuning AND inference

Chapter four

The verdict: what actually transfers

Read the setup before trusting any number. Pretraining is leave-one-database-out: six databases, pretrain on five, target unseen. And a control in the appendix kills the obvious objection: the column-name overlap between pretraining and target schemas is essentially zero, at most 4.3%. Whatever transfers, it isn't lexical memorisation.

On the Formula One podium task, on a database RT met at inference time, EntityMean gets 85.0 and RT gets 89.3. Gemma-27B actually beats both at 91.4, and it's worth noting why: Formula One is famous, and its data is plausibly in the giant's training corpus. Watch what happens to that advantage when the data isn't famous.

The detective story

So it works. Now the question loaded since chapter one: why does a model pretrained on shops and forums predict Formula One? Four suspects: schema semantics, in-context learning from other entities' labels, the entity's own past labels, and the architecture itself. The authors interrogate each by deleting it and measuring the damage.

There's the confession. What transfers, above all, is the ability to read an entity's past answers and forecast the next one: time-series forecasting, learned once, applied to any schema. The label counts cross-examine cleanly: Formula One driver reliability averages 19 self labels per window and scores 82.0; the podium task 17 labels, 89.1; clinical trial study-outcome has zero self labels (a study's outcome is asked once) and scores 54.5.

Two rebuttals before writing it off as a dressed-up average. The rel-avito cell above, where averaging is worse than chance and RT still gets 59.5. And the cleanest version of the question: delete self labels and fine-tune anyway: transfer from pretraining survives, positive but modest, 26.7 against 33. The magic is mostly the labels; not only the labels.

The honest reading, flagged as speculation: what generalises to an unseen schema is the disciplined operations: compare within a column, join a row to its parents. Unrestricted attention learns pretraining-specific shortcuts that mean nothing on a new database. The constraints are the transfer. Scope it carefully though: after fine-tuning all four columns blur back together, and on classification the differences were minor all along. The upset is a zero-shot regression story.

The paper's own final ranking of what drives zero-shot transfer:

1. Time-series forecasting from the entity's own past labels
2. Column attention, generalising to new value distributions
3. Feature attention, generalising to new entity types
4. Schema semantics from table and column names
5. In-context learning from other entities' task rows

Honest limits

• No link prediction or recommendation: the whole framing is "fill a cell", and a recommendation isn't one.
• Foreign-key column names go unread, so a buyer_id and a seller_id pointing at the same table are indistinguishable to the model.
• Six pretraining databases, per-task checkpoint selection, and pretraining that overfits.
• Zero-shot regression at R² ≈ 23% is useful, not magic.

Could you have invented it?

A cell is the only schema-proof atom: a value and two names, the names read by a frozen language model so unseen columns still mean something. Staple the task on as another table and every question becomes fill in the masked cell. Drop positions, because a database has no order. Then put the structure back as constraints on who may look at whom: same column, same row and parents, children, everyone. Five moves, each forced by the last.

And the finding I keep turning over: the theoretically indispensable component was dead weight, and the theoretically redundant one was load-bearing. Restricting what a model is allowed to look at turned out to be the thing that travels.