The question

Gradient-boosted trees have been the default answer for tabular data for a decade. Tabular foundation models take a different route, and NVIDIA’s Kumo Tabular reports first place on the TabArena and TALENT benchmarks.

I wanted to check two things on a free Colab GPU:

1. With zero tuning, does it beat default XGBoost and LightGBM on small and medium classification tables?
2. How does it compare with the hand-tuned models from a Bank Marketing notebook I wrote in 2020?

How in-context learning works

A gradient-boosted tree model is trained. You give it your training rows, it builds hundreds of trees to fit them, and you then use those trees to predict new rows. Every dataset gets its own freshly fitted model, usually with hyperparameter tuning on top.

Kumo Tabular is never trained on your data. Its weights were fixed once, by NVIDIA, after pretraining on tens of millions of synthetic tables. The tables were generated from random causal graphs, so the model has seen a huge variety of “features cause a target” patterns, but no real dataset. To make a prediction, you pass it two things in a single forward pass:

1. your labelled training rows (the context), and
2. the unlabelled rows you want predictions for (the queries).

Inside the transformer, each query row attends to the context rows and their labels and infers the relationship on the fly, much as an LLM picks up a task from worked examples in its prompt. Nothing is fitted, nothing is tuned and no weights change. The “learning” happens inside one forward pass, which is why it’s called in-context learning (ICL).

flowchart TB
  subgraph T["XGBoost / LightGBM: train, then predict"]
    direction LR
    A["Training rows<br/>+ labels"] --> B["Fit hundreds of trees<br/>(+ tuning)"]
    B --> C["New model for<br/>this dataset"]
    D["Test rows"] --> C
    C --> E["Predictions"]
  end
  subgraph K["Kumo Tabular: in-context learning"]
    direction LR
    F["Training rows<br/>+ labels"] --> H["Frozen pretrained transformer<br/>one forward pass, no weight updates"]
    G["Test rows"] --> H
    H --> I["Predictions"]
  end
  T ~~~ K

So this isn’t zero-shot. The model needs labelled examples, and in every comparison here it gets exactly the same training rows as XGBoost and LightGBM. It’s zero-training. The pretrained prior is what lets it do well from very few rows.

Setup

• No tuning anywhere except the 2020 models, which keep their 2020 tuned settings. The GBDTs use library defaults with native categorical handling.
• Kumo settings: 8 ensemble members, fp16, at most 10k context rows per member.
• Hardware: Colab T4 GPU for the in-context models, Colab’s 2 vCPUs for the trees.
• Reproducibility: a full rerun reproduced every Part A ROC-AUC to 4 decimal places.

Results

What I take from this

1. For small and medium tables, a pretrained in-context model is now a strong default. With no tuning, Kumo Tabular beat untuned GBDTs on every split I tested, often by a margin bigger than the split-to-split noise. On Bank Marketing it also beat my tuned 2020 XGBoost on PR-AUC.
2. The less data you have, the bigger the win. Kumo needed roughly 3–10× fewer rows than the trees to reach the same ROC-AUC. As data grows towards the context limit, the GBDTs close in.
3. Kumo small is the practical choice. It came within 0.004 ROC-AUC of large at about one-eighth of the cost.
4. Thresholds still need thought. The in-context models aren’t told about class imbalance, so on imbalanced problems pick the operating point from the precision-recall curve rather than using 0.5.
5. The rematch was most useful for checking my old work. It surfaced a leaky feature (duration) and a train/test leak that likely inflated my 2020 random forest results.

Limitations

• Small benchmark. Five binary datasets and one extra case study. NVIDIA’s claims rest on TabArena’s 51 datasets. These results are a sanity check on public data, not a ranking.
• Untuned baselines. Except for the 2020 models, the GBDTs ran at library defaults with no early stopping. Tuned GBDTs would close some of the gap.
• Context cap. Kumo saw at most 10k training rows per ensemble member, and used 8 members rather than the 16 in NVIDIA’s benchmark.
• Public datasets. The shuffled-label control rules out memorisation of these datasets, but it can’t rule out that the model’s design was tuned on public benchmarks like these. A dataset published after the model, or private data, would be a stronger test.
• Timing compares a GPU with a 2-vCPU CPU. It reflects practical cost on free Colab, not algorithmic efficiency.
• Fine-tuning not tested. Every Kumo number here is from the frozen pretrained model.

Data and reproducibility

Bank Marketing: S. Moro, P. Cortez and P. Rita (2014), A Data-Driven Approach to Predict the Success of Bank Telemarketing, Decision Support Systems 62:22–31.

Everything, including setup, every table, every chart and the controls, is in the full notebook. It runs top to bottom on a free Colab T4 with the Open in Colab button. The raw results CSVs are on GitHub.