Skip to main content

Everything AI/ML

Thought pieces and hands-on tests of new models, libraries and releases. Almost every post comes with a runnable notebook.

Judging research novelty cheaply: comparing Ai2’s benchmarks to an embedding distance metric, and a small QWEN-model (with and without reasoning).

Embedding-distance novelty metrics vs LLM-as-judge on Ai2’s new novelty benchmark
evaluation
embeddings
llm-as-judge
research
benchmarks

On 154 pairs of human-written ICLR 2026 ideas, distance to prior work in BGE-M3 embedding space picked the more novel idea 60% of the time, level with a 3B LLM judge. Qwen3-4B reached 69% without reasoning and 72% with it, close to the 74% to 84% of Ai2’s frontier judges.
Nov 1, 2026
Josh Chettiar

Context rot in a small local agent: offloading works, summarisation fails silently

Testing LangChain Deep Agents’ offloading and summarisation with qwen3.5:4b and a 16k window
agents
context engineering
deep agents
langchain
local models
evaluation

Once its context overflowed, a plain tool-calling loop got every answer wrong without any warning. Deep Agents’ offloading recovered about half of those questions. Summarisation helped too, but only after three non-default settings; with one of them, it silently threw the history away.
Oct 18, 2026
Josh Chettiar

Spotting warehouse hazards: the prompt mattered as much as fine-tuning

LiquidAI’s d1 decision models vs generative VLMs and a 7-minute fine-tune, on synthetic warehouse safety images
decision models
vision-language models
fine-tuning
unsloth
evaluation
prompting

Better class definitions lifted a zero-shot Qwen3.5-4B from 62% to 81%, level with a Qwen3.5-0.8B fine-tuned on 627 images. d1’s decision post-training didn’t beat its own base model on accuracy, but it did ship calibrated. Once thresholds were chosen on held-out data, the best models caught 125–129 of 138 hazards.
Oct 12, 2026
Josh Chettiar

Can a pretrained model beat XGBoost without training?

NVIDIA Kumo Tabular vs gradient-boosted trees, plus a rematch with my 2020 Bank Marketing models
tabular
foundation models
in-context learning
xgboost
benchmarking

With zero tuning, NVIDIA’s Kumo Tabular beat default XGBoost and LightGBM on 25 of 25 splits, won most on small data, and edged out my hand-tuned 2020 models on Bank Marketing. Rebuilding those 2020 models also exposed a train/test leak.
Oct 2, 2026
Josh Chettiar
No matching items