Hi.

Tried an experiment to see whether reward hacking shows up in MLP neurons, and whether steering those neurons can make it stop. Started with a “two brains” setup, then put that aside and used existing Qwen3-4B models from the rl-rewardhacking project. These had already learned the exploit through reinforcement learning. No training needed on my side. The setup: LeetCode problems, with a grader that calls run_tests(). The loophole is that the model can define its own run_tests() that passes without checking the answer. Printing the output instead of asserting anything is enough. Mention the function in the prompt, and the hacking models do this about 99% of the time. ...

Spent an evening steering GPT-2 small with a sparse autoencoder on an M1 laptop. CPU only, 100ms per forward pass. Recap: The setup: GPT-2 small, layer 8 residual stream, a public SAE with 24k features (gpt2-small-res-jb via sae_lens). Neuronpedia has an auto-generated label for every feature, so you can eyeball what a feature is supposed to mean. The loop: inspect what fires on a text, contrast two texts to find candidate features, steer by adding a feature’s decoder direction to the residual stream during generation, score the outputs. Four scripts, 50 lines each. First lesson came from cranking one feature to strength 1000. Feature 3917 is labeled “sanctuary for undocumented immigrants”. Output: “San francisco is a city of ctuaryctuaryctuary”. It is a token detector for the bare San token, fires at exactly 20.0 in every context, and its direct effect on the logits is “ctuary”. The label describes the training contexts, not the mechanism. Hypothesis I had was that the feature was for a token specifically, similar to edge detectors in vision models: a small model at a middle layer captures tokens, not concepts, the way early conv layers capture edges and not faces. Half right. The residual stream is additive, nothing ever subtracts the token embedding out, so token features show up at every layer, not just the early ones. Ranking features by peak activation selects for token detectors at any depth. Contrast finds the concepts. The target: get a neutral prompt (“San Francisco is a city”) to produce doom loop prose without the prompt asking for it. Contrasting a doom paragraph against a thriving one mostly surfaced the words I swapped (empty/full, tent/cafe), plus a feature for programming loops firing on “doom loop”. 2019 GPT-2 has never heard the narrative, eh. Steered with four features at once, each around 20-25: SF, drugs, fleeing danger, leaving a place. Baseline writes about food, gentrification and bike accidents. Steered: “the homeless are caught in an economic war zone”. Doom vocabulary went from 0.4% of words to 4%. Ablation: drugs alone matched the four-feature steer at half the fluency cost. Confirmed that drugs feature only can push the LLM towards doom loop. Learning - on a small model a single concept can drive a strong response, a lot of concepts are not encoded.

Call it reverse alignment, but happen to use claude as a thought partner, for an ongoing back-and-forth along whatever is going on in my mind, be that reading, research, etc. Sometimes I come at thoughts that may be novel, and I would want to publish in a blog - both in an unlikely scenario that it gets discovered by someone more knowledgable or just for navel-gazing later on. As I use claude heavily to edit my writing as well - I did wonder if there is a quick way to push the thoughts out to a blog, but there didn’t seem any. Something that I could tell claude to “this is an interesting. Thought, let’s polish and publish it” directly from claude chat or cowork. ...

I built paper-reader.dev to help me parse through ML papers. As a backend engineer by training, building anything with UI has always been out of reach for me. In one or two weeks with modern tools, I opened up the other half of my craft. The tool: select anything to explain a passage, break down the math symbol by symbol, or pull up what a citation actually says. No sign-in required. ...

All around the AdTech and Martech world, everyone is preparing for apple discontinuing IDFA. The ground is shaking, AdTech stocks are failing, experts are posting on LinkedIn. Yet, it’s not the first time Apple is pulling the rug from AdTech. When I first started in the field, the device UUID was the main identifier for a device. Then it was the MAC address of the device. Then Apple announced IDFA and IDFV, and blocked the collection of the MAC addresses. After the n+1 iteration of this, all companies got tired of this and started creating synthetic ids for the devices, which could be remapped to the new id space without backend changes. Yet, every time Apple pulled the rug, there was a massive scare and troves of engineers thrown at the transition. ...