Ai

Tried an experiment to see whether reward hacking shows up in MLP neurons, and whether steering those neurons can make it stop. Started with a “two brains” setup, then put that aside and used existing Qwen3-4B models from the rl-rewardhacking project. These had already learned the exploit through reinforcement learning. No training needed on my side. The setup: LeetCode problems, with a grader that calls run_tests(). The loophole is that the model can define its own run_tests() that passes without checking the answer. Printing the output instead of asserting anything is enough. Mention the function in the prompt, and the hacking models do this about 99% of the time. ...

Spent an evening steering GPT-2 small with a sparse autoencoder on an M1 laptop. CPU only, 100ms per forward pass. Recap: The setup: GPT-2 small, layer 8 residual stream, a public SAE with 24k features (gpt2-small-res-jb via sae_lens). Neuronpedia has an auto-generated label for every feature, so you can eyeball what a feature is supposed to mean. The loop: inspect what fires on a text, contrast two texts to find candidate features, steer by adding a feature’s decoder direction to the residual stream during generation, score the outputs. Four scripts, 50 lines each. First lesson came from cranking one feature to strength 1000. Feature 3917 is labeled “sanctuary for undocumented immigrants”. Output: “San francisco is a city of ctuaryctuaryctuary”. It is a token detector for the bare San token, fires at exactly 20.0 in every context, and its direct effect on the logits is “ctuary”. The label describes the training contexts, not the mechanism. Hypothesis I had was that the feature was for a token specifically, similar to edge detectors in vision models: a small model at a middle layer captures tokens, not concepts, the way early conv layers capture edges and not faces. Half right. The residual stream is additive, nothing ever subtracts the token embedding out, so token features show up at every layer, not just the early ones. Ranking features by peak activation selects for token detectors at any depth. Contrast finds the concepts. The target: get a neutral prompt (“San Francisco is a city”) to produce doom loop prose without the prompt asking for it. Contrasting a doom paragraph against a thriving one mostly surfaced the words I swapped (empty/full, tent/cafe), plus a feature for programming loops firing on “doom loop”. 2019 GPT-2 has never heard the narrative, eh. Steered with four features at once, each around 20-25: SF, drugs, fleeing danger, leaving a place. Baseline writes about food, gentrification and bike accidents. Steered: “the homeless are caught in an economic war zone”. Doom vocabulary went from 0.4% of words to 4%. Ablation: drugs alone matched the four-feature steer at half the fluency cost. Confirmed that drugs feature only can push the LLM towards doom loop. Learning - on a small model a single concept can drive a strong response, a lot of concepts are not encoded.

I built paper-reader.dev to help me parse through ML papers. As a backend engineer by training, building anything with UI has always been out of reach for me. In one or two weeks with modern tools, I opened up the other half of my craft. The tool: select anything to explain a passage, break down the math symbol by symbol, or pull up what a citation actually says. No sign-in required. ...