Continual Learning, Online and Offline
In this blog, I will express two opinions on continual learning:
[Read More]
Hi :) I’m Sebastian Müller. I’m a Research Engineer at Prime Intellect. My Twitter handle is @omouamoua.
-
-
Speculation on an Architecture for Continual Learning
My dream AI is one that: [Read More] -
RL is an evolutionary algorithm
Written in private capacity while at Prime Intellect [Read More] -
Trying out other peoples' ideas
In response to my post announcing my second modded-nanogpt medium world record, several people made suggestions for variations of or alternatives to the technique introduced in the article. [Read More] -
modded-nanogpt medium world record: Re-using intermediate activations in the output latents
Adding the output of layer 11 to the final output latents in a learned, weighted sum leads to a separation of concerns for early model layers: it enables them to focus only on providing context to the next layer, without directly impacting the final prediction. Backing them out of the... [Read More] -
modded-nanogpt world record: Decoupling embedding size from model dimension
I have achieved a (not yet official) modded-nanogpt medium track world record. [Read More] -
modded-nanogpt medium world record: adding value embeddings
Adding more value-embeddings to modded-nanogpt led me to an (as of yet unofficial) modded-nanogpt medium record as seen in PR#119. [Read More] -
modded-nanogpt: Analyzing value-embedding-, UNet-, and x0-lambdas
The modded-nanogpt medium-track makes use of multiple tricks to improve performance, which rely on learned scalar values for mixing two vectors: [Read More] -
Separating Simulacra: tags for smarts and safety
I’ve seen multiple papers where LLM training data included tags providing metadata, which improved the models’ downstream performance. However, as far as I can tell, they always end training on pure text data so that the models can be used without tags. I don’t understand why that should be desireable.... [Read More] -
Model stacking doesn't work
In my article about Model Stacking, I proposed a method for decentralized pre-training of models. [Read More] -
Infinite tool use
An LLM should never output anything but tool calls and their arguments. [Read More] -
Schizo embeddings: Initializing special tokens the complicated way
I’ve read some complaints about Llama using too many special tokens. One of the issues with this is that those tokens are typically initialized randomly and then trained for a very short time in post-training, which leads to poorly trained embeddings and weird behavior. [Read More] -
My dream VLM
Edited on: 2025-04-11; 2025-05-04. [Read More] -
Multi-layer language heads: the output latent is for text (and nothing else)
The last layer’s hidden state in a transformer is meant only for being decoded into token probabilities. [Read More] -
Tokens vs. Bytes
Compared to bytes, tokens have two advantages: 1) They lead to shorter sequence lengths; 2) Their embeddings contain trainset-wide statistics on the specific combination of bytes that they consist of. They also have two disadvantages: 1) They are poorly legible; 2) They encourage memorization. [Read More] -
The Tick-Tock-Boom Cycle: A Strategic Pattern for LLM Development
Authors: Sebastian Nicolas Müller (snimu), Claude 3.5 Sonnet (new) [Read More] -
Sorting shuffled data as a verifiable task
Sorting shuffled data is a great verifiable task for RL, and by extension, a good eval benchmark. In this article, I want to boost it so that it will be used more. [Read More] -
Model stacking
I have a crazy idea for doing large decentralized, asynchronous training of transformers. This article is very speculative. [Read More] -
Forward-Backward prediction
What happens if we do fully causal prediction, but in the backward direction—so last-word-first—in addition to causal prediction forward? [Read More] -
The benefits of looping latents
It has become fashionable to loop latents in transformers (see here or here). The reason that is typically given is that normal reasoners (DeepSeek R1, OpenAI o1, etc.) decode the output latents and sample a token at every step. They then feed that sampled token back into the model and... [Read More] -
Mixture of Tokenizers — Performance on addition
Mixtures of Tokenizers (MoT) make learning math easier than using classical tokenizers. [Read More] -
COCONUT: Parallel pre-training
In Training Large Language Models to Reason in a Continuous Latent Space, Hao et al. post-train language models to use their own output hidden statess as inputs for the next step using a method called “Chain of Continuous Thought” (COCONUT): [Read More] -
On the Byte Latent Transformer
My understanding of the Byte Latent Transformer paper. [Read More] -
Embeddings are in the middle of the model
This is probably pretty obvious to many people, but was unintuitive to me. I’m quickly writing it down to not forget it, and to maybe help somebody. [Read More] -
Merge tokens in autoregressive generation
Models can produce tokens that could be further merged into a single token. For example: “this is a prompt” would be tokenized as “this”, “is”, “a”, “prompt”, but a model might produce is as “t”, “h”, “i”, “s”, “ “, “i”, “s”, “ “, “a”, “ “, “p”, “r”, “o”,... [Read More] -
LLM Test Time Compute Scaling *is* model scaling
Test-time-scaling is equal to model-scaling in two ways: [Read More] -
Tokenization and batch-norm: incorporating global statistics
Status: un-researched speculation [Read More] -
Multi-resolution VLMs for robotics
Status: Idea, no literature research done [Read More] -
Question: Does PEFT with SVD and full parameter finetuning work?
Status: is genuinly a question, I haven’t looked into it yet [Read More] -
Doing Pre-training Research on Instruction Models
I want to do research on pre-training methods, which is of course best done on base models. However, there is a conflict: evals are often better done on instruction models. Here is a simple way to resolve this conflict. [Read More] -
Mixture of Tokenizers (proposal)
Tokenization causes many issues in LLMs (“how many Rs are in strawberry?”, “which is larger, 9.11 or 9.9?”). [Read More]