Ilya's Top 30 Papers

Why does complexity in physical systems rise, peak, and fall — unlike entropy which only grows? Introduces 'complextropy' as a bounded complexity measure.

RNNs trained character-by-character on raw text can produce surprisingly coherent outputs — code, Shakespeare, math papers. A must-read for building intuition about sequence models.

The clearest explanation of how LSTM gates (forget, input, output) enable long-term memory. Required reading before diving into Transformers.

Dropout for LSTMs: apply it only on non-recurrent connections. Simple fix that significantly improves generalization on language modeling tasks.

Apply Minimum Description Length (MDL) to networks: add Gaussian noise to weights to compress them. A 1993 precursor to modern weight regularization and Bayesian deep learning.

Attention mechanism that points to positions in the input instead of a fixed output vocabulary. Solves variable-output problems like convex hull and TSP.

The paper that started the deep learning revolution. AlexNet used ReLU activations, GPU training, and dropout to crush ImageNet by a 10%+ margin.

The order you feed inputs into seq2seq models matters significantly — even for set-structured problems. Proposes methods for learning optimal input/output orderings.

Split model layers across accelerators and pipeline micro-batches through them. Enables training billion-parameter models on commodity hardware setups.

Skip connections solve the degradation problem in very deep networks. ResNet-152 wins ImageNet 2015; residual blocks are now everywhere in deep learning.

Dilated convolutions expand the receptive field exponentially without losing resolution. Key for semantic segmentation tasks.

Unifies GNN variants under a single Message Passing Neural Network framework for predicting molecular properties from graphs.

Introduced the Transformer — replacing RNNs entirely with self-attention. The foundation of every modern LLM.

The original attention paper. Decoder learns to focus on relevant source words dynamically — precursor to the Transformer's self-attention.

ResNet v2: move BN and ReLU before the convolution for clean identity mappings. Enables training ResNet-1001.

Relation Networks: a small module that computes all pairwise object relations. State-of-the-art on visual QA with a strikingly simple design.

Combines VAEs with autoregressive models: use autoregressive decoder to capture local details, let the VAE latent capture global structure.

Relational Memory Core: uses multi-head attention for memory-to-memory interactions. Improves on tasks requiring tracking relations over time.

Cellular automata model of complexity: shows complexity peaks at intermediate times using coarse-grained Kolmogorov complexity. Companion to paper #1.

Neural network + differentiable external memory = can learn algorithms (copy, sort, associative recall). Content + location-based addressing.

End-to-end deep learning for ASR matching human performance on benchmarks. Works across English and Mandarin, different accents, noisy environments.

Loss follows power laws in N (params), D (data), C (compute). Optimal allocation: scale N and D together. Directly led to GPT-3 and the LLM scaling era.

MDL principle: the best model is the one that compresses the data most. A bridge between Kolmogorov complexity and practical statistics / model selection.

Theoretical foundations of machine superintelligence: formal definition of intelligence, pathways to superintelligence, early AI safety framing.

Comprehensive technical textbook on Kolmogorov complexity: incompressibility, algorithmic randomness, mutual information. The math behind intelligence measures.

The gold standard CNN course — backpropagation, convolutions, batch norm, transfer learning. Still the best technical intro to deep learning for vision.

Instead of predicting one next token, predict the next k tokens in parallel with k independent heads. Faster inference + better code/reasoning performance.

Dual-encoder BERT for dense retrieval dramatically outperforms sparse BM25. The foundation of modern RAG systems.

Combine a pre-trained seq2seq model with a dense retriever over Wikipedia. Factual, updateable knowledge without retraining the model.

Distill alignment from a larger teacher LLM to a smaller student using dSFT + dDPO — no PPO, no reward model. Zephyr-7B beats larger RLHF models.