readings

A collection of papers, blog posts, and books I've read or skimmed. My main interests are:

• Foundational or historically important computer science
• AI/ML
• Systems - storage, networking, operating systems, distributed systems
• Practical programming/software engineering

papers

• Amazon Aurora: Design Considerations for High Throughput Cloud Native Relational Databases, 2017
• Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service, 2022
• Analysis of the Increase and Decrease Algorithms for Congestion Avoidance in Computer Networks, 1989
• An Algorithm for the Machine Calculation of Complex Fourier Series, James Cooley and John Tukey
• A Mathematical Theory of Communication, Claude Shannon
• A Name-Based Mapping Scheme for Rendezvous
• A New Solution to Dijkstra's Concurrent Programming Problem, Leslie Lamport
• A Note on Distributed Computing
• A Path Towards Autonomous Machine Intelligence, Yann LeCun 2022
• Architectural Styles and the Design of Network-based Software Architectures, Roy Fielding
• Are you living in a computer simulation?, Nick Bostrom
• Are You Sure You Want to Use MMAP in Your Database Management System?
• Attention Is All You Need, 2017 (transformer neural networks)
• Backpropagation Applied to Handwritten Zipcode Recognition, LeCun et al.
• Bayesian Flow Networks, Graves et al. 2023
• Berkeley DB
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• BigTable: A Distributed Storage System for Structured Data
• Bitcoin: A Peer-to-Peer Electronic Cash System
• BwE: Flexible, Hierarchical Bandwidth Allocation for WAN Distributed Computing, Google 2015
• Cache-Oblivious Algorithms and Data Structures, Erik Demaine 2002
• Can programming be liberated from the von Neumann style? A functional style and its algebra of programs
• Can we survive technology?, John von Neumann
• Cassandra - A Decentralized Structured Storage System
• Cicero - Human-level play in the game Diplomacy
• Chain Replication for Supporting High Throughput and Availability
• Challenges of Real-World Reinforcement Learning
• Characteristics of multithreading models for high performance IO-driven network applications
• Cloud Control with Distributed Rate Limiting, sigcomm 2007
• CliqueMap: Productionizing an RMA based distributed caching system, Google 2021
• Colossus under the hood: a peek into google's scalable storage system
• Communicating Sequential Processes, Tony Hoare
• Computing Machinery and Intelligence, Alan Turing
• Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web
• Convolutional Networks for Images, Speech, and Time-Series, Yan LeCun, Yoshua Bengio
• Dapper, a Large-Scale Distributed Tracing Infrastructure
• Deep Learning Recommendation Model for Personalization and Recommendation Systems, Facebook 2019
• Distributed Representations, Geoffrey Hinton et al.
• Dremel: Interactive Analysis of Web-Scale Datasets
• Dynamo: Amazon's Highly Available Key-value Store
• Efficient Estimations of Word Representations in Vector Space
• Efficient Transformers: A Survey, 2020
• Engineering a Sort Function, Bentley & McIlroy at Bell Labs
• Experiments on Learning by Back Propagation, Plaut, Nowlan, Hinton
• FlumeJava: Easy, Efficient Data-Parallel Pipelines
• Formal Analysis of the Remote Agent Before and After Flight on NASA remote debugging
• Formal Algorithms for Transformers, Deepmind 2022
• F1: A Distributed SQL Database That Scales (built on spanner)
• F1 Query: Declarative Querying at Scale
• Generative Adversarial Nets, Ian Goodfellow et al
• Generative Agents: Interactive Simulacra of Human Behavior
• Gorilla: A Fast, Scalable, In-Memory Time Series Database
• Gradient Based Learning Applied to Document Recognition, LeCun et al
• High-Resolution Image Synthesis with Latent Diffusion Models, 2021
• Hints for Computer System Design, Butler Lampson
• Hive - A Petabyte Scale Data Warehouse Using Hadoop
• How do Committees Invent?, Conway (Conway's Law paper)
• How to Write a Proof, Leslie Lamport
• Human-level control through deep reinforcement learning, DeepMind
• ImageNet Classification with Deep Convolutional Neural Networks
• In Search of an Understandable Consensus Algorithm (RAFT)
• Indexing the Distance: An Efficient Method to KNN Processing
• Interpreting the Data: Parallel Analysis with Sawzall
• Is this the simplest (and most surprising) sorting algorithm ever?
• It's as simple as one, two, three..., Richard Feynman
• John Carmack's 1998 .plan file
• Kafka: a Distributed Messaging System for Log Processing
• The Llama 3 Herd of Models
• Language Models are Few-Shot Learners aka GPT3, OpenAI 2020
• Logstore: A Cloud Native and Multi-Tenant Log Database, Alibiba SIGMOD 2021
• LVars: Lattice Based Data Structures for Deterministic Parallelism
• Large-scale cluster management at Google with Borg
• Live Migration of Virtual Machines
• Machine Learning: The High Interest Credit Card of Technical Debt, Google 2014
• Maglev: A Fast and Reliable Network Load Balancer
• Making CRTs Byzantine Fault Tolerant, Martin Kleppmann
• Making reliable distributed systems in the presennce of software errors, Joe Armstrong Erlang Thesis
• MapReduce: Simplified Data Processing on Large Clusters
• MapReduce: A major step backwards, Dewitt & Stonebreaker, 2011
• Mastering the game of Go without human knowledge, Deepmind (AlphaGo Zero)
• Megastore: Providing Scalable, Highly Available Storage for Interactive Services, Google
• Mercator: A Scalable, Extensible Web Crawler
• Millions of Tiny Databases, Marc Brooker et al from AWS 2020
• Monarch: Google's Planet-Scale In-Memory Time Series Database
• Metastable Failures in Distributed Systems
• NICE: Non-Linear Independent Components Estimation, Dinh, Krueger, Bengio 2014
• NinjaMail: the Design of a High-Performance Clustered, Distributed E-mail System, Behren et al., UC Berkeley 2000
• On Computable Numbers, With an Application to the Entscheidungsproblem
• On Designing and Deploying Internet Scale Services, James Hamilton
• On Performance Stability in LSM-based Storage Systems
• On a Heuristic Point of View Concerning the Production and Transformation of Light External, Einstein photoelectric effect
• On the Electrodynamics of Moving Bodies, Einstein special relativity 1905
• On the Measure of Intelligence, Francois Chollet
• Out of the Tar Pit
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture of Experts Layer, Noam Shazeer et al 2017
• Pathways: Asynchronous Distriuted Data Flow for ML, Google 2022
• PaLM: Scaling Language Modeling with Pathways
• Paxos Made Live - An Engineering Perspective
• Performance Isolation and Fairness for Multi-Tentant Cloud Storage, OSDI 2015
• Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams
• Preserving I/O Prioritization in Virtualized OSes
• Procella: Unifying serving and analytical data at YouTube
• Programming as Theory Building, Peter Naur
• Profiling a warehouse-scale computer, Google 2014
• RAMP-Tao: Layering Atomic Transactions on Facebook's Online Tao Data Store
• Recursive Functions of Symbolic Expressions and Their Computation By Machine, John McCarthy (i.e. the LISP paper)
• Reflections on Trusting Trust
• RocksDB: Evolution of Development Priorities in a Key-Value Store Serving Large Applications
• Scaling Laws for Neural Language Models
• Slicer: Auto-Sharding for Datacenter Applications
• Snap: a Microkernel Approach to Host Networking, Google 2019
• Spanner: Google's Globally Distributed Database
• Sparks of Artificial General Intelligence: Early Experiments with GPT-4, Microsoft Research 2023
• Solving Quantitative Reasoning Problems with Language Models
• Swift: Delay is Simple and Effective for Congestion Control in the Datacenter
• Tao: Facebook's Distributed Data Store for the Social Graph
• The Anatomy of a Large-Scale Hypertextual Web Search Engine
• The Byzantine Generals Problem, Leslie Lamport
• The Case for Learned Index Structures
• The Cathedral and the Bazaar, Eric Raymond
• The Chubby lock service for loosely-coupled distributed systems
• The Datacenter as a Computer, Luis Barroso and Urs Holzle
• The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Oout-of-order Data Processing
• The Design and Implementation of a Log-Structured File System, Mendel Rosenblum and John K. Ousterhout
• The Forward-Forward Algorithm: Some Preliminary Investigations, Geoffrey Hinton 2022
• The Google File System
• The Hadoop Distributed File System
• The Linux Scheduler: A Decade of Wasted Cores
• The PageRank Citation Ranking: Bringing Order to the Web
• The Part-Time Parliament, Leslie Lamport
• The Power of the Context, Alan Kay
• The RAMCloud Storage System, Ousterhout et al.
• The Slab Allocator, An Object-Caching Kernel Memory Allocator
• The Swirlds Hashgraph Consensus Algorithm
• The Tail at Scale
• Time, Clocks, and the Ordering of Events in a Distributed System
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Top 10 Algorithms in Data Mining
• Transformer Feed-Forward Layers Are Key-Value Memories
• virtio: Towards a De-Facto Standard For Virtual I/O Devices
• Virtual Consensus in Delos
• Web search for a planet: the Google cluster architecture, Barosso, Dean, Holzle
• What Every Computer Scientist Should Know about Floating Point Arithmetic
• What Every Programmer Should Know about Memory
• Why Google Stores Billions of Lines of Code in a Single Repository
• Zanzibar: Google's Consistent, Global Authorization System
• ZooKeeper's Atomic Broadcast Protocol - Theory and Practice

blog posts

• The Bug in Paxos Made Simple, Marc Brooker
• Developers spend most of their time figuring out the system
• The Illustrated Transformer, Jay Alammar
• Google Research, 2022 & Beyond: Language, Vision and Generative Models, Jeff Dean et al

books

• Designing Data Intensive Applications, Martin Kleppmann
• Distributed Systems, Andrew Tanenbaum
• Introduction to Algorithms, Cormen, Leiserson, Rivest, and Stein
• Linux Kernel Development, Robert Love
• Society of Mind, Marvin Minsky
• The Algorithm Design Manual, Skiena
• The Mythical Man Month, Fred Brooks
• The Structure and Interpretation of Computer Programs, Abelson and Sussman
• Artificial Intelligence: A Modern Approach, Peter Norvig and Stuart Russell
• Computer Architecture: A Quantitative Approach, David Patterson and John Hennessy