Technology world tree

Left open

It served one district with direct current, far from today's interconnected AC grid.

Left open

It solved sending power far; where power comes from and how a grid is run were left to later plants and grids.

• Complements · Pearl Street

  IEEE records the Adams plant as a key victory of AC over DC; Pearl Street supplied DC.

Left open

It answers what can be computed, not how fast.

Left open

The game measures whether a machine passes for a person, not understanding or intelligence itself.

• Adapts · Turing machines

  Section 5 argues that digital computers can mimic any discrete-state machine, which makes them universal machines.

Left open

It was an unfinished draft; many engineering details waited for the machines that were actually built.

Left open

A point-contact transistor is built very differently from the MOS transistors in today's chips.

Shannon gave his own example. Send 1,000 symbols a second, both equally likely, and let noise corrupt one in a hundred. The tempting answer is 990 bits per second, but the receiver does not know which symbols are wrong. By Shannon's measure, the real rate is only 919 bits per second.

The capacity theorem shows good codes exist without saying how to build them. Later codes closed in on the limit. A 1993 paper presented turbo codes under the title “near Shannon limit”. In 1996, long-neglected LDPC codes were rediscovered and, in simulations, came almost as close to the limit as turbo codes.

Left open

Entropy counts possibilities, not whether a message is true or useful, and capacity must be recomputed when the noise changes.

Take the half-response dose. On trial n, give dose x_n. A response lowers the next dose by a_n/2; no response raises it by a_n/2. The steps must meet two conditions: the squares of a_n have a finite sum, while the a_n themselves sum to infinity. The choice 1/n meets both.

Stochastic gradient descent (SGD) later faced the same kind of problem: each step estimates the gradient from one random example, so the direction is noisy, and the usual step-size conditions are the same. In Bottou's 2010 text-classification comparison, SGD reached the same test error in 1.4 seconds that SVMLight reached in 23,642 seconds.

Left open

It guarantees eventual convergence but says nothing about how many trials are enough.

A few numbers make the rule concrete. A move that raises the energy by one kT is accepted about 37% of the time. Raise it by three kT, and the chance is about 5%. So the system mostly drifts downhill but sometimes climbs. A rejected move still counts: the old configuration is recorded again, or some states would be underweighted.

In 1983 Kirkpatrick, Gelatt, and Vecchi carried the rule over to optimization. A cost function replaces energy, and the system is first “melted” at high temperature, then cooled slowly: simulated annealing. A search that accepts only improvements gets stuck in local optima; the Metropolis rule allows controlled uphill steps.

Left open

It proves the process tends toward the right distribution, but not how long that takes.

The difference can be counted. With 10 stages and 10 choices each, there are 10¹⁰ complete sequences. Bellman asks instead: starting from this state with N stages left, what is the best possible return? After the first decision, the best that can follow is the answer for N−1 stages from the new state, so only the first decision's options need comparing.

Reinforcement learning later borrowed the framework directly. Watkins and Dayan's 1992 paper describes Q-learning as a form of asynchronous dynamic programming: without knowing the rewards and transition probabilities in advance, it improves its estimate for each action in each state through repeated trials.

Left open

Every possible state needs its own value, so more state variables mean more to compute and store.

Coordinate selection relied on half currents. In Forrester's patent, each wire carries half the current needed to switch a core. A core on only one energized wire does not change state; only the core where the energized horizontal and vertical wires cross gets both halves and switches.

The price was that reading destroyed the contents: reading a core flips it, so the information is lost unless it is saved elsewhere. Whirlwind simply wrote the data back after every read. Transients from the rewrite disturbed the sense amplifiers, so a faster-recovering amplifier was designed.

Left open

Every read still had to be followed by a rewrite, and circuit interference needed better amplifiers.

Left open

Several approaches ran side by side, and none was bound to lead to LLMs.

• Complements · Information theory

  Shannon's part of the proposal applies information theory to computers and brain models.

• Complements · Logic Theory Machine

  At the 1956 Dartmouth project, Newell, Shaw and Simon presented IPL 2, the list language they had written the Logic Theorist in.

• Complements · LISP

  McCarthy recalled that his wish for an algebraic list-processing language for AI arose during the 1956 Dartmouth summer project.

Left open

The report mainly specifies the system and says little about running it on a computer.

The paper's example is 3-octanone. Valences alone allow 790 graphs, and 698 acyclic structures are chemically stable. Reading the peaks, the program concluded it was an ethyl ketone and put alcohols, ethers, aldehydes, and other ketones on its must-not list. It then generated just three structures, and the correct one was among them.

Looking back in 1984, Buchanan and Shortliffe wrote that MYCIN grew out of DENDRAL by reusing its lessons, foremost the power of production rules, the if-then kind.

Left open

It neared a chemist's skill only for a few classes of molecules, struggled with rings, and often returned several candidates.

• Complements · MYCIN

  Buchanan and Shortliffe call MYCIN an outgrowth of DENDRAL that reused its lessons on encoding knowledge.

Left open

How to keep the knowledge maintained, and what to do when no rule applies, stayed open.

Left open

When the graph has loops, local message passing may not converge, and exact inference can be intractable.

• Complements · MYCIN

  Shortliffe, MYCIN's lead author, and Heckerman later wrote that belief networks overcome many limits of MYCIN's certainty factors, which their lab had long stopped using.

• Complements · HMM

  Smyth, Heckerman and Jordan draw the HMM as a belief network and show that Rabiner's forward–backward and Viterbi algorithms are special cases of general graph inference.

Left open

Storing and forwarding at every hop left delay and congestion for later protocols.

Left open

Fibre solved transmission; routing and switching still happened in electronics.

Left open

The first attempt crashed after sending "lo"; it was a research network, far from the public Internet.

• Uses · Dijkstra

  From May 1979 every ARPANET node computed its routes with shortest-path-first, based on Dijkstra's algorithm.

Left open

The paper describes an experimental shared-cable network.

• Complements · ARPANET

  Metcalfe and Boggs place Ethernet after the ARPA network's work on packet switching and resource sharing.

• Complements · Packet switching

  The Ethernet paper is titled “Distributed Packet Switching for Local Computer Networks”, and its introduction credits the ARPA network with advancing computers as packet switches.

• Complements · Low-loss optical fibre

  The Ethernet paper says the Ether, its transmission medium, can be built from coaxial cable, twisted pairs or optical fibres.

Left open

When the switch date came, some hosts were still not ready; February surveys found services that did not respond.

• Adapts · ARPANET

  RFC 801 plans moving the operational ARPANET from its original host protocol to TCP/IP.

The paper offers a case the authors admit proves little on its own. A programmer took a one-day FORTRAN course and wrote a job in 47 statements in four hours. The 704 compiled it in six minutes into about 1,000 machine instructions. He estimated three days to code it by hand, not counting debugging.

Left open

It was written for the IBM 704 alone, and other formula translators were already running before it, though most did not try to optimize.

• Adapts · EDVAC report

  Backus later called FORTRAN and its successors “von Neumann languages”: variables imitate storage cells, and assignment imitates fetching and storing one word at a time.

Left open

It made symbolic reasoning easy to write, while numerical work still went to languages such as FORTRAN.

Left open

In 1969 work had only just begun; the system did not even have a name yet.

A string shows the difference. B had only the word, so several characters were packed into one word; to handle them one at a time, a program called library routines to unpack them and then packed them back. With C's char type, the i-th character is simply a byte that can be read directly.

Left open

Ritchie himself called C quirky and flawed: declarations read inside out, and arrays and pointers often trip up beginners.

• Complements · Unix

  C developed alongside early Unix, and in the summer of 1973 the PDP-11 Unix kernel was rewritten in C.

Left open

It covers only basic operations; solving equations or finding eigenvalues is left to libraries such as LAPACK.

Left open

The announcement came with no working system or compiler; actual work began in January 1984.

IBM's manual estimated about 600 milliseconds per disk read. With three items per invoice on average, the machine handled about 450 invoices an hour while updating every related record. An operator could also query any account from the console keyboard and get its current balance almost at once.

Left open

Seeking was still mechanical: the manual planned on about 600 milliseconds, and an engineer recalled about 800 in practice.

• Complements · The relational model

  IBM's history page credits RAMAC's random-access disk with paving the way for the relational database; this is IBM's own claim.

• Uses · Magnetic-core memory

  Every data transfer in the 305 RAMAC passed through a 100-character magnetic-core unit.

Left open

When the data is not in the fast store, the program still waits on the slow one.

Left open

The charge leaks away, so every cell must be refreshed periodically.

• Complements · Magnetic-core memory

  The patent contrasts its refreshed capacitor storage with remanent storage such as magnetic cores.

• Uses · MOSFET

  Dennard's one-transistor DRAM patent, “Field-effect transistor memory”, stores each bit with one field-effect transistor and a capacitor.

Left open

On a miss, the processor still has to go to main memory.

• Uses · System/360

  The Model 85 is a System/360 model. Liptay writes that its cache had to be loaded and replaced without program assistance because the Model 85 had to remain compatible with the rest of the System/360 line.

Left open

Each cell can be erased only so many times before it wears out.

Left open

Packing parts ever closer left heat and ever finer wiring to the decades that followed.

• Uses · The transistor

  Kilby's patent builds transistors, diodes, resistors and capacitors into one body of semiconductor.

Left open

As the devices shrank, leakage and heat grew harder to control.

How different were the models? The smallest, the Model 30, fetched 8 bits at a time from main storage with a 2.0-microsecond cycle. The largest, the Model 70, fetched 64 bits with a 1.0-microsecond cycle. On the Model 30 the sixteen general registers even sat in main storage, yet programmers always saw the same set.

Why 8-bit characters rather than 6? Numbers appear in business records more than twice as often as alphanumeric data, and packing each decimal digit into 4 bits makes better use of storage, tapes, and disks.

Left open

It guaranteed results only for valid programs that did not depend on timing; programs for IBM's older machines were not covered.

Would 65,000 components fit? Moore did the arithmetic. With existing tolerances, transistors could sit two thousandths of an inch apart: 500 per linear inch, a quarter million per square inch. So 65,000 components would need only about a quarter of a square inch, and existing optical techniques were enough. Components here include resistors and diodes as well as transistors.

Left open

It was an extrapolation from a few years of data, and Moore called the longer term a bit uncertain.

• Uses · Integrated circuit

  Moore's 1965 paper forecasts integrated circuits: how the cheapest number of components per circuit will grow.

Left open

Programs still lived in external ROM, so it was not yet a complete computer on a chip.

• Complements · Moore, 1965

  In a 2003 retrospective, Moore placed the 4004 at the start of the microprocessor series on his complexity and performance curves.

• Uses · Integrated circuit

  The MCS-4 announcement presents the chip set as four new ICs.

Left open

It needs the voltage to fall too; when voltage stops falling, power density rises.

• Uses · MOSFET

  The paper gives the rules for shrinking a MOSFET: scale dimensions and voltage together and power density stays the same.

Left open

Simpler instructions make programs longer and shift more of the work to the compiler.

• Complements · System/360

  The RISC paper cites one IBM 360 compiler in which 10 instructions accounted for 80% of execution.

Left open

It only works when no edge has a negative weight.

Left open

The original form assumes linear systems and Gaussian noise; nonlinear problems need approximations.

Try N = 1,024. Direct computation takes about 1.05 million operations; the new method needs fewer than 20,480. That is roughly fifty times fewer, and the gap widens as N grows. The price is scrambled output: result j lands at the position given by reversing the binary digits of j.

In a 2005 paper, the authors of FFTW wrote that their widely used free library still relies mainly on the Cooley–Tukey algorithm, but the hard part has moved: splitting causes scattered memory access that works against caches. So FFTW times many ways of splitting and picks the fastest.

Left open

It does not help when N is prime, and on today's machines memory access matters as much as arithmetic.

Left open

Every intermediate value must be kept, so longer computations need more memory.

• Complements · Backpropagation

  Griewank credits Linnainmaa's 1970 thesis with what is now called the reverse mode, and treats back-propagation in neural networks as a special case of it.

Left open

No step makes things worse, but it can stop at a local optimum, so a poor start can leave it stuck.

Databases later ran machine learning themselves: the 2012 MADlib paper implemented regression and clustering inside PostgreSQL and Greenplum, so the computation stayed next to the data.

Left open

The model itself says nothing about distributed transactions, physical execution plans, or data quality.

Picture a perceptron with two responses, one for “square” and one for “circle”. An image on the sensory points activates some association units; each response sums the units wired to it, and the stronger wins and suppresses its rival. When images in a class resemble one another, an unseen image can be classified correctly with better-than-chance probability.

Rosenblatt also stated a boundary. A perceptron could learn responses to concrete stimuli, but once a response required a relation between stimuli, such as “name the object left of the square”, the problem became excessively difficult.

Left open

It could not handle judgments about relations between stimuli, as Rosenblatt himself admitted.

Left open

It had no supervised training such as backpropagation; each layer learned through a purpose-built self-organising rule.

In simulations with 30 units and five stored memories, a start within 5 bits of a memory returned to it more than 90% of the time. Fed a 60–40 mix of two memories, it usually picked the larger one; a linear network would have returned a blend.

In 2024 Hopfield and Hinton received the Nobel Prize in Physics. The announcement says the Hopfield network borrows the physics of atomic spin to give stored images low energy, and that Hinton built on it to develop the Boltzmann machine.

Left open

It only associates; it cannot learn hidden features, and recall breaks down once too many memories are stored.

• Complements · Perceptron

  Hopfield sets his model against perceptrons: strong backward coupling and asynchronous units.

In one experiment the task was to block a door with box 2. Partway through, the experimenters put an unknown box in front of a door on Shakey's route. The camera saw that the way was blocked, and the box went into the model. PLANEX did not replan from scratch; it went around through a different set of rooms and still blocked the door.

This worked because STRIPS stored plans with parameters. Instead of “go through these rooms”, a plan said “go from some room through an intermediate room into a third, then block the door there.” The next task, pushing the new box aside, reused the first four steps of that plan. The report treated this as a form of learning.

Left open

It worked only in a few specially arranged rooms, with doors and walls written into its model in advance.

• Uses · FORTRAN

  Shakey's software was mostly LISP but could call FORTRAN subroutines, and its camera pictures were stored as FORTRAN binary files.

• Uses · LISP

  Shakey's technical note names LISP as the team's principal programming tool.

Left open

It was compiled by hand, so which words appear and how they link depends on its editors.

Left open

Gradients alone did not make deep networks easy to train, or make them work on new data.

Left open

It was tested only on a small vowel-recognition task.

• Complements · Backpropagation

  The paper opens by noting that one network trained by backpropagation on several subtasks suffers strong interference; the mixture splits the work among networks.

Left open

Gates ease the problem, but over long enough distances the network can still forget.

Three problems need to be kept apart: how likely an observation sequence is, which state path is most likely, and how to learn the parameters from data. The first sums over all paths, the second picks the best one, and the third estimates parameters; each needs its own algorithm.

Left open

It assumes the next state depends only on the current one, which keeps computation simple but limits the dependencies it can express.

• Complements · EM algorithm

  Rabiner notes that Baum–Welch re-estimation for HMMs is equivalent to the EM algorithm.

The paper gives an example: English nodding corresponds to the French phrase faire signe que oui. Models 1 to 3 link nodding only to signe and oui; Model 5 links it to the whole phrase.

Later statistical translation often treated word alignment as an intermediate step. The 2003 phrase-based model of Koehn, Och, and Marcu first word-aligned a corpus with Giza++, a toolkit for training the IBM models, then extracted phrase pairs from the result.

Left open

It ignores word forms, so each of a French verb's dozens of forms is learned separately, and it was tested only on French and English.

• Uses · EM algorithm

  IBM's five translation models estimate their parameters by iterating the EM algorithm.

Left open

Kernel methods predate this paper; what it added was a way to handle data that cannot be cleanly separated.

• Adapts · Perceptron

  Cortes and Vapnik start from Rosenblatt's perceptron: a fixed non-linear map, then a linear surface.

Left open

Convergence conditions for tables do not automatically hold once a neural network replaces the table.

• Adapts · Dynamic programming

  The paper calls Q-learning incremental, asynchronous dynamic programming: without knowing rewards and transition probabilities in advance, it approaches from experience the optimal action values that dynamic programming would compute. It cites Bellman and Dreyfus's 1962 book rather than this 1954 article.

Left open

It was still only a proposal: no website was running, and the name World Wide Web did not exist yet.

Three things are worth separating. The kernel handles processes, memory, and devices. Shells, compilers, and the C library run on top of it. A distribution packages them so they can be installed and used. The 1991 release was only a kernel; its release notes say the shell, compilers, and library had to be found elsewhere.

When the license changed, it waited on approval from the people who had helped write code, so others were already contributing. Linux also turns up in later stories: the 2004 MapReduce paper describes Google's cluster machines as typically running Linux.

Left open

The 1991 release was only a kernel; making it usable took the GNU tools and many contributors.

Several traits trace back to this history. Indentation for grouping and high-level built-in data types came from ABC; exceptions came from Modula-3, and van Rossum says error handling in Amoeba taught him how much they matter. The official FAQ still lists extensibility in C or C++ among its basic traits.

Scientific computing and machine learning later leaned on that extensibility. The 2019 PyTorch paper observes that since 2014 most deep learning frameworks have treated a Python interface as essential.

Left open

It was not first meant for numerical work; fast array operations come from extensions written in C.

Left open

MPI specifies interfaces, not a single implementation, and the early standard left out task management and I/O.

• Uses · FORTRAN

  MPI-1 standardised bindings for Fortran 77 and C.

The input was tiny: the video image had just 30×32 points. The output was a row of 45 units, the middle one meaning straight ahead and outer ones meaning sharper turns. With a realistic road simulator, backpropagation produced a fairly good road follower in half an hour; traditional methods had taken many months of tuning to reach a similar level.

A 1990 follow-up learned by watching a person drive. A network that only sees good driving never learns to recover from drifting, so each image was shifted and rotated in software into 14 more. The network processed 25 images a second, enough to drive at the NAVLAB's top speed of 20 miles per hour.

Left open

The 1988 network was tested on the vehicle only along one campus path on sunny days, and at forks its steering swung back and forth.

• Uses · Backpropagation

  ALVINN is a three-layer backpropagation network, trained on 1,200 simulated road images with a backpropagation simulator on the Warp.

Left open

It relied on navigation behaviors and support from the ground, not on reinforcement-learning training.

• Uses · C

  All of Sojourner's flight software was written in C and ran on an Intel 8085.

Most of the chess knowledge was put in by people. The opening book of about 4,000 positions was prepared by hand, mainly by grandmaster Joel Benjamin, and the weights of the 8,000-plus evaluation features were mostly tuned by hand.

The 2016 AlphaGo paper used Deep Blue as a contrast. AlphaGo evaluated thousands of times fewer positions, making up for it with a policy network that chose positions better and a value network that scored them more precisely; Deep Blue relied on a handcrafted evaluation function.

Left open

It played chess and nothing else, and its evaluation was fixed in silicon, so no new features could be added.

The terms are concrete. Shipping a compiled program requires either the complete source or a written offer, valid for at least three years, to supply it for no more than the cost of distribution. Source code means the preferred form of the work for making changes.

It also set limits. Putting an independent work on the same storage medium is mere aggregation and does not bring that work under the license. If a program says “any later version” applies, users may follow the newer terms.

Left open

The duty to share source applies only when you distribute the program, and the license promises nothing about code quality.

• Complements · GNU

  In the GNU project's account, keeping GNU software from being turned into proprietary software required matching distribution terms; the GPL is the copyleft used for most GNU software.

The plan was about 100 submissions a year; it received 400 in its first half year. Old papers were supposed to be deleted after three months, but people found it easier than hunting for a photocopy, so the deletion job was never installed and nothing was ever deleted.

Ginsparg recalls that physicists in less developed countries soon felt finally in the loop, receiving new ideas promptly and having their own work read on equal terms.

Left open

Moderation checks topic and scholarly value; papers posted there have not been peer reviewed.

• Uses · The Web

  It began as an email server and added FTP within months; not long after the Mosaic browser appeared in 1993, it became a Web server as well.

In a controlled comparison, four annotators corrected machine-tagged text at an average of 20 minutes per 1,000 words, against 44 minutes tagging from scratch. Disagreement between annotators was 4.1% when correcting and 7.2% when tagging by hand.

Later parsing research used it as a standard training and test set. Collins's 1997 parser, trained and tested on the treebank's Wall Street Journal portion, reached 88.1% constituent precision and 87.5% recall.

Left open

Its syntax is only a skeleton, with noun phrases mostly left unanalyzed, and the text is mainly American English.

The paper works through a small example. In past ratings, Ken and Meg rose and fell together, while Ken and Lee mostly disagreed. On a new article Meg gave a 5 and Lee a 2. Because the person who liked it shares Ken's taste and the one who disliked it has the opposite taste, Ken's predicted rating is 4.56.

The authors also planned how to compare algorithms: delete some ratings from the matrix and see who recovers them best. The Netflix Prize, from 2006, compared predictions the same way, by hiding known ratings.

Left open

It cannot predict anything for an article nobody has rated, and new users must build up ratings before predictions help.

Here is the paper's formula with its usual parameters. In a document of average length, one occurrence of a term scores 1. Ten occurrences score only 2.5. The same ten occurrences in a document twice the average length score about 2.22.

The formula later went into widely used open-source search software: in 2016, Apache Lucene made BM25 its default scoring.

Left open

BM25 only counts query terms and document length; it knows nothing of synonyms or word order.

Left open

With design and manufacturing separated, the most advanced processes sit with a few foundries.

Weight sharing means using the same local detectors at every position in an image instead of learning a separate set for each position. Later layers then combine those local features.

Left open

Its subsampling and output layers work differently from today's usual max pooling and softmax.

• Uses · Backpropagation

  It names backprop-trained multilayer networks as the leading example of gradient-based learning.

• Complements · Neocognitron

  The LeNet paper notes that the Neocognitron had already implemented convolution with subsampling, though no supervised procedure such as backpropagation existed then.

Two pages with the same number of incoming links need not score the same: which pages the links come from, and how many outgoing links those pages have, change how much weight passes along. PageRank is still only one ranking signal and cannot replace text matching or indexing.

Left open

Many links show attention; whether a page is right, or relevant to a query, is a separate question.

• Uses · The Web

  The prototype crawls and indexes the Web and exploits its hypertext links.

Left open

The original was hard to read, so in 2001 Lamport retold it plainly in “Paxos Made Simple”.

Word counting makes it clear. Map emits a count for each word, the runtime groups the records by word, and reduce adds them up. The price is that the computation must fit this batch-processing pattern.

Left open

It is a batch model and cannot cover arbitrary interactive computation.

• Uses · Linux

  The paper's clusters are dual-processor x86 machines running Linux, with local IDE disks and commodity networking.

High performance per watt at full load and low power during quiet periods are different properties. The 2017 TPU study measured power across a range of loads.

Left open

It was a design goal; when it was proposed, the industry had not achieved it.

A worked example: a facility draws 15 million kWh from the grid in a year, and 10 million kWh reaches the IT equipment. Its PUE is 1.5. Lawrence Berkeley National Laboratory measured 22 datacenters, with PUE between 1.3 and 3.0.

The boundary of the measurement matters. Google reports a fleet-wide average PUE of 1.09 for 2025 and says it counts overhead such as substations and transformers; these are Google's own measurements.

Left open

PUE describes the facility, not the computing: energy burned by idle servers still counts as IT energy.

Left open

The integrated-service model in the book does not describe every colocation facility.

• Uses · PUE

  Its efficiency breakdown starts with PUE, citing The Green Grid.

• Complements · Flash memory

  The book notes that flash, like DRAM, cost too much per gigabyte for large or rarely used data, so disks still dominated.

TVM's 2018 system includes a code-generation path through LLVM's intermediate representation. AI compilers can reuse such general infrastructure, but tensor layouts, operator fusion, and hardware scheduling still need specialized work.

Left open

It is a low-level intermediate representation, far from a full model compiler, and performance still depends on later optimization.

A NumPy array is a block of memory plus a few pieces of metadata: element type, shape, and strides. On top of this sits vectorization: arithmetic or sums over a whole array, which might take tens of lines of loops in C, are often one Python expression.

By 2020, SciPy, Matplotlib, pandas, and scikit-learn were built on NumPy, and it was part of the software used to detect gravitational waves and produce the first image of a black hole.

Left open

NumPy handles arrays in one machine's memory on the CPU; GPUs and data beyond one machine need other libraries.

• Uses · Python

  NumPy is Python's array library but not part of the standard library; the paper notes that Python added features and special syntax over the years to make NumPy's array notation more concise.

• Uses · The FFT

  NumPy provides a suite of FFT and inverse-FFT functions.

• Uses · BLAS

  NumPy accelerates its linear algebra with back-ends such as OpenBLAS or Intel MKL.

Compare vector addition, c = a + b. Through a graphics interface, you load a and b into two textures, write a shader, set up a render target, draw a rectangle covering it, and read the result back from the “image”. With CUDA, each thread uses its index i to compute c[i] = a[i] + b[i].

Deep learning later took the same road. The AlexNet authors wrote their own GPU convolution code in 2012, and in 2014 NVIDIA presented cuDNN; plugged into Caffe, it made one standard model 36% faster while using less memory.

Left open

CUDA then ran only on NVIDIA's own GPUs, so programs could not simply move to another vendor's hardware.

• Uses · The FFT

  CUDA 1.0 shipped two maths libraries, one of them the FFT library CUFFT.

Left open

The whole graph had to be built before compiling, which made debugging and dynamic structures awkward.

The 2017 TPU paper adapted the model to compare neural-network bottlenecks on CPUs, GPUs, and TPUs, recounting operations and bytes for quantized computation.

Left open

It gives only a ceiling; actual speed still depends on latency, dependencies, and available parallelism.

Left open

The HD 7970 was announced in late 2011 but did not reach retail until January 2012.

Picture a planar transistor as a channel laid flat, with a gate pressing down from above. Tri-Gate stands the channel up as a thin wall and clamps it from left, right, and top. With control from three sides, as much current as possible flows when it is on, current comes as close to zero as possible when it is off, and switching is faster.

At low voltage, Intel reported up to 37% more performance than its own 32 nm transistors. It showed a 22 nm processor code-named Ivy Bridge on the day of the announcement, and the chips went on sale in April 2012 as the third-generation Core family.

Left open

Tri-Gate improves the gate's control over current; a chip's speed depends on much else.

• Adapts · The transistor

  Intel calls Tri-Gate a reinvention of the transistor: a thin 3-D fin replaces the flat gate.

Left open

It compares against reference translations, and with fewer references, valid alternative wordings are more likely to be penalised.

Left open

The crawl holds everything, so it needs heavy filtering and deduplication before training.

Left open

The full ImageNet collection differs from the subsets later used in the ILSVRC competitions.

• Uses · WordNet

  ImageNet uses WordNet's hierarchy and aims to fill about 80,000 noun synsets with images.

The 2016 TensorFlow paper lists RoCE alongside gRPC over TCP for communication between tasks, so RDMA did reach machine-learning systems. RoCE is a different protocol path, though, so this does not show that TensorFlow used RDMAP.

Left open

Fewer copies still leave the costs of communication, synchronization, and buffer management.

• Uses · TCP/IP

  RDMAP runs over DDP/MPA on TCP to avoid the copies TCP/IP normally needs.

Because every switch is the same, all of them can be cheap commodity parts. The paper estimates the switching hardware at $8.64 million, against about $37 million to give 20,000 hosts full bandwidth with high-end switches.

The hard part is using all those paths. Ordinary routing picks just one, so traffic piles onto a few core switches. The paper spreads traffic by destination address while keeping each host's packets on one path. In a 16-host emulation under random traffic, this reached 75% of the ideal bandwidth, against 53.4% for a conventional tree oversubscribed 3.6:1.

Left open

The experiment emulated only 16 hosts, and the large-scale costs were estimates that left out the far greater cabling.

• Complements · MapReduce

  The paper cites MapReduce as an example of inter-node bandwidth becoming the bottleneck: map output must be shuffled across machines before the reduce phase can begin. Fat-tree targets this kind of cross-machine traffic.

Left open

Its single-writer, many-reader model targets large-file throughput and offers no database-style transactions.

The list has 128 numbers: a 16×16 sample grid around the keypoint, split into 4×4 cells, each with an 8-direction gradient histogram. Matching uses a practical rule: discard a match if the nearest distance is more than 0.8 times the second nearest. On a database of 40,000 keypoints, this removed 90% of false matches.

SIFT was still among the leaders at the 2012 ImageNet competition. A method built on SIFT and Fisher vectors took second place with 26.2% top-5 error, against AlexNet's 15.3%.

Left open

SIFT ignores color, and it recognizes specific objects already in a database, not categories like “cat.”

The simplest case is a count. Each person is a row with value 0 or 1, and the query asks how many 1s there are. Replacing one person changes the answer by at most 1, so the sensitivity is 1, and adding Laplace noise of scale 1/ε is enough. A smaller ε hides each person better and needs more noise.

The paper called its definition ε-indistinguishability; the later standard name is differential privacy. The U.S. Census Bureau says it moved to a protection framework based on differential privacy for the 2020 Census.

Left open

It limits one person's influence, not what statistics reveal about groups, and more questions demand more noise.

The price was a modified operating system. Porting Linux took about 3,000 changed or added lines, 1.36% of its x86-specific code. Management lives in a privileged domain called Domain0, which creates and destroys other virtual machines. The design target was about 100 virtual machines per server.

Cloud services later built on it. When Amazon released the EC2 beta in 2006, the launch post stated that EC2 was built on Xen.

Left open

Guest systems had to be modified; when the paper was written, Windows XP could run only simple programs.

• Uses · Linux

  The paper ports Linux 2.4 to Xen as XenoLinux, which also hosts the Domain0 management software.

Barr's launch post cited an internal inventory application used only from 7 a.m. to 7 p.m.; shutting the instance down at night would cut server costs by about half.

A boot disk was packaged as an AMI and stored in S3, and web service calls launched instances from it; the virtualization underneath was Xen. During the beta an account could run up to 20 servers at once.

Left open

At launch, local data vanished when an instance stopped, and there was no load balancing or Auto Scaling yet.

• Uses · Xen

  The launch post states that EC2 was built on the Xen virtual machine monitor; Xen supplied isolation, while hourly rental was EC2's own service design.

Small pose errors got magnified. The lasers hit the ground up to about 30 meters ahead, so an error under half a degree in estimated pitch or roll created phantom obstacles on flat ground. The team had a person drive only over clear ground and used those rough labels to tune parameters automatically. The share of drivable area misread as obstacle fell from 12.6% to 0.002%.

Vision mattered too. The paper estimates that without the camera, capped at 25 miles per hour the whole way, Stanley would have taken about 7 hours and 5 minutes and possibly finished behind CMU's Sandstorm.

Left open

The race environment was static; the authors say Stanley could not drive in traffic or tell tall grass from rocks.

• Adapts · ALVINN

  To tune its laser mapper, Stanley used obstacle-free ground driven by a person as drivable labels, which the paper calls similar in spirit to Pomerleau's work; its vision-based road finding also cites the 1991 ALVINN follow-up. Unlike ALVINN, Stanley steered through a path planner and controller, and used its vision results only to set speed.

• Uses · Kalman filter

  Stanley estimates the vehicle's position, orientation and velocity with an unscented Kalman filter.

The article gives a small example. Say the average rating is 3.7 stars, Titanic runs 0.5 stars above average, and Joe, a critical user, rates 0.3 stars below average. From biases alone, Joe's estimate for Titanic is 3.9 stars; only the rest is left for the user and movie vectors to explain.

The prize went to BellKor's Pragmatic Chaos, whose winning submission was 10.06% better than Netflix on the official leaderboard. The Ensemble had exactly the same score but submitted about 20 minutes later.

Left open

The winning entry blended more than 100 predictor sets, and new users and movies with few ratings stayed hard.

• Complements · GroupLens

  The article presents matrix factorisation as the alternative to neighbourhood methods, the like-minded-users approach GroupLens used.

Left open

Computing probabilities over the whole vocabulary made training slow for large vocabularies.

Left open

It is guaranteed to pick the best action only as simulations grow; with few simulations there is no such promise.

A few changes made training work: ReLU activations trained several times faster; dropout and random crops fought overfitting; and since one GPU could not hold the network, it was split across two.

Left open

It picks one answer from a thousand categories; what is in the picture, where, and how things relate were out of reach.

• Uses · ImageNet

  Training and evaluation use ILSVRC subsets of ImageNet; the dataset and competition setup are part of the result.

• Uses · CUDA

  The paper provides GPU convolution code through cuda-convnet. Training is also constrained by the memory and communication of its two GPUs.

• Complements · SIFT

  Its results tables compare against SIFT + Fisher Vector pipelines, the previous best.

• Complements · Multi-column GPU CNNs

  AlexNet calls its two-GPU layout somewhat similar to the “columnar” CNN of Cireşan et al., citing the group's 2011 GPU CNN paper.

Left open

Its results were on small-image datasets; large-scale collections such as ImageNet were not attempted.

Left open

Once networks were trainable, adding depth still did not always help.

• Uses · Faster R-CNN

  ResNet's detection experiments use Faster R-CNN as the framework and only swap VGG-16 for ResNet-101, raising the standard COCO metric by 6.0 points.

Left open

A game can be reset after every failure; a physical robot pays a different price.

• Adapts · Q-learning

  DQN is a convolutional network trained with a variant of Q-learning.

Left open

Each word gets one fixed vector that does not adapt to the sentence it appears in.

• Adapts · Neural language model

  word2vec starts from the feedforward neural network language model of Bengio et al. and drops its costly non-linear hidden layer to learn word vectors alone.

Left open

The whole sentence is squeezed into one fixed-length vector; attention later let the decoder look back at every input word.

The paper's alignment plots make this concrete. “European Economic Area” becomes “zone économique européenne”, in reverse order: when producing “zone” the model skips the first two words to look at “Area”, then steps back one word at a time.

The longer the sentence, the further the old encoder–decoder fell, while the new model showed no drop even beyond 50 words. Later systems kept the idea: the 2017 Transformer called it additive attention, then switched to scaled dot-product attention and dropped recurrence altogether.

Left open

It scores every source word for every target word, and its 30,000-word vocabulary left rare words unsolved.

• Adapts · Seq2Seq

  Extends the RNN encoder–decoder of Cho et al. and Sutskever et al.: the encoder keeps a vector per word, and the decoder recomputes weights at each step instead of relying on one fixed-length vector.

The paper's log-mining example takes three lines: read logs from HDFS, keep the lines that start with ERROR, and ask for the result to stay in memory. Later queries for errors mentioning MySQL run on the in-memory data. If a machine holding some of those lines fails, Spark refilters just the matching part of the original logs.

In a k-means run on 75 machines, the authors killed one machine at the start of the sixth iteration. That iteration rose from about 58 seconds to 80; once the lost partitions were rebuilt, the next one was back to 58.

Left open

It suits batch work that applies one operation to a whole dataset, not fine-grained updates to shared state like web-app storage.

• Uses · HDFS

  The log example builds an RDD from an HDFS file, and the iterative experiments read text from HDFS in their first iteration; data meant for reuse then stays in memory, so durable storage and in-memory reuse are separate layers.

• Adapts · MapReduce

  The paper shows that MapReduce can be expressed with flatMap and groupByKey, and it borrows MapReduce's backup tasks for slow nodes; the difference is that intermediate results can stay in memory for reuse.

Take gradient descent. Workers push gradients to the servers, the servers sum them and update the weights, and the workers pull the new weights back. Each worker pulls only the parameters that appear in its own data; with 100 workers, each needed 7.8% of the total.

Synchronization is tuned with a maximum delay τ: τ = 0 waits for everyone every round, and a larger τ waits less. On sparse logistic regression the authors found τ = 8 the best trade-off. Their system needed 300 lines of code for an algorithm that took more than 10,000 lines in the special-purpose systems they compared against.

Left open

Relaxed synchronization is not free: less waiting means more iterations to converge.

Here is how it worked. A job states an upper limit on resources. Every few seconds Borg estimates a reservation from actual usage, and the gap between limit and reservation runs batch work that can tolerate lower-quality resources. In a median cell, about 20% of the workload ran in reclaimed resources.

Most Borg tasks ran in Linux containers, not virtual machines. The paper ends with how these lessons carried over to Kubernetes; for example, Borg's alloc became the Kubernetes pod.

Left open

These figures come from Google's own clusters and workloads; another company's could differ.

• Uses · Linux

  Each Borg task is a set of Linux processes in a cgroup-based container whose limits the Borglet adjusts.

• Uses · Paxos

  Borgmaster keeps its state in a highly available, Paxos-based distributed store and elects its master with Paxos.

Left open

Orchestration is only one part of the container toolchain, separate from the compute resources underneath.

• Adapts · Borg

  Joe Beda recalls that Kubernetes built on ideas Google had proven with Borg over ten years, though the code was written anew.

Left open

HBM itself predates 2015; this was the year it entered a GPU product.

Looking back, the authors were direct: most of the gain came from letting the network use neighboring frames and from predicting context-dependent states directly. Layer-wise pre-training reduced overfitting, but similar savings in training time could come from carefully scaling the initial weights.

The cost was computation, since a deep network uses nearly all its parameters for every frame. Recognizing 1 second of speech took 1.6 seconds. Quantizing the weights and using fixed-point instructions on a CPU brought that to 210 ms; a GPU took 66 ms.

Left open

The deep network replaced only the GMM; the HMM, language model, and decoder stayed, so end-to-end recognition was still far off.

• Uses · HMM

  The deep network replaces the GMM: it outputs posteriors over HMM states, which are converted to scaled likelihoods for HMM alignment and decoding.

The key to the proposal network is the anchor. Each position on the feature map gets 9 reference boxes: 3 sizes times 3 aspect ratios. For every anchor the network answers two questions: does it contain an object, and how should the box shift and resize?

In the paper's timing table, the old pipeline took 1,830 ms per image, 1,510 of them for proposals; with the proposal network the total was 196 ms. Accuracy held up too: on the PASCAL VOC 2007 test set, mAP was 73.2%, against 70.0% for the Selective Search version.

Left open

It is still two-stage, propose then classify, and its rectangular boxes cannot trace an object's outline.

• Uses · ImageNet

  The convolutional layers shared by the RPN and the detector are initialized from models pre-trained for ImageNet classification.

Left open

Touching cells of the same class tend to merge, and separating them takes a loss that weights the borders between them.

The paper illustrates the style with a handwritten-digit classifier. First the program builds the graph: inputs, weights, a matrix multiplication, ReLU, a loss, and an optimizer, with nothing computed yet. Then it opens a session, feeds in batches, and runs the training node. Because execution waits until the whole program is known, the runtime can plan globally.

The parameter server is no longer a separate system: some tasks take on that role, but they can run any graph. The 2019 PyTorch paper groups TensorFlow with frameworks that build a static graph first, the contrast for its own eager design.

Left open

The graph is built before it runs, which constrains algorithms like deep reinforcement learning whose structure unfolds at run time.

• Adapts · Parameter server

  The paper places Li and colleagues' system in the parameter server line as the step that added consistency models, fault tolerance, and elastic rescaling; TensorFlow has no separate parameter server and lets some tasks play the PS role while running arbitrary graphs.

• Complements · Theano

  TensorFlow's related-work section says Theano's programming model is the closest to its own.

Picture two parameters, one with gradients around 1,000 and one around 0.001. SGD with a learning rate of 0.001 moves the first by 1 per step and the second by a millionth. Under Adam, when a gradient's direction is steady, both move by about one learning rate per step.

Later models simply adopted it: the 2017 Transformer and BERT's 2018 pretraining both used Adam.

Left open

Its convergence proof covers only convex problems, and later work found a simple convex case where Adam misses the optimum.

Left open

Training two networks together is unstable, and the generator can collapse to a few kinds of samples.

Left open

Research on graph neural networks came earlier; GCN is one representative paper.

Left open

Whether it generalizes still depends on node features, neighborhoods, and the data distribution; not every new graph will work.

• Adapts · Graph convolutional networks

  Its convolutional aggregator is an inductive version of Kipf and Welling's GCN.

Left open

It appeared almost alongside GraphSAGE; the two are parallel paths.

• Adapts · Transformer

  GAT brings Transformer-style self-attention to graph neighbourhoods and names Vaswani et al. as its inspiration.

For 50 tokens, a recurrent network takes 50 sequential steps. Bidirectional self-attention can connect any two positions in one layer; a decoder's causal mask blocks future positions. Standard dense attention forms a 50 × 50 score matrix, so this computation grows quadratically with sequence length at a fixed hidden size. Parallel training does not make autoregressive generation emit every token at once.

Left open

Generation still emits one word at a time, and the quadratic cost on long inputs was left for later work.

• Adapts · Attention for translation

  The paper lists Bahdanau et al.'s additive attention as one of the two common attention functions; Transformer uses scaled dot-product attention instead and drops the recurrent network.

• Uses · GNMT

  The English–French experiments reuse GNMT's 32,000-wordpiece vocabulary, and GNMT with RL is a baseline in Table 2.

• Uses · Adam

  Training uses Adam with β1 = 0.9, β2 = 0.98 and ε = 10⁻⁹; the learning rate rises linearly for 4,000 steps, then decays with the inverse square root of the step number.

• Complements · Turing machines

  Later theory proves that a Transformer with positional encodings and hard attention can simulate any Turing machine.

• Uses · BLEU

  The Transformer reports its results in BLEU: 28.4 for English–German and 41.0 for English–French.

The paper compares neighbors of “play”. With GloVe, the closest words are playing, game, and football, nearly all about sports. With ELMo, “play” in a baseball sentence finds a sentence praising a player's play, while “play” in a sentence about a Broadway production finds one about actors in a successful stage play.

Different layers learn different things. For word sense, the second layer does better; for part-of-speech tagging, the first does. So ELMo lets each task learn its own mix of layers. Added to a baseline for SQuAD question answering, it raised F1 from 81.1 to 85.8.

Left open

Each task still needed its own model, and the two directions were separate one-way models joined together.

• Uses · LSTM

  ELMo's language model is a two-layer bidirectional LSTM, and word representations are weighted combinations of its layers.

• Uses · Penn Treebank

  ELMo tests its representations with part-of-speech tagging on the WSJ Penn Treebank.

Left open

It looks both ways, which suits understanding; writing text word by word still needs a model that only looks back.

• Uses · GLUE

  The paper reports fine-tuning results on GLUE; the final version's abstract gives 80.5% (80.4% in the first 2018 version).

• Uses · SQuAD

  The paper evaluates question answering on SQuAD; its abstract reports a test F1 of 93.2 on SQuAD v1.1 and 83.1 on SQuAD v2.0.

• Uses · Okapi BM25

  Nogueira and Cho fine-tune BERT to re-rank BM25's top 1,000 passages, beating BM25 and earlier neural rankers.

Training had three stages. A policy network learned to imitate human experts from 30 million positions. Reinforcement learning then improved it through games against its earlier versions. Finally, a value network was trained on self-play positions. One value-network evaluation came close to the accuracy of playouts with the policy network while using 15,000 times less computation.

AlphaGo Zero later dropped the human games and learned from the rules and self-play alone. After three days of training it beat the version that had defeated Lee Sedol 100–0.

Left open

Go has known rules, a fully visible board, and unlimited simulation; most real decisions come with no such simulator.

• Complements · Deep Blue

  AlphaGo evaluated thousands of times fewer positions than Deep Blue, chosen by its networks.

• Uses · UCT

  AlphaGo's introduction cites Kocsis and Szepesvári's UCT for Monte Carlo tree search.

Left open

Clipping only reins in each update; the actual KL gap between old and new policies can still grow.

Left open

Learning a skill in simulation does not make it safe on physical hardware.

• Complements · PPO

  SAC is an off-policy alternative to on-policy PPO, which it also uses as a baseline.

Suitable neural-network operations can use it, but data still has to move in and out, and other operations still have to run. Peak matrix throughput does not tell you how fast a whole program will be.

Left open

This generation of Tensor Cores does not do FP64 matrix multiplication.

• Adapts · NVLink

  The V100 whitepaper states that NVLink was introduced in 2016 with the P100; V100's second generation raises the per-pair rate from 20 to 25 Gb/s and the link count from four to six.

The binding constraint was response time: one application had to answer 99% of requests within 7 milliseconds. Within that limit the TPU reached 80% of its maximum throughput. The CPU and K80 had to run small batches and reached only 42% and 37%.

The bottleneck was often memory, not arithmetic. Of the six applications, the two MLPs and two LSTMs were limited by memory bandwidth and only the two CNNs by compute. The authors estimated that GDDR5 memory like the K80's would make the TPU two to three times faster.

Left open

It was compared with 2015-era chips, so the ratios do not carry over to later hardware, and training still ran on GPUs.

• Adapts · Roofline

  The paper uses Roofline to locate bottlenecks for six applications on the CPU, GPU, and TPU, replacing floating-point with integer operations and redefining operational intensity as operations per byte of weights read.

• Uses · Energy proportionality

  Section 6 cites Barroso and Hölzle's 2007 case for energy proportionality and measures power for the three chips from 0% to 100% load; at 10% load the TPU still draws 88% of its full-load power.

• Complements · RISC

  The TPU paper says its instructions follow the CISC tradition, averaging 10 to 20 clock cycles each.

Left open

Documentation supports scrutiny; it does not certify that data is fair, accurate, or fit for use.

• Uses · Gender Shades

  Datasheets opens with Gender Shades as an example: commercial gender classifiers err more on darker-skinned women.

The paper built a test from the complete works of Shakespeare: each speaking role became a client, 1,146 in all, with very different amounts and kinds of text. When clients ran several local passes before uploading, the rounds of communication needed for the same accuracy fell 95-fold compared with computing one gradient per round.

Updates can still leak. With sparse bag-of-words features, the nonzero gradients show exactly which words a user typed. Stronger guarantees can be layered on with differential privacy or secure multiparty computation.

Left open

Raw data stays put, but updates can still leak, and the paper set aside deployment problems like devices dropping out.

• Complements · Differential privacy

  The paper names differential privacy as a stronger guarantee that can be layered on, at some cost in utility from added noise.

Left open

Filling in a card is only a first step; safety audits and independent evaluation still have to be done.

• Complements · Datasheets

  One reports model uses and behavior; the other documents data provenance and limits. The Model Cards paper explicitly treats them as complementary.

• Uses · Gender Shades

  The Model Cards introduction cites Gender Shades on face-attribute detection performing unevenly across groups.

Left open

It audited gender classification alone; other tasks need their own breakdowns.

To see far enough back, it uses dilated causal convolutions: each layer looks only at earlier samples, and the gap doubles from layer to layer, from 1 up to 512. One such block sees 1,024 samples. Training runs in parallel; generation must compute each sample and feed it back before the next.

Naturalness was rated on a five-point scale. In US English WaveNet scored 4.21, against 4.55 for real recordings. Generating one sample at a time suited parallel hardware poorly and made real-time use hard. The later Parallel WaveNet trained a network from a WaveNet that generates in parallel, running more than 20 times faster than real time.

Left open

It only turns linguistic features into a waveform; getting those features from text still needed separate models.

• Adapts · ResNet

  WaveNet uses residual connections (He et al.) plus skip connections throughout.

The gate scores every expert, adds a little noise, and keeps only the top few; experts not chosen do no work. The catch is that the gate drifts toward a few experts, which then train faster and get picked even more. The authors added extra losses to spread the load evenly.

On a 100-billion-word corpus, the model grew to 65,536 experts and 68 billion parameters. At equal compute, its perplexity, where lower is better, was 39% below the baseline. Going to 131,072 experts made it worse, possibly from too much sparsity.

Left open

Computation shrank, but the parameters still had to sit in memory and move between machines, making bandwidth and batch size hard problems.

• Uses · LSTM

  The MoE layer sits between stacked LSTM layers and is called once per position, possibly choosing different experts each time.

• Adapts · Mixture of experts

  Sparse MoE scales the experts into the thousands and traces the idea to Jacobs et al. 1991.

The paper's examples are plain. A layer is a Python class: the constructor creates the weights, and the forward method multiplies and adds a bias. Because each line runs immediately, print statements, standard debuggers, and matplotlib all work, and intermediate results can be inspected at any point.

Speed comes from what sits underneath. The core library, libtorch, is written in C++ and runs operators and automatic differentiation, and a caching allocator reuses GPU memory. On the six models the authors measured, PyTorch trained within 17% of the fastest framework.

Left open

Running line by line gives up optimizations that need the whole program; the authors were building a JIT to optimize outside Python.

• Complements · NumPy

  PyTorch provides torch.from_numpy() and .numpy(); both sides share one memory region, so conversion copies nothing and takes constant time regardless of array size.

• Complements · Theano

  The PyTorch paper groups Theano with frameworks that build a static dataflow graph, at the cost of ease of use, debugging and flexibility.

Left open

Baidu released only a small demonstration C++ library, meant as a template for framework authors.

• Uses · MPI

  Baidu's ring-allreduce library requires CUDA and MPI.

Ring-allreduce arranges N nodes in a ring, and each node talks only to its two neighbors. In the first half of the steps, received values are added to the local buffer; in the second half, they replace it. With a large enough buffer, this uses the full network bandwidth.

Converting a single-GPU TensorFlow program takes four changes: initialize, give each process one GPU, wrap the existing optimizer, and broadcast the starting variables. In Uber's benchmarks, Inception V3 and ResNet-101 reached 88% scaling efficiency, about twice standard distributed TensorFlow.

Left open

At first it handled only models that fit in one server, and installing MPI on a cluster still took effort.

• Uses · TensorFlow

  Horovod was written for Uber's TensorFlow training, replacing TensorFlow's built-in parameter-server style of distributed training with a wrapped optimizer and a broadcast operation.

• Uses · MPI

  An MPI implementation such as Open MPI launches the copies of the training program and sets up communication among them; the paper's example uses mpirun to start 16 processes on four servers.

• Adapts · Ring-allreduce

  Horovod started from Baidu's draft ring-allreduce implementation, repackaged it as a standalone Python package, swapped in NVIDIA's NCCL and added several GPUs per server.

One example sentence in the paper begins “Jet makers feud”, which becomes “_J et _makers _fe ud”. The less common “Jet” and “feud” split in two, common words stay whole, and the underscore marks the start of a word. Out-of-vocabulary words need no special handling, and sequences stay far shorter than with character-by-character modeling.

Bilingual raters scored translations from 0 to 6. For English to Chinese, the old phrase-based system averaged 4.035 and human translation 4.987. GNMT scored 4.594, closing 58% of the gap between them.

Left open

It still dropped words, mistranslated names and rare terms, and translated each sentence in isolation, without context.

• Uses · LSTM

  Both encoder and decoder are eight-layer LSTMs, with a bidirectional bottom encoder layer.

• Uses · Attention for translation

  The paper states that its attention module is similar to Bahdanau et al.'s, scoring source positions with a one-hidden-layer feed-forward network.

• Adapts · ResNet

  Inspired by ResNet, it adds residual connections between stacked LSTM layers; without them, the authors report, four layers worked, six barely, and more than eight very poorly.

• Complements · IBM translation models

  GNMT names statistical MT (citing Brown et al. 1993) as the dominant paradigm it is measured against.

• Uses · BLEU

  GNMT reports its results in BLEU, alongside human side-by-side ratings.

Candidate generation is posed as a huge classification problem: among millions of videos, which will this user watch to completion next? Embeddings of the last 50 watched videos are averaged, as are recent search tokens, joined with features such as region and device, and passed through several layers. At serving time no probabilities are computed; the system searches for the nearest videos by dot product.

The paper records some hard lessons. If the model knows a user just searched for “taylor swift”, it copies that results page onto the homepage, which performs poorly, so searches enter as an unordered bag of tokens. Ranking by click-through rate promotes clickbait that users click but do not finish, so the ranker weights examples by watch time.

Left open

Watch time is only a proxy; the authors say they rarely get ground truth on whether users are satisfied.

• Adapts · word2vec

  The video embeddings were inspired by continuous bag-of-words language models; the paper cites Mikolov et al.'s companion 2013 paper, Distributed Representations of Words and Phrases, rather than the CBOW paper used by the word2vec node.

• Adapts · Netflix Prize

  Its candidate generator began by imitating a matrix-factorisation recommender and is framed as a non-linear generalisation of factorisation.

A small example: a convolution followed by an element-wise addition. Unfused, the convolution writes its result back to GPU memory and the addition reads it again; fused into one kernel, the intermediate never makes the round trip. On a Titan X the paper measured fused operators running 1.2–2× faster.

Each operator is first written as what to compute, then a schedule decides how: how to tile, which loop order, which cache level holds the data. The choices multiply quickly, so TVM measures candidate code on the target hardware and trains a cost model on those times to predict which version is faster.

Left open

Its speedups were measured on a few devices and inference models, and the cost model must still be trained by measuring on real hardware.

• Uses · LLVM

  In the paper's Figure 2, TVM's low-level loop programs can go to an LLVM IR back end or be emitted as CUDA, Metal, or OpenCL.

• Uses · The Metropolis method

  TVM searches schedules with simulated annealing: moves that lower predicted cost are kept and most that raise it are rejected, the Metropolis rule.

The paper's first example is about precipitation. Asked what causes precipitation to fall, the answer is “gravity”: a system must find the sentence saying it falls under gravity and see that “under” marks a cause, not a location. To keep workers from copying the passage, the interface disabled copy and paste and asked them to use their own words.

Two years later a BERT ensemble passed the leaderboard's human score, but that needs qualification. In 2017 Jia and Liang automatically inserted a sentence that left the answer unchanged and would not mislead people but distracted models. The average F1 of published models dropped from 75% to 36%.

Left open

Every question has an answer in the passage, so a model never has to decide that the text does not say.

Among the first version's baselines, the best average was 60.3, from a BiLSTM with ELMo. The average also hid large differences: many baselines scored near 0 on CoLA but above 80 on SST-2.

Scores rose faster than expected. The first BERT paper reported 80.4 on GLUE, 7.6 points above the previous best. By May 2019 the top score was only 3.3 points below the authors' estimate of human performance, and some of the same authors introduced the harder SuperGLUE.

Left open

Some tasks might yield to superficial tricks, and a plain average does not mean every ability is equally strong.

• Adapts · SQuAD

  GLUE's QNLI task is converted automatically from SQuAD: each question is paired with every sentence of its passage, and the task is to judge whether that sentence contains the answer.

The P100's four links cannot reach seven neighbors, so the eight-GPU DGX-1 split its GPUs into two groups of four, each fully connected by NVLink, with extra NVLink connections between the groups. All GPU-to-GPU traffic went over NVLink, leaving PCIe free for network cards and host memory.

Later generations continued the line. Volta V100 brought second-generation NVLink, with six links and 300 GB/s in total. For Hopper, in 2022, NVIDIA cited 900 GB/s of NVLink bandwidth per GPU in DGX H100.

Left open

NVLink connects only NVIDIA GPUs and supporting CPUs; machines still needed other networks between them.

Liquid cooling changed facility layout first: the TPUs had to sit in several adjacent racks to share the cost of the cooling equipment. Training supercomputers already looked like that. Inference needs to be near users worldwide, and some datacenters were too full. So TPUv4i, the 2020 inference chip, held its thermal design power to 175 W and went back to air cooling.

Pumps used far less power than fans. Pump power per liquid-cooled TPU v3 was less than 5% of the fan power air cooling would need at a comparable thermal design power. Since TPU v3, Google's training systems have used liquid cooling only.

Left open

The heat still has to leave the building, and the price is grouped racks, plumbing, and backup power for cooling.

Left open

It finds approximate neighbours and can occasionally miss the true nearest one.

Left open

It was tried only on Atari games and simulated robots.

Left open

In-context learning leaves the weights alone, and it may stop working on a different task.

• Uses · Scaling laws

  Following this analysis, GPT-3 trained much larger models on far fewer tokens than was typical, and cites it for the batch sizes and learning rates of its larger models.

• Uses · Common Crawl

  GPT-3 drew 60% of its training mix from Common Crawl after quality filtering and deduplication, about 410 billion tokens.

Several results could be used directly. Each time the model grows 8 times, about 5 times more data avoids a penalty from too little data. As compute grows, the best model size grows roughly as compute to the power 0.73, and training steps as only 0.03. Extra compute should mostly buy a larger model.

In 2022, DeepMind's Chinchilla paper reached a different split. After training more than 400 models, its authors concluded that model size and training data should grow in equal proportion. Chinchilla itself, with 70 billion parameters, used the same compute as the 280-billion-parameter Gopher and outperformed it across a large range of downstream tasks.

Left open

These are empirical fits on one dataset, not derived laws, and they predict loss, not downstream ability.

• Uses · Transformer

  The scaling fits are measured on Transformer language models, with a separate comparison against LSTMs.

• Uses · Stochastic approximation

  Chinchilla treats the data-limited part of the loss as a stochastic-approximation term falling at the rate of stochastic first-order methods, citing Robbins and Monro.

Left open

If retrieval misses or fetches the wrong passage, the answer goes wrong with it.

• Complements · Okapi BM25

  RAG's ablations swap its learned dense retriever for a fixed BM25 one; on FEVER, BM25 did best.

• Uses · BERT

  In RAG's retriever, queries and documents are each encoded into vectors by a BERT model.

• Uses · HNSW

  RAG builds a single MIPS index with FAISS and speeds it up with an HNSW approximation.

HumanEval has 164 hand-written problems, each with a function signature, a docstring, and unit tests. They were written by hand because solutions to existing problems were already on GitHub. With one sample per problem, the 12-billion-parameter Codex solved 28.8%.

The paper is direct about failures. The more operations a docstring chains together, the worse Codex does; each extra step cut the pass rate by a factor of roughly two to three. It also mixes up which operation goes with which variable.

Left open

Generated code can look right without doing what was asked, or be insecure, so a person has to review it.

• Adapts · GPT-3

  Codex starts from the GPT-3 model family and is fine-tuned on GitHub code; the paper reports that starting from a pretrained model did not improve final results but converged faster.

Left open

The architecture can be shared, but images and text are still different kinds of data.

• Adapts · Transformer

  Image patches become a Transformer sequence; this does not make image and language data identical.

Left open

The idea of diffusion came earlier; DDPM turned it into the landmark result.

• Complements · GANs

  DDPM lists GANs among earlier generative models and compares its sample quality with ProgressiveGAN.

• Uses · U-Net

  DDPM's denoising network uses a U-Net backbone.

To tell a bicycle from a car, encode the phrases “a photo of a bicycle” and “a photo of a car,” then compare each with the image. No new classifier is needed for those two labels. But rewording the phrases can change the answer, and the model itself gives no explanation.

Left open

Aligned images and text still do not, by themselves, produce grounded answers.

Left open

A predicted structure does not explain all of a protein's function or motion, and experiments are still needed.

Say four GPUs each hold a full copy of the optimizer state. ZeRO can instead give each GPU only part of it. The duplicate copies go away, but data has to be exchanged when computation needs it. Not every part of training memory shrinks to a quarter.

Left open

The memory savings cost extra communication, and it is one of several ways to parallelize.

• Complements · Megatron-LM

  The ZeRO paper combines ZeRO-powered data parallelism with Megatron-LM's model parallelism and uses the Megatron-LM code open-sourced in September 2019 as its baseline.

Take the MLP: multiply by a matrix A, then apply GeLU. Split A by columns, and each GPU computes half the outputs and applies GeLU on its own, with no communication. Split the second matrix by rows to take those outputs directly, then one all-reduce sums the results across GPUs.

In training, the eight GPUs in one server formed a model-parallel group, and 64 such groups ran data parallelism. The full system reached 76% scaling efficiency. The 8.3-billion-parameter model trained for 9.2 days.

Left open

Splitting inside a layer communicates constantly, so it pays off only within one server; across machines, efficiency drops fast.

• Uses · PyTorch

  Model parallelism is implemented by adding a few communication operations in native PyTorch, with communication done through Python calls to NCCL and no new compiler.

• Uses · Volta

  Experiments ran on 32 DGX-2H servers with 512 V100 GPUs in total, using mixed-precision training to take advantage of the V100's Tensor Cores.

• Uses · Transformer

  The split relies on the structure of a Transformer layer: the MLP's two matrix multiplications are split by columns and then by rows, and attention is split by heads.

Left open

A low-rank update can only change so much; tasks far from the original model may need more.

• Adapts · GPT-3

  LoRA is tested chiefly on GPT-3 175B, freezing its weights and training only low-rank updates.

The hard part is that almost everything absorbs EUV light. It must be made and carried in a high vacuum, and lenses are out: multilayer mirrors shrink the mask pattern fourfold onto the wafer. To make the light, a laser hits a passing droplet of tin twice, turning it into a plasma that emits EUV, up to 50,000 times a second.

EUV did not replace all lithography. It prints the most intricate layers of a chip, while the rest still use DUV machines. By the end of 2021, ASML says, 127 of its latest-generation EUV machines were in use at customer sites.

Left open

EUV alone was not enough; chipmakers also had to tune masks, photoresists, and design software around it.

• Complements · Moore, 1965

  ASML presents EUV as what keeps Moore's Law going with ever smaller transistors.

• Complements · TSMC

  TSMC announced N7+ as the first EUV process delivering customer products to market in high volume.

Left open

It still handles general-purpose computing, but its design centers on gaming graphics.

• Adapts · AMD GCN

  RDNA's dual compute unit replaces the GCN compute unit and stays compatible with GCN.

Left open

TF32 covers the same range of values as FP32, but with lower precision.

• Adapts · Volta

  The A100 whitepaper presents its SM, Tensor Core and memory changes as gains over V100.

Left open

MI100 drops rasterization and display hardware; it is not a gaming card.

• Adapts · AMD GCN

  CDNA's compute units derive from GCN, with added Matrix Core engines.

Left open

Multiple-choice scores cannot stand in for testing open-ended generation, tool use, or real deployment.

Using FlauBERT, a BERT model pretrained on French, takes two lines: one loads the tokenizer, one loads the model, both by the same name. From the name, the library finds the right classes and downloads and caches the weights. Swapping the “head” on the same model adapts it to classification or question answering.

It grew out of internal tools and was renamed twice. When the 2019 paper was written, the library was about ten months old and more than 200 papers reported using it. By 2020 its model hub held 2,097 community-uploaded models.

Left open

Reusing models got easier; pretraining from scratch stayed expensive.

• Uses · Transformer

  The library's models are Transformer variants implemented from the original computation graphs and tested against the original authors' reported results.

• Uses · Model Cards

  The 2020 conference version says model pages on the hub can carry a model card (citing Mitchell et al.) covering training, data, and known biases.

The paper's first example is matrix multiplication. The main loop loads a tile of A and a tile of B, takes a dot product, and adds it into a tile of C. Tile sizes are given as a few candidates for the auto-tuner to try. Thread assignment, memory coalescing, and shared-memory synchronization are left to the compiler.

The tests used one NVIDIA GTX 1070. Matrix multiplication matched cuBLAS and ran up to 3× faster than other compiler approaches on recurrent-network and Transformer shapes, though cuBLAS stayed faster on shallow Transformer shapes.

Left open

Kernels got easier to write, but tile size and memory access still matter; the compiler and auto-tuner now handle them.

• Adapts · LLVM

  Triton-IR extends LLVM IR with tile-level data-flow and control-flow constructs, and the compiler emits LLVM bitcode.

• Complements · BLAS

  Triton benchmarks against NVIDIA's cuBLAS: matrix multiplication is on par, and cuBLAS stays faster on shallow Transformers.