Is ‘Attention’ the Essence of Intelligence?

The modern AI revolution traces back to 2014, when 23-year-old intern Dzmitry Bahdanau worked in a University of Montreal lab. His task was to improve English-to-French translation using artificial neural networks, computer systems modeled after biological brains. These networks learn by adjusting connections between nodes based on training data. Bahdanau’s adviser, computer scientist Yoshua Bengio, and postdoc Kyunghyun Cho believed neural networks could solve complex problems like translation. Translation is challenging because languages have different grammar rules, idioms, and connotations, making rule-based automation difficult. Neural networks learn from large datasets of paired sentences to absorb linguistic patterns.

Bahdanau solved the translation bottleneck by inventing a mathematical mechanism called attention. This allowed neural networks to handle long sentences as effectively as short ones, making the approach scalable. Attention works by enabling the network to identify which words in a sentence are most relevant for determining the next word in a translation. For example, in the sentence 'The animal didn’t cross the street because it was too tired,' attention helps the system determine whether 'it' refers to the animal or the street. The mechanism was so effective that other researchers quickly adopted it, leading to widespread integration into neural network designs.

In 2017, a Google research team developed a new neural network architecture called a transformer, which built upon attention. Their paper, titled 'Attention Is All You Need,' introduced a system that achieved state-of-the-art translation while requiring significantly less computation than prior models. The transformer’s architecture consists of layers of nodes that process words as numerical representations called vectors. Each word is converted into three vectors: a query, a key, and a value. The query vector requests information from other words, while the key vector advertises relevant features. The value vector contributes to the updated representation of the word based on how well the query and key match.

In a transformer, the encoder processes the input sentence by using attention to build a contextual understanding of each word. For example, the word 'it' in a sentence generates a query vector that seeks information from earlier nouns. If the query for 'it' is interpreted as 'I need an earlier noun that could plausibly be tired,' the key for 'animal' might match better than the key for 'street.' This causes the value associated with 'animal' to contribute more to the updated representation of 'it.' Successive layers refine these representations, enabling the network to capture relationships like noun-verb partnerships or associations between living beings and feelings.

The ability to translate sentences effectively is a subtask of the broader task of predicting the next word in any context. This is the core function of large language models (LLMs) like ChatGPT and Claude. These models are trained on vast amounts of text from the internet, learning statistical patterns between words. During training, the model makes trillions of adjustments to its connections, or weights, to improve its predictions. The result is a system that can generate fluent text, answer questions, and even perform tasks like arithmetic or fact retrieval, all by predicting the next word in a sequence.

Llion Jones, a co-author of the 'Attention Is All You Need' paper and now chief technology officer of Sakana AI, suggests that predicting the next word may be an AI-complete task. This means solving it could be equivalent to achieving general artificial intelligence, as it encompasses many human-like abilities. For example, if given the prompt '2 + 3 is [blank],' the model must perform arithmetic. If given 'The president of the United States is [blank],' it must retrieve factual information. The internet contains examples of translations, arithmetic, and facts, so the model learns to perform these tasks by predicting the next word in such contexts.

While attention enables AI models to process language and build context, human intelligence is not purely linguistic. Cognitive scientists note that human thought involves multiple brain regions beyond those associated with language, including sensory impressions, memories, and reasoning abilities. This suggests that language modeling is a powerful component of human cognition but not its sole source. Additionally, the human brain achieves language mastery far more efficiently than transformers, which require massive computational resources and energy. Jones argues that the current AI paradigm may not solve this efficiency problem without a fundamental shift in approach.