Language models
Models that process and generate language. Used in communication and content tasks.
A language model is a computational system designed to model the generative likelihood of word sequences to predict the probability of future or missing tokens. The field has progressed through four major stages:
Statistical Language Models (1980s–1990s): Focused on n-gram models that predict the next word using probability and Markov assumptions.
Neural Language Models (2000s–2010s) : Introduced word embeddings and recurrent neural networks (RNNs) like LSTMs to model longer sequences and semantic relationships.
Pre-trained Language Models (PLMs) : Models like BERT and early GPT versions utilized the Transformer architecture for context-aware representations.
Large Language Models (2020–Present) : Characterized by a massive scale in parameters (often >10B), data, and compute, enabling them to serve as general-purpose task solvers.
Technical Foundations
Most modern LLMs are built on the Transformer architecture, which uses self-attention mechanisms to draw global dependencies between inputs and outputs without relying on sequential computation. The success of LLMs is largely driven by scaling laws, which suggest that model performance follows a power-law relationship with respect to model size, dataset size, and total training compute.