Deep learning uses neural networks with many layers. Understanding a few parts of it explains how chat assistants work.
Neurons, layers and training
A neuron takes numbers in, multiplies them by weights, adds them up and passes the result through an activation function that decides how strongly to fire. Layers of neurons are stacked so each layer builds on the one before. During training, backpropagation works out how much each weight contributed to the error, and an optimizer (such as Adam) updates the weights to reduce it.
Embeddings
An embedding is a vector that represents the meaning of a word, sentence or document. Items with similar meaning get vectors that are close together. This is the bridge between language and maths.
Attention and the transformer
Older models read text one word at a time and forgot earlier parts. The attention mechanism lets a model look at every word in the input and decide which ones matter most for the word it is processing now. In "The nurse told the patient she was late", attention helps work out who "she" refers to.
The transformer is the architecture built on attention. It is the basis of almost every modern language model.
- Encoder-only models (like BERT) are good at understanding text: classification and search.
- Decoder-only models (like Claude and GPT) generate text one token at a time.
- Encoder-decoder models translate or transform one sequence into another.
Pretraining and fine-tuning
A model is first pretrained on a huge amount of text so it learns language in general. It can then be fine-tuned on a smaller, specific dataset or trained to follow instructions and be helpful. Most products start from an existing pretrained model rather than training one from scratch.