A language model has one job, given the current context guess what word comes next. That is the core feature behind ChatGPT, and it is also the trick behind the autocomplete strip on your phone’s keyboard. To show how it works, I built the small one below. It learned everything it knows from just one book, Alice in Wonderland, and you can write with it by tapping/clicking on the suggestion chips.
The most important feature here is the memory slider. It controls how many previous words the model looks at when guessing the next one. At 0 it looks at nothing and just picks whatever words are most common. At 1 it only looks at the last word, which is close to what your phone’s autocomplete does. At 2 it looks at the last two words, and with just that it starts to sound like the book it learned from.
Set the slider to 10 and the suggestions stop coming. This model guesses by searching the book for the exact words you just wrote and looking at what came right after them. At 10 words the search is too specific. Whatever you wrote has almost certainly never appeared in the book, so the search comes back with nothing.
ChatGPT does not search stored text at all. Every time it guesses a word, the entire conversation so far is fed into a function as input, and the function outputs a probability for every possible next word. That is the same kind of function from part 1, and its weights were found with gradient descent from part 2. A function does not need to have seen your exact sentence before to give an answer for it. That ability is called generalization, and it is the reason neural networks replaced lookup tables.