AI, Decoded by Cavemen, Scrabble Tiles, and a Poker Dealer

The goal of this article is to break down generative AI into simple, understandable terms without all the technical jargon. After spending years trying to decode the process, this is what I have learned. My goal is to use simple analogies to walk you through the complexity in this space by focusing on why most people use generative AI. For most people, they want to type something into the AI mode in Google and get a well-informed answer. 

* All technical jargon will be in parentheses and I will dive into the technical mechanics in future articles. 

Putting It All Together

Here are the steps that I will walk through to examine the entire process step by step.

1.      Every AI model starts with a giant library of information pulled from the internet.

2.      That library gets organized by topic, the same way similar books end up shelved near each other (vector database).

3.      Language models also break every word into smaller pieces, like letters on a Scrabble board (tokens).

4.      Those pieces start out scrambled at random, then the model rearranges them over and over until real patterns emerge (training).

5.      When you ask a question, it gets sorted into that same library system so the model can find the closest-matching book (embedding).

6.      That information gets handed back and added to your question, like a librarian bringing you the right book.

7.      Then, like a poker dealer reading the table, the model picks the best next word one at a time, paying closest attention to whatever matters most for your question (self-attention).

8.      And the longer you chat, the more clues it gathers, like playing Guess Who, narrowing in on smarter answers as the conversation goes.

Two Models, Working Together

To understand this better, let's explore two types of models. Think of these two models as the brain (embedding) and the mouth (Generative Pre-trained Transformer). The brain (embedding) is the entire collection of information from the internet, broken into pieces (chunks) and organized. Think of this as how a library sorts thousands of books into logical categories where similar books are stored near each other. The mouth (GPT), on the other hand, is the person who goes to the library with a question. The brain (embedding) takes the question and gets information from a book.  It sends it back to the mouth (GPT) and then the mouth translates that into a spoken message. Now, assume you are a college student writing a paper. You try to choose the most relevant information for your paper. However, if the library only has outdated books or you choose the wrong source, you could confidently provide inaccurate information (hallucinate). Just like the student is limited to what is at the library, so is the model.

When you type a question into a chat and get a response, these two models work together. One model (the embedding model) finds the most relevant piece of existing information to answer your question. The other model (GPT) takes that information and writes the actual answer, word by word.

To understand how they work together, it helps to first understand how one model (the embedding model) searches for information, and how the other model (GPT) takes that information and uses it to predict words, one at a time, into a meaningful response.

Setting the Scene: A Caveman's Internet

Let's break this into smaller pieces. Assume we are cavemen who only understand the eight sentences below. This is an extremely simplified example, meant to represent the idea of pulling all possible content from the internet.

1. We hunt animals for food today.

2. We need fire for heat.

3. Run fast from danger.

4. Fire keeps us warm tonight.

5. That animal is dangerous.

6. We share food with friends.

7. We made a new tool for hunting.

8. Friends help us hunt for food.

Finding the Right Information: The Embedding Model

The brain (embedding model) takes all of the knowledge available (in a real system, this would be content pulled from across the internet) and breaks it into smaller pieces of information (chunks). Each piece is converted into a list of numbers that captures its meaning (vector embedding) and stored for later (in a vector database).   

When you send a question, that same model (embedding model) converts your words into a combination of numbers (vector embedding- [-0.1362, 0.4893, 0.2519, 0.1071, -0.3734, -0.3735, -0.4798, 0.3975]) as well, then compares it against every stored chunk. The computer searches through all the information (chunks) and compares these numbers (vector embeddings) to each other (using a method called cosine similarity) to find the closest match. Since an exact match is never found, the model returns the chunk with the highest similarity score. That chunk is then returned to the other model (GPT).

Notice how the caveman’s knowledge is organized on a plot chart.  Topics that are similar are grouped together. A real embedding model does this same sorting, just across hundreds or thousands of categories instead of the simplified two-dimensional version shown here.

Building the Language Model

To build the chat model (GPT), the first step is to break all of these sentences into individual words. However, this process actually goes a bit deeper and breaks the words into word pieces (tokens). Longer or less common words often split into more than one token. For example, "hunting" becomes "hun" and "ting" and dangerous becomes “danger” and “ous” rather than staying as one word. Once every sentence has been broken into these smaller pieces (tokens), the model builds a full vocabulary out of all of them.  These 29 words from the caveman’s internet convert into 34 tokens to build the vocabulary.  In the real world, a small language model would have a vocabulary of around 50,000 tokens, and a large one can run well beyond that.

As the model works through the sentences one word piece (token) at a time, it separates the data into two related groups (tensors). The first group holds the word pieces (tokens) seen so far, and the second holds the word piece (token) that comes right after them. This is shown below using words for clarity, but the computer actually represents each token as a number. See the example below:

"We"   --------------------------------> " made"

"made" --------------------------------> " a"

"a"    --------------------------------> " new"

"new"  --------------------------------> " tool"

"tool" --------------------------------> " for"

"for"  --------------------------------> " hun"

"hun" --------------------------------> "ting"

These pairs are then converted into two matching sets (tensors). The left side becomes one set, and the right side becomes the other. Now imagine this same process repeated across millions of words. Over time, the model starts to see how certain words tend to line up with the words that follow them.

["We", "made", "a", "new", "tool", "for", "hun"]

["made", "a", "new", "tool", "for", "hun", "ting"]

Lining up all of these words correctly takes a lot of work. The model is trying to figure out the relationship between the first set and the second set, out of billions of possible options. At first, the computer does not understand any of these relationships, so it starts with random guesses.

Imagine this like receiving seven letters on your Scrabble board. You place the letters in a random order at first (the initial layer). Then you start sorting the letters and find a few two- or three-letter words (a hidden layer). As you keep going, you find better and better patterns that improve your score (these adjustments are the weights and biases). Finally, you land on the best possible word (the final layer).

• Initial layer — This is the random set of letters you receive: HCASFIT

• Hidden layer — Sorting them the first time gets you AS, FIT, HC

• Hidden layer — Sorting them again gets you CHAT, SIF

• Hidden layer — Sorting them again gets you CAT and FISH

• Final layer — Finally, you get CATFISH

  
  

Since the first layer starts out completely random, the model keeps adjusting it, one small step at a time. This process is really about moving related words (or, in the Scrabble example, related letters) closer and closer together. You can picture this as organizing words on a two-dimensional graph: as the model finds real relationships between words, it pulls similar words closer together on that graph.

Now that you understand how the model makes multiple attempts to improve the results, it is important to understand how it understands actual context. In the example below, food, heat, and hunting can all come after the word "for":

"We hunt animals for food today."

"We need fire for heat."

"We made a new tool for hunting."

What if the cavemen provided this as the prompt: "Why do we need tools?" The brain (embedding model) would have to retrieve this answer since it is the only sentence in the model that matches the word tools:

We made a new tool for hunting.

It then adds this to the original question to form something like this:

Why do we need tools? We made a new tool for hunting.

Now the mouth (GPT) has to take the above information and create an intelligent answer. Say it began with "We need tools" because that came from the sentence. According to the primitive vocabulary of the caveman, the next word must be "for." The last word is where the context comes in. Since "food," "heat," and "hunting" can all come after "for," which one should the model choose? Knowing that the word "hunting" is in the retrieved sentence helps guide the model to choose that as the next word.

Now imagine that you are a poker dealer who can see everyone's hand, every card dealt, and everyone's betting tendencies. Their vocabulary is the 52-card deck. Since the dealer knows all the cards that are dealt, he knows that player 2 has a 4% chance of a royal flush since the ace is available and there are 28 more cards left in the deck.  However, player 3 has a 32% (9/28) chance of getting a pair for his next draws.  In other words, the dealer pays far more attention to the cards that best enhance his hand, and calculates the odds accordingly (self-attention). The model (GPT) also has a list of the highest probability words (top k) to come next. It may then choose randomly one word from that short list.

Learning from You

Now that you understand the basic mechanics, let’s take it one step further.  The final step is understanding how the model (GPT) temporarily stores memory.  As you chat with the model it starts to get smarter and smarter.  This works because the model stores your response and the answer.  As you ask more questions, it stores more questions and answers, giving it more context.  Think of the game “Guess Who”.  You may first ask if the person is a female.  Then if they have brown hair.  Then if they have green eyes.  If the answers are all yes, then you know that the person is a female with brown hair and green eyes.  This eliminates most choices giving you a higher probability of guessing the correct person.  The model does the same thing. By gathering more context, it starts to find more patterns.

The Final Response

This topic is very complex and you can never understand it all but the basic principles are still rooted in traditional data science algorithms.  Since all the words are translated to number, everything becomes a math problem where you are trying to find the closest correlation.

If this article helped your understanding or you have feedback to improve this, let me know.  I will continue to write pieces breaking down this complex subject.