RAG Models Explained Without the Black Box

A hands-on walkthrough of embeddings and similarity search using FIFA player data.

Most of us who have played around with generative AI are familiar with building a RAG (or Retrieval Augmented Generation) model. In essence, you import your source documents to create a corpus, chunk the text into smaller pieces, and send those chunks into an embedding model to generate numerical vectors.  From there, your query searches the vector database for relevant chunks, which are then passed alongside your question to the GPT model as context. The process is relatively simple. After building a few, you can set up a basic proof of concept in under a day.

However, once you start testing the results for accuracy, the whole project changes. Hallucinations and half-truths are all over the place. AI engineers start adjusting prompt templates, swapping embedding models, and devising better chunking strategies.  Since the embedding model is a black box, you can't just resolve the issue by updating a line of code or two.

So how do these embedding models work in theory? This article helps demystify the process using a very simple FIFA dataset that anyone can easily download from Kaggle. I will break down the vector into pieces and adjust for better recommendations.

To start, you will need to log into Kaggle and create a token. Then run the below code in a Jupyter or Google Colab notebook. It will install the Kaggle library, import the needed libraries, and convert the dataset into a data frame.

# Install library

!pip install -q kaggle 

# Import Libraries

from kaggle.api.kaggle_api_extended import KaggleApi

import os

import zipfile

import pandas as pd

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

# Set environment variables

os.environ['KAGGLE_USERNAME'] = "your_user_name"

os.environ['KAGGLE_KEY'] = "your_kaggle_token"

# Authenticate and download

api = KaggleApi()

api.authenticate()

DATASET = "stefanoleone992/fifa-21-complete-player-dataset"

DATA_FOLDER = "fifa_21_data"

FILE_NAME = "players_21.csv"

api.dataset_download_files(DATASET, path=".", unzip=False)

# Read zipped file

with zipfile.ZipFile("fifa-21-complete-player-dataset.zip", 'r') as zip_ref:

    zip_ref.extractall("fifa_21_data")

# Connect to file

file_path = os.path.join(DATA_FOLDER, FILE_NAME)

# Create the DataFrame

df = pd.read_csv(file_path)

display(df)

Below is a simple data dictionary to help better explain this dataset.

The next step is to clean up the data. Rarely will you ever get data that is clean. It’s the reason why a solid data engineer is worth their weight in gold. For this data frame, we need to remove the goalkeepers since they have NaN values for some positions. Next, the position ratings are stored in a string format and will need to be converted to a float data type. Then we will change the weak_foot and skill_moves columns from a 1 to 5 scale to a 0 to 1 scale. Finally, normalize the age, weight, and height into the 0 to 1 scale using preprocessing in the sklearn library.  The below code will resolve these issues. If you have done this correctly, you should see the below results.

# Convert these columns from strings to floats

position_cols = ['lwb', 'ldm', 'cdm', 'rdm', 'rwb', 'lb', 'lcb', 'cb', 'rcb', 'rb']

# Parse ratings that include a chemistry boost (e.g. "78+2" becomes 80)

def parse_rating(x):

    x = str(x).strip()

    if x in ('', 'nan', 'NaN', 'None'):

        return None

    if '+' in x:

        parts = x.split('+')

        try:

            return (int(parts[0]) + int(parts[1])) / 100

        except (ValueError, IndexError):

            return None

    try:

        return float(x) / 100

    except:

        return None

# Remove goalkeepers — they have NaN values for outfield position ratings

df = df[df['player_positions'] != 'GK'].copy()

# Convert strings to floats

for col in position_cols:

    df[col] = df[col].apply(parse_rating)

# Convert weak_foot and skill_moves from 1 to 5 to a value between 0.2 and 1

df[['weak_foot', 'skill_moves']] = df[['weak_foot', 'skill_moves']] / 5

# Normalize age, height_cm, and weight_kg to a 0 to 1 scale

df[['age', 'height_cm', 'weight_kg']] = scaler.fit_transform(df[['age', 'height_cm', 'weight_kg']])

display(df)

Now that we have a cleaned data frame, we will use our sofifa_id as our unique identifier and use the below columns as our matrix of features. Since our above conversion makes everything a value between 0 and 1, it is now time to create a vector with all of the below columns.

['lwb', 'ldm', 'cdm', 'rdm', 'rwb', 'lb', 'lcb', 'cb', 'rcb', 'rb', 'weak_foot', 'skill_moves', 'age', 'height_cm', 'weight_kg']

Run the below code to see the vector embedding.

vector_cols = ['lwb', 'ldm', 'cdm', 'rdm', 'rwb', 'lb', 'lcb', 'cb', 'rcb', 'rb', 'weak_foot', 'skill_moves', 'age', 'height_cm', 'weight_kg']

# Create a vector database

sofifa_id_list = df['sofifa_id']

vector_df = pd.DataFrame(sofifa_id_list, columns=['sofifa_id'])

vector_df['position_vector'] = df[vector_cols].values.tolist()

display(vector_df)

Since we passed in 15 columns, there are 15 values in each array. If this had been the OpenAI text-embedding-3-small model, we would have had 1,536 values. This illustration is simplified to allow you to understand and play around with the values you put into the model.

Now run the below code. This code picks Argentina superstar Messi and tries to find players that are similar. Messi's sofifa_id is 158023. We look this up and create the messi_vector. The positions_df is used to compare to the messi_vector. Then we use the corrwith function to find players with similar traits. The final results are the top 10 players with correlation scores similar to Messi.

# Get Messi's position vector as a Series

messi_vector = pd.Series(

    vector_df[vector_df['sofifa_id'] == 158023]['position_vector'].values[0],

    index=vector_cols

)

# Expand position vectors into a DataFrame with sofifa_id as index

positions_df = pd.DataFrame(

    vector_df['position_vector'].tolist(),

    columns=vector_cols,

    index=vector_df['sofifa_id']

)

# Transpose so players are columns, then correlate with Messi's vector

correlations = positions_df.T.corrwith(messi_vector)

# Get top 10 most similar players excluding Messi himself

top_10 = correlations.drop(158023).nlargest(10).reset_index()

top_10.columns = ['sofifa_id', 'correlation']

# Join back to get player names

top_10 = top_10.merge(df[['sofifa_id', 'short_name', 'overall']], on='sofifa_id')

display(top_10)

When you go back and compare the individual embeddings to each other, here is how Messi and Gomez compare. You can see how strikingly similar most of the values are. This illustration shows the same core mechanic behind RAG: taking one vector and finding the closest matching vector in the database. In a real system, that starting vector would come from embedding the user’s prompt rather than from another stored player like Messi, but the underlying search works the same way.

Notice that age, height_cm, and weight_kg are included in vector_cols. Try removing them and running the results again. This is where you get to adjust the model yourself. A good data scientist keeps only the features that are closely correlated with what they are measuring. Here, physical attributes turn out to be weaker predictors of positional performance than the position ratings themselves.

To break it down further, an embedding model converts your prompt into a vector, then compares it against every stored vector in the database using cosine similarity to find the closest match. It’s an overly simplified example, but the overall concept applies. That closest match is then passed to the GPT transformer as context, which generates an answer that reads naturally.

The corrwith function works similarly to cosine similarity. Both measure whether vectors point in the same direction but differ only in that Pearson correlation (corrwith) centers each vector around its mean first. They give comparable results here, but production models use cosine similarity since it is a simpler calculation and scales better across the hundreds or thousands of dimensions real embeddings use. Also, if the vector database had billions of rows, a cosine similarity search would force a complete table scan. In the real world, approximate nearest neighbors (ANN) are used like indexes in a traditional SQL database, narrowing the search to a group of closely related vectors instead of scanning everything. Both the GPT transformer step and how ANN indexing actually works will be covered in a future article. Follow me to get notified when it is out.