Your AI Doesn’t Need a Better Prompt. It Needs a Harness.

23 hours ago

7 min read

In 2023, my consulting team was asked to build courseware using AI for a client. The work meant reading long legal documents, and legal experts charged $400 an hour to help. AI seemed like the perfect solution. We started by asking an AI model to explain SEC regulations and write simple multiple-choice questions. On the common topics, it did fine. On the more nuanced ones, it either skipped them or made things up. Either way, it sounded confident, even when it was completely wrong. That was our first problem.

We didn’t fix that with a better prompt. We fixed it by building a system around the model that fed it the right documents, remembered past answers and checked its work. It took us months.

Today that system is called a harness, and many of the pieces we built by hand now come ready-made.

In this article, I’ll show how we got from prompt engineering to harness engineering, what a harness looks like in code, and how its five parts turn a six-word request into a complete, checked answer.

My first AI project

Our source material was the SEC’s rules, published in the Electronic Code of Federal Regulations at ecfr.gov.

Working with AI back then was unreliable. Models only knew what was in their training data, which usually ended a year or two earlier. They couldn’t see recent rule changes. Many models also had small context windows. The original GPT-3.5 accepted only about 4,000 tokens, or roughly 3,000 words. This meant that every prompt had to be lean.

To stop the made-up answers, we built our own context. We scraped the regulations from ecfr.gov and used LangChain to split them into chunks of about 2,000 tokens. Then we ran those chunks through an embedding model to build a vector database. Now the model could look up the relevant regulation before answering. This approach is called retrieval-augmented generation, or RAG, and it improved our results.

From there, we stored each user’s conversation in a database, which gave us a basic form of memory. Once the answers were decent, we saved every prompt and response and had a second model grade them for accuracy.

It worked, but it was slow and fragile, and we built every piece ourselves. Looking back, we were building a harness before the word existed.

How we got here

Each stage built on the one before it.

Prompt engineering (2022–2023). After ChatGPT launched in late 2022, the focus was on wording the prompt with a role, clear instructions and examples.

Retrieval and chaining (2023–2024). Teams like mine added RAG to give models outside knowledge, and frameworks such as LangChain made it possible to chain prompts together and keep simple memory.

Context engineering (2025). In June 2025, Shopify CEO Tobi Lütke and AI researcher Andrej Karpathy popularized the term context engineering, which means deciding everything a model sees, beyond just the prompt.

Harness engineering (2026). As AI agents began running tools and multi-step tasks on their own, the focus moved to the system around the model. In February 2026, HashiCorp co-founder Mitchell Hashimoto wrote that he had “grown to calling this ‘harness engineering.’” Six days later, OpenAI published its own post on harness engineering, and LangChain soon adopted the term too. The harness is the loop that calls the model, runs its tools, manages its memory, checks its output and enforces its limits.

A Cubs pitcher in one minute

Think of the AI model as a Cubs pitcher. He’s talented, but he doesn’t win games alone. Here’s how each layer improves the simplest possible instruction.

START Pitch the ball.

+ PROMPT You're the Cubs' starter. Throw a four-seam

fastball, low and away. Stay off the middle.

+ CONTEXT Top of the 7th, 2 outs, runner on 1st, 1-2 count.

Batter hits .275 on fastballs up in the zone.

Wind blowing in off Lake Michigan at 10 mph.

+ HARNESS Tools for scouting reports, the bullpen phone, replay.

Update the situation after every pitch.

Pulled at 100 pitches. Stop at 3 outs.

Prompt engineering is the catcher’s signs. Context engineering is the scouting report and the game situation. Harness engineering is the manager and the dugout, with the tools, the pitch-by-pitch loop and the limits that the system enforces rather than simply requests. Three words became a plan for a full inning.

The five parts of a harness

Every harness, simple or complex, is built from the same five parts. Keep these in mind, because each one appears as a comment in the code that follows.

1. Context management decides what the model sees right now.

2. Memory (state persistence) keeps a record that lasts beyond a single turn.

3. Tool integration and execution lets the model call real tools and APIs, then returns the results.

4. Permissions and guardrails (sandboxing) limit what the model can access and do, with rules enforced by code.

5. Verification loops (sensors) check the output before anyone sees it.

What a harness looks like in code

At its core, a harness is a loop. Here’s a simplified version in Python, with each of the five parts labeled.

def ask(messages, question, max_steps=6):

messages.append({"role": "user", "content": question})

for in range(maxsteps): # guardrail (hard stop)

reply = call_model(messages, tools=TOOLS) # context goes in

messages.append(reply) # memory (this conversation)

if reply.tool_calls: # tools

for call in reply.tool_calls:

if call.name in TOOLS: # permissions

result = run_tool(call)

else:

result = "Not allowed"

messages.append(tool_result(call, result))

continue

problems = verify(reply.content) # verification

if not problems:

return reply.content

messages.append({"role": "user",

"content": f"Fix these problems. {problems}"})

return "Stopped at the step limit."

The helper functions (call_model, run_tool, tool_result and verify) depend on which model and tools you use. You don’t have to write this loop yourself, either. Frameworks such as LangGraph and the OpenAI Agents SDK handle it for you, so your work shifts to choosing the context, tools, rules and checks.

The memory in this loop is short-term and covers only the current conversation. Long-term memory, like Big Joe’s usual order (in the example below), usually lives in a database that the harness searches before calling the model.

A note on terms. “Harness” is a newer word, and you’ll often see the same idea under other names. If you want to find tools to build one, these are the searches to try.

• AI agent frameworks, such as LangGraph, CrewAI and Semantic Kernel. They give you the loop, memory, tool calling and guardrails as building blocks.

• Agent SDKs, such as the OpenAI Agents SDK and the Claude Agent SDK. These are model makers’ own toolkits for the same job.

• Orchestration layers, the general term for the software that sits between your app and the model and coordinates prompts, tools and data.

The terms overlap but aren’t identical. A framework is the toolkit, and the harness is what you build with it by combining your model with your choice of context, memory, tools, rules and checks.

Five steps, one order

Here’s each part at work in a Chicago pizza kitchen, starting with a six-word prompt.

Big Joe wants to order a pizza.

Memory looks up Big Joe, a regular, and finds his usual order.

CUSTOMER Big Joe · regular since 2019

USUAL large deep-dish, Italian sausage and onion,

extra-crispy crust

Context adds the recipe for that one pizza, and nothing about the rest of the menu. Chicago style puts the cheese on the dough and the sauce on top.

1. Press the dough into an oiled 14-inch pan and up the sides.

2. Mozzarella on the dough first.

3. Raw Italian sausage in one layer, then sliced onion.

4. Crushed-tomato sauce on top.

5. Grated Parmesan.

Tools supply what the model can’t know on its own, so it calls the kitchen’s oven system for the bake settings.

get_bake_settings("large deep-dish", "sausage, onion", "extra crispy")

→ 425°F · 38 minutes · turn the pan at 20 minutes

Permissions and guardrails limit where the recipe can come from and what the AI can touch. That keeps a Detroit-style or New York–style recipe from slipping in.

ALLOWED Recipes from the kitchen's private site (read-only login)

NOT ALLOWED Changing recipes · Big Joe's payment details ·

Cooking steps from public websites

Verification checks the result against three other deep-dish recipes before the order goes out. If a check fails, the system fixes the problem and checks again, up to the step limit. One common way to build this is to have a second model grade the first, an approach called LLM-as-a-judge.

✓ 425°F other recipes use 400–450°F

✓ 38 minutes other recipes bake 30–40 minutes

✓ Layer order matches all three

! MISSING Check that the sausage reaches 160°F inside

→ FIX ADDED 7. Check the sausage has reached 160°F before slicing.

Here’s the finished prompt. Six words have become a complete, checked set of instructions.

Big Joe wants to order a pizza.

ORDER Big Joe's usual. Large deep-dish, Italian sausage

and onion, extra-crispy crust.

RECIPE House Deep-Dish · Sausage & Onion, version 7,

from the kitchen's private recipe site

1. Press the dough into an oiled 14-inch pan and up the sides.

2. Mozzarella on the dough first.

3. Raw Italian sausage in one layer, then sliced onion.

4. Crushed-tomato sauce on top.

5. Grated Parmesan.

6. Bake at 425°F for 38 minutes, turning the pan at 20.

Rest 5 minutes.

7. Check that the sausage has reached 160°F before slicing.

CHECKED Against 3 other deep-dish recipes · approved

The real skill is organization

The model was never the whole story. Our 2023 project struggled at first not because the model was weak, but because nothing around it fed it the right facts, remembered what happened or checked its work.

Today, the tools to build that system are far easier to use. You no longer need to spend late nights reading the LangChain documentation to get reliable results.

What you do need is organization. Look back at the pizza example. Someone had to decide that the private recipe site is the only trusted source, that payment details are off-limits, and that every order needs a temperature check. None of that is code. It’s knowing what the user wants, which sources to trust and what “correct” means. Those are the core skills of business analysts and project managers, and they’re quickly becoming some of the most valuable skills on any AI team.

So the next time an AI gives you a confident, wrong answer, don’t just rewrite the prompt. Ask what your harness is missing.

What does your harness look like? Share what’s working for you in the comments.