Every founder I talk to has the same hiring plan: "we'll get an AI engineer, first hire after the seed." It sounds reasonable and it's usually wrong — not because the person doesn't exist, but because the job description describes the wrong job, and by the time you notice, you've spent the one hire that matters most.
Why the job title lies to you
"AI engineer" sounds like one role. It's at least three:
- Model developer — owns training, fine-tuning, and evaluation of the model itself. Needed when model capability is the bottleneck.
- Systems engineer who works with models — owns the product around the model: prompts, context, retrieval, tooling, contracts, the evaluation harness, deployment, monitoring.
- Applied researcher — explores whether new techniques could move product metrics. A luxury until you have revenue.
At an early-stage company with a product-market-fit problem, role #1 is rarely what you need and role #3 is imaginary. What you actually need is #2 — someone who treats the LLM as a component with a contract, and who becomes more valuable as the model underneath changes, because their work survives model swaps.
The evaluation is the product
Here's the sentence I say to every founder: the evaluation is the product, and the model is the dependency.
The model will be replaced in six months — by a newer one, a cheaper one, an open-source one. The evaluation harness you build is the one thing that outlives every model. It defines what "good" means, catches regressions, and — what most founders miss — it's the internal communication layer. When you can say "this degrades on these inputs, here's the tape," you can have rational conversations about scope, risk, and release. Without it, every conversation is vibes.
Hiring an "AI engineer" out of a big-lab culture often means hiring someone who optimizes the model. That's a genuinely hard skill — just the wrong one for the seat. Your first hire should be the person who builds the harness, because that person makes everything else — domain work, prompt work, product work — measurable.
The hiring order that works
The pattern that works, in my experience:
- First: a founding engineer who owns the product end-to-end and treats the LLM as a component with a contract. They build the abstraction layer — inputs, outputs, cost, failure modes — so nothing outside the boundary ever knows which model is inside.
- Second: a domain expert who writes the evaluation set. The person who knows what "wrong answer" looks like better than either of you — the lawyer, the clinician, the accountant — and can encode that into a golden set of 50–200 cases with justified labels.
- Third — only then — someone whose title actually says AI, once the system has enough surface area that model work is a bottleneck you can feel in the metrics.
A useful heuristic: the first AI hire should be the person whose absence you feel first. At seed stage, that is almost never a model optimizer.
The interview test
Give candidates a broken evaluation: a few golden examples, a model that fails them, and half an hour. Watch what they do.
The wrong candidate starts fine-tuning. The right candidate fixes the harness, quantifies the failure — "three of twelve golden cases fail, all in the same input class" — and tells you the cheapest fix, even if it's a prompt tweak, a doc update, and two new golden cases. The signal you're watching for is judgment about systems, not fluency about transformers.
There's a second, cheaper test you can run on yourself: take your current prototype and ask "if the model I'm using today disappeared tomorrow, what would I lose?" If the answer is "everything," you're not hiring an AI engineer — you're hiring someone to rebuild your product on a dependency you never understood.
The conversation nobody wants to have
Part of why founders hire the "AI engineer" is signaling — to investors, to candidates, to themselves. It's understandable. But the cost is real: the model optimizer is the hardest person to keep productively busy at seed stage, and the title tends to flatline the rest of the engineering function around "we have an AI person, that's covered."
Hire for judgment about systems, not fluency about transformers. The model changes; the judgment compounds. The person you need is the one who is still useful and sharp when the model you're excited about today is a footnote in the release notes.