AI Engineer | Agent Evals & Model Quality
HartleyCoSan Francisco, CAposted 2d ago
- PayUSD / yr
- $300k – 350k
- Where
- San Francisco, CA
- Posted
- Oct 16
- 2 days ago
- Type
- Full-time
- Setup
- On-site
EMPLOYMENT
Full-time
COMPENSATION
$300k - 350k/yr
ARRANGEMENT
On-site
At a glance
Build the evaluation systems that determine whether production AI agents are ready to ship, from benchmarks and LLM judges to failure-driven feedback loops.
Summarized by AI from the original posting
What you'll do
- Build evaluation frameworks for multi-step agents and tool use
- Create golden datasets, benchmarks, and regression suites
- Develop LLM judges, deterministic graders, and human evaluation workflows
- Compare models across quality, latency, reliability, and cost
- Build feedback loops that improve agents based on production failures
Requirements
- 01Production AI/ML systems experience
- 02Strong Python programming skills
- 03Ability to explain personally designed, shipped, and improved systems
- 04Substantial technical ownership
Skills
PythonLLM JudgesDeterministic GradersMulti Step AgentsTool UseGolden DatasetsBenchmarksRegression SuitesHuman Evaluation WorkflowsModel ComparisonFeedback LoopsAi/ml Systems
Full description
AI Engineer | Agent Evals & Model Quality
Own how a production AI company decides whether its agents are good enough to ship.
I’m partnering with a technical, founder-led team hiring an AI Engineer to build the evaluation systems behind its platform.
You’ll own:
• Evaluation frameworks for multi-step agents and tool use.
• Golden datasets, benchmarks and regression suites.
• LLM judges, deterministic graders and human evaluation workflows.
• Model comparisons across quality, latency, reliability and cost.
• Feedback loops that turn production failures into better agents.
The team wants someone who has built production AI/ML systems, writes strong Python and can explain what they personally designed, shipped and improved.
Around 5+ years of software engineering experience is the guideline. Exceptional relevant ability and ownership matter more than a rigid experience threshold.
This should appeal if you want substantial technical responsibility and a direct say in how the product develops.
Message me with your resume and a brief example of an evaluation or production AI system you’ve owned.