About the company
Our client is an AI technology company.
The role
Raydar is recruiting for this role on behalf of our client. Own the measurement of model quality from start to finish, defining metrics, building the tooling to compute them and turning the results into concrete improvements. The work centers on evaluating complex multi-component machine learning systems where no ground truth exists. You will carry projects from the initial question through to a written conclusion without handoffs.
What you'll do
- Define evaluation metrics and what improvement means for outputs that evolve over time, keeping those definitions current as methods change.
- Build data pipelines and test harnesses covering data ingestion, labeling, versioning, reruns and automated judging.
- Analyze run traces and results to find real failure modes rather than only what is easy to measure, and propose fixes.
- Develop simulation tooling that lets many scenarios be modeled and measured in parallel to speed up iteration.
- Own each evaluation loop end to end, from the question to the pipeline, rerun and final writeup.
Requirements
What we're looking for
- Experience designing, building and running evaluations for large language models from start to finish.
- Prior work in areas such as memory, personalization, agent evaluation or agent optimization.
- Published research as a first or co-author at a top machine learning venue, or open-source work of comparable quality.
- A university-level background in a quantitative discipline such as computer science or statistics.
- Production-quality Python, including testing, maintenance and regular releases.
- Working knowledge of PyTorch and common machine learning tooling and experiment tracking.
Bonus points
- Experience with data pipeline tooling such as Kafka and SQL.
Benefits
Compensation and benefits
- Base salary: USD 220,000 to 300,000 per year
- Equity
- Health, dental and vision coverage
- Retirement plan
- Paid time off
- Relocation support
Location and work model
- New York, NY, United States
- On-site, 5 days per week in office
- Full-time
Similar Jobs
What you need to know about the NYC Tech Scene
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
