AI Project Lead

Who we are and what we do

Contra is building the world’s first professional network for independent creatives and the companies that hire them. We are commission-free, global, and focused on enabling the future of flexible work. Alongside our network and marketplace, we’re launching new product lines including Contra for Companies, Contra Ads Network, and Creative RLHF & Evaluation Services for AI labs and creative-tool companies.

We believe human creativity is the defining force of the next era of AI. That’s why we’re building the Human Creativity Benchmark—a new standard for evaluating how foundation models perform on creative, subjective, and open-ended tasks.

We've raised over $51M from leading investors like NEA, Unusual Ventures, and Cowboy Ventures—and we're just getting started.

Responsibilities:

We are seeking a AI Project Lead to lead Contra’s human data and evaluation business. This is a pivotal role that combines AI evaluation strategy, human data operations, and benchmark design. You will manage and own the quantitative frameworks that define how creative foundation models are evaluated across the industry. You will be responsible for:

• Designing, building, and operating Contra’s Human Creativity Benchmark (our flagship evaluation framework for creative foundation models).
• Scaling Contra’s human-data services - from one-off annotation projects to ongoing evaluation partnerships with top AI labs and creative-tool companies.
• Develop mathematical scoring systems, annotation protocols, rubrics, and evaluation methodologies to measure creativity, usability, factuality, and safety across multiple model outputs.
• Applying probability, statistics, and experimental design (A/B testing, randomized trials, inter-rater reliability studies) to validate benchmark integrity, analyze performance, and ensure reproducibility.
• Use sampling techniques, regression analysis, and variance/error analysis to ensure statistical rigor and representative data coverage.
• Establishing quality control systems to ensure reliability, reproducibility, and credibility of results.
• Lead and mentor a team of analysts, annotators, and domain experts, setting performance goals, managing workflow, and ensuring delivery of high-quality results.
• Collaborating with product and engineering teams to create the annotation and evaluation platform, including task routing, dashboards, and analytics.
• Publishing and evangelizing Contra’s benchmark in the AI ecosystem - positioning Contra as the gold standard for creative AI evaluation.

Requirements:

• Experience: 2-4+ years in data science, human-in-the-loop systems, ML evaluation, or related fields. Background in AI/ML research, creative tools, or human computation is a plus.
• Technical Skills:
  • Proficiency in statistical modeling, hypothesis testing, experimental design, and regression analysis.
  • Familiarity with RLHF, LLM evaluation frameworks, annotation systems, and benchmark design.
  • Hands-on experience with data pipelines, ML ops, or research tooling.
• Creative Orientation: Strong understanding of creative domains (design, writing, media, coding) and how AI intersects with them.
• Execution: Track record of taking ambiguous initiatives from 0 → 1 and scaling them.
• Communication: Exceptional ability to write, present, and advocate - internally and externally.

Total Comp:

• Salary: $175,000-$220,000 USD + Equity
• Medical, Dental, Vision Benefits
• 401k Matching
• We will provide you with a company laptop on your start date

Interview Process

1. Interview with Recruiting Team (20 minutes)
2. Interview with CEO & Co-Founder (30 minutes
3. Interview with VP of Product and CTO & Co-Founder (45 minutes)
4. Paid Case Study and Presentation (60 minutes)

Note: Contra communicates with applicants through @contra.com domains only. We never ask for money from potential employees. For the latest job postings, visit Contra Careers.