AI Quality Associate

Please note: This position is not eligible for immigration sponsorship. 

We are the better way to work in finance. As private equity’s value creation partner, we sit at the heart of PE—where sponsors and CFOs meet. Through financial consulting rooted in data, technology, and AI, we help clients drive value—supporting the office of the CFO to drive end-to-end value creation.

If you crave challenging work and are looking to grow, come solve complex issues alongside 1,600+ finance & technology experts in a supportive, collaborative environment.

Backed by premier private equity firms and headquartered in New York with 11 offices around the globe, we are a high-growth, entrepreneurial firm looking for people who want to be part of building something great. Come make your mark.

The AI Quality Associate focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. This impactful role will be critical in building benchmark scenarios, defining scoring rubrics, evaluating business usefulness, and identifying failure patterns that conventional pass or fail software testing would not catch.

This position will ideally be based in one of our US offices and is a hybrid role with the flexibility to work remotely 2 days per week (also eligible for remote candidates).

This position is not eligible for immigration sponsorship.

AI Output Evaluation

• Design and execute structured evaluations for AI-enabled features and workflows
• Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk
• Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations

Benchmark & Rubric Development

• Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards
• Develop rubrics that allow quality to be measured consistently across releases and changes

Workflow & Model Change Validation

• Compare performance across prompt versions, workflow revisions, tools, and models
• Support release decisions with evidence on quality regression or improvement

Business & Domain Partnership

• Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts
• Help define human-review thresholds and escalation patterns for higher-risk use cases

Production Feedback

• Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time

• 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles
• Strong writing, analysis, and pattern-recognition skills
• Experience evaluating outputs against nuanced criteria rather than only binary correctness
• Experience with finance, accounting, FP&A, transaction services, or business process design preferred
• Ability to work with structured rubrics, scenario libraries, and evidence-based reviews – you are thoughtful, precise, and highly discerning
• Ability to spot subtle output problems others miss
• Comfort with ambiguity but disciplined in scoring and documentation
• Comfort collaborating across Engineering and business teams, with a focus on trust, usefulness, and business reality
• Bachelor's degree preferred

• Excited to be part of a growing team, with a focus on driving future growth
• Passionate about delivering exceptional client service
• Knowledgeable about the relationship between Private Equity sponsors and their portfolio companies, particularly within the office of the CFO
• Willing to get your hands dirty in the details of a project while simultaneously seeing the whole picture
• Comfortable managing projects with multiple complex workstreams while also focused on your single workstream (you are managing and doing)
• A self-starter with a strong work ethic
• Full of entrepreneurial spirit and comfortable in a fluid, flat organization
• Able to effectively communicate complex issues and solutions, and raise issues to senior team members when necessary
• A team player, able to work with team members across all levels
• Someone who enjoys mentoring others and doing meaningful work
• A leader of others; you lead by example

The annual salary for this role ranges from: $110,000 to $145,000 + benefits + bonus.

Actual compensation packages are determined by evaluating a wide array of factors unique to each candidate, including but not limited to geographic location, skill set, years and depth of experience, education, certifications, cost of labor and internal equity.

Accordion is an Equal Opportunity Employer. We are committed to building a team that represents a variety of backgrounds, perspectives, and skills.  We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical​​​ condition, pregnancy, genetic information, gender, sexual orientation, gender identity or ​expression, veteran status, or any other status protected under federal, state, or local law.

Please note that as per Accordion policy, we do not accept unsolicited resumes from third-party recruiters unless such recruiters are engaged to provide candidates for a specified opening and in alignment with our inclusive diversity values. Any employment agency, person or entity that submits an unsolicited resume does so with the understanding that Accordion will have the right to hire that applicant at its discretion without any fee owed to the submitting employment agency, person or entity.

#LI-HA1