Design realistic, multi-step data analysis benchmark tasks and create reproducible reference analyses. Evaluate AI-generated analyses for correctness, statistical validity, and interpretation. Collaborate with researchers to improve benchmark quality and document methodologies in Jupyter/Colab notebooks.
This role is for one of our clients
Compensation: $60-$90 per hour
Join a pioneering AI initiative focused on developing the next generation of evaluation benchmarks for frontier AI models. We are seeking experienced Data Scientists and Quantitative Analysts to bring real-world analytical rigor to AI evaluation by designing sophisticated benchmark tasks based on practical data science workflows.
In this role, you will create complex, multi-step analytical challenges that mirror real research and business scenarios—from cleaning datasets and comparing statistical methods to interpreting results and presenting actionable insights. Working closely with AI researchers, you'll help identify where advanced AI models succeed, where they fail, and how evaluation benchmarks can better measure analytical reasoning.
This is a fully remote, full-time engagement requiring approximately 35 hours per week.
RequirementsKey Responsibilities
- Design realistic data analysis challenges inspired by real-world research and analytical workflows, including data preparation, statistical modeling, hypothesis testing, and comparative analysis.
- Develop reproducible reference analyses using Jupyter Notebooks or Google Colab, documenting methodologies and findings with clarity.
- Create benchmark tasks that require objective comparisons between analytical techniques, supported by statistical validation and evidence-based recommendations.
- Evaluate AI-generated analyses for correctness, statistical validity, reasoning quality, and interpretation accuracy.
- Identify analytical errors, flawed assumptions, and reasoning gaps that experienced data professionals would immediately recognize.
- Collaborate with AI researchers and fellow subject matter experts to improve benchmark quality, consistency, and analytical rigor.
- Master's degree, PhD, or equivalent practical experience in Data Science, Statistics, Mathematics, Economics, Operations Research, or another quantitative STEM discipline.
- Minimum 1 year of professional experience in research, research engineering, quantitative analysis, data science, or another data-intensive analytical role.
- Strong hands-on experience with data cleaning, exploratory data analysis, statistical testing, correlation analysis, predictive modeling, and interpretation of analytical results.
- Proficiency with Jupyter Notebooks or Google Colab for building reproducible analytical workflows.
- Strong programming skills in Python, including experience with libraries such as pandas, NumPy, SciPy, scikit-learn, or similar analytical frameworks.
- Working knowledge of Git and collaborative software development practices.
- Excellent written communication skills with the ability to present analytical findings clearly to both technical and non-technical audiences.
- Experience with AI evaluation, benchmark development, AI model assessment, or task authoring is preferred.
- Exceptional analytical thinking, creativity, attention to detail, and the ability to solve complex, open-ended problems independently.
- Ability to commit approximately 35 hours per week on a consistent basis.
- Experience designing reproducible research workflows or analytical evaluation frameworks.
- Familiarity with machine learning, large language models, or AI-assisted data analysis.
- Background in benchmark design, statistical modeling, or quantitative research methodology.
- Experience reviewing analytical work, mentoring analysts, or contributing to research publications.
- Help shape the future of AI by improving how advanced models are evaluated on real-world analytical reasoning.
- Collaborate with leading AI researchers developing frontier evaluation benchmarks.
- Apply your expertise in statistics and data science to advance AI reliability and decision-making capabilities.
- Contribute directly to benchmark development that influences the evolution of next-generation AI systems.
- Enjoy the flexibility of a fully remote engagement while working on impactful AI research initiatives.
We are committed to fostering an inclusive and diverse environment where all qualified applicants receive equal consideration. Reasonable accommodations are available throughout the application and engagement process.
Contract & Engagement Details- Independent contractor engagement.
- Fully remote with flexible working hours.
- Expected commitment of approximately 35 hours per week.
- Project duration may be extended, shortened, or concluded based on project requirements and individual performance.
- Work does not require access to confidential or proprietary information from any current or former employer.
- Payments are issued weekly based on approved work completed.
- At this time, we are unable to support H1-B or STEM OPT candidates.
Similar Jobs
Fintech • Financial Services
Serve as first point of contact for clients managing self-directed retirement accounts via chat, phone, and email. Educate customers on products, resolve inquiries end-to-end, build trust, provide proactive recommendations, and collaborate with internal teams to ensure satisfaction and retention.
Top Skills:
MS Office
Artificial Intelligence • Cloud • Sales • Security • Software • Cybersecurity • Data Privacy
Manages global treasury operations, including cash and liquidity management, forecasting, FX exposure and hedging, investments, banking relationships, debt facilities, reconciliations, compliance, and treasury systems. The role leads process optimization, automation, controls, insurance renewals, and the implementation of Trovata Treasury Management System while partnering with finance, accounting, tax, IT, banks, and senior leadership.
Top Skills:
Banking PortalsGlobal AchHost-To-Host ConnectivityTreasury Management SystemsTrovata
eCommerce • Information Technology • Retail • Industrial
Lead development, deployment, and operationalization of statistical and machine learning models for Grainger's digital products. Design and automate experiments, analyze large datasets, perform feature engineering, and present actionable insights to stakeholders. Support digital product areas (search, recommendations, checkout), mentor junior team members, and apply techniques including regression, classification, boosting, NLP, and time series forecasting.
Top Skills:
AWSDashOraclePowerPointPythonRShinySnowflakeSQLStreamlitTeradata
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
