The Research Engineer will design scalable evaluation systems for multimodal generative AI models, integrating metrics and automated pipelines for model assessment and improvement.
About Luma AI
Luma's mission is to build multimodal AI to expand human imagination and capabilities. We believe that multimodality is critical for intelligence. To go beyond language models and build more aware, capable, and useful systems, the next step function change will come from vision. So, we are working on training and scaling up multimodal foundation models for systems that can see and understand, show and explain, and eventually interact with our world to effect change.
About the Role
Luma is pushing the boundaries of generative AI, building tools that redefine how visual content is created. We're seeking a Research Engineer to design and scale the infrastructure that powers our model evaluation efforts. This role is about building the pipelines, metrics, and automated systems that close the loop between model output, evaluation, and improvement. You'll work across research, engineering, and product teams to ensure our models are measured rigorously, consistently, and in ways that directly inform development.
Responsibilities
- Design and implement scalable pipelines for automated evaluation of generative models, with a focus on visual and multimodal outputs (image, video, text, audio).
- Develop novel metrics and evaluation models that capture qualities like fidelity, coherence, temporal consistency, and alignment with human intent.
- Integrate evaluation signals into training loops (including reinforcement learning and reward modeling) to continuously improve model performance.
- Build infrastructure for large-scale regression testing, benchmarking, and monitoring of multimodal generative models.
- Collaborate with researchers running human studies to translate human evaluation frameworks into automated or semi-automated systems.
- Partner with model researchers to identify failure cases and build targeted evaluation harnesses.
- Maintain dashboards, reporting tools, and alerting systems to surface evaluation results to stakeholders.
- Stay current with emerging evaluation techniques in generative AI, multimodal LLMs, and perceptual quality assessment.
Qualifications
- Master's or PhD in Computer Science, Machine Learning, or a related technical field (or equivalent industry experience).
- 5+ years of experience building ML evaluation systems, model pipelines, or large-scale infrastructure.
- Hands-on experience working with visual data (images and/or video), including evaluation, modeling, or data preparation.
- Proficiency in Python and ML frameworks (PyTorch, JAX, or TensorFlow).
- Familiarity with human-in-the-loop evaluation workflows and how to scale them with automation.
- Strong background in machine learning, with experience in generative models (diffusion, LLMs, multimodal architectures).
- Strong software engineering skills (CI/CD, testing, data pipelines, distributed systems).
Nice to Have
- Experience with reinforcement learning or reward modeling.
- Prior work on perceptual metrics, multimodal evaluation benchmarks, or retrieval-based evaluation.
- Background in large-scale model training or evaluation infrastructure.
- Experience designing metrics for perceptual quality
- Familiarity with creative media workflows (film, VFX, animation, digital art).
- Contributions to open-source evaluation libraries or benchmarks.
The base pay range for this role is $190,000 – $375,000 per year.
About LumaLuma’s mission is to build unified general intelligence that can generate, understand, and operate in the physical world.
We believe that multimodality is critical for intelligence. To go beyond language models and build more aware, capable and useful systems, the next step function change will come from vision. So, we are working on training and scaling up multimodal foundation models for systems that can see and understand, show and explain, and eventually interact with our world to effect change.
Similar Jobs
Information Technology • Insurance • Software
Support review, rebranding, updating, and restructuring of English and French product documentation. Validate workflows, capture screenshots, apply adult learning principles, update release-driven changes, partner with product teams, and help establish repeatable documentation and version-control processes to improve usability and bilingual consistency.
Top Skills:
Content Management SystemsDocumentation ToolsKnowledge BasesMS OfficeSharepoint
Artificial Intelligence • Cloud • Consumer Web • Productivity • Software • App development • Data Privacy
Lead vision, strategy, and execution for Dropbox's next-generation design system and design technology platform. Build reusable patterns, components, tokens, and governance; partner with product, engineering, research, and brand; raise interaction, accessibility, and implementation quality; create adoption and measurement models; and develop a multidisciplinary team to scale coherent, high-quality multi-product experiences including AI-native capabilities.
Top Skills:
Accessibility InfrastructureAi-Assisted ExperiencesComponent ArchitectureContent SystemsDesign SystemsDesign TechnologyDesign TokensFront-End EngineeringPrototyping Tools
.png){kind=logo}
AdTech • Cloud • Digital Media • Information Technology • News + Entertainment • App development
Lead data curation and validation for production ML: coordinate with ML/product/TPM teams, define data and training specs, oversee pipelines, ensure representative, high-quality behavioral datasets, and translate product goals into actionable data requirements.
Top Skills:
ConfluenceData Visualization ToolsExperiment Tracking FrameworkGitGit ServerJIRAPythonSlackUnix Shell
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
