Cerence Inc. Logo

Cerence Inc.

Senior Principal AI Engineer

Posted 3 Months Ago
Remote
Hiring Remotely in USA
141K-226K Annually
Senior level
Remote
Hiring Remotely in USA
141K-226K Annually
Senior level
Design, build, and operate distributed GPU training infrastructure for large neural networks. Optimize multi-node/multi-GPU execution, diagnose compute/memory/network bottlenecks, improve stability and fault tolerance, and partner with research teams to productionize large-model training pipelines for reliable, high-throughput training at scale.
The summary above was generated by AI

AI That Moves the World. Careers That Move You Forward.
At Cerence AI, you’ll help transform advanced AI into trusted, real-world solutions that make complex technology – think vehicles, robots, smart manufacturing, and emerging intelligent devices – more natural, reliable, and useful. Building on decades of innovation and global deployment in automotive, we are extending our expertise into new products, markets, and intelligent systems. 

As part of our global team, you’ll work alongside leading innovators, researchers, engineers, and business leaders to solve meaningful challenges and turn breakthrough ideas into technology people use every day. You’ll have the opportunity to make a visible impact, help shape Cerence AI’s next chapter, and move your own career forward.

What You Will Work On  

  • Design and operate distributed training systems for large neural networks (autoregressive, diffusion, State Space Models etc.) across GPU clusters 
  • Optimise multi‑node, multi‑GPU execution to maximize throughput and utilization 
  • Diagnose & resolve bottlenecks across compute, memory, and network 
  • Improve training stability and fault tolerance at scale 
  • Partner with research and applied ML teams to productionize large‑model training pipelines  

 Core Responsibilities  

  • Distributed Training Infrastructure  
  • Build and optimize GPU cluster orchestration using: 
  • Slurm  
  • Kubernetes  
  • Ray  
  • RunAI  
  • Ensure efficient scheduling, isolation, and fairness across training workloads 
  • Communication & Networking  
  • Optimize and debug distributed communication using:  
  • NCCL  
  • RDMA  
  • InfiniBand  
  • NVLink  
  • Minimize networking bottlenecks that dominate end‑to‑end training time  

  

  • Training Frameworks  
  • Scale large-model training using:  
  • PyTorch Distributed  
  • Megatron‑LM  
  • DeepSpeed  
  • Own multi‑node launch configurations, failure recovery, and performance tuning  
  • Memory & Performance Optimization  
  • Apply advanced memory optimization techniques:  
  • Activation checkpointing  
  • ZeRO (Stage 1–3) and offload strategies  
  • Balance compute, memory, and communication to push model size and batch scale 

What Success Looks Like  

  •  GPU utilization consistently stays high (>80–90%)  
  • Training scales cleanly from single node to dozens or hundreds of GPUs 
  • Communication overhead is minimized and predictable  
  • Large training jobs run stably for days or weeks without failure  
  • New models can be trained faster, larger, and more reliably than before  

 Required Experience & Skills  

  •  Strongly Required  
  • Deep hands‑on experience with distributed systems or ML systems 
  • Experience running large‑scale workloads on GPU clusters  
  • Production experience with PyTorch distributed training  
  • Strong understanding of parallelism strategies (data, tensor, pipeline parallelism)  
  • Low‑level understanding of GPU communication and networking  
  •  Critical Technical Skills  
  • GPU orchestration: Slurm, Kubernetes, Ray, RunAI  
  • Communication libraries: NCCL, RDMA, InfiniBand, NVLink  
  • Training frameworks: PyTorch Distributed, Megatron‑LM, DeepSpeed 
  • Memory optimisation: activation checkpointing, ZeRO offload techniques  

  

  

Common Problems You’ll Be Solving  

  • Many teams fail at scale because:  
  • GPU utilization is low despite large clusters  
  • Networking and communication dominate training time  
  • Training jobs crash or become unstable at large scale  
  • You will be explicitly focused on eliminating these failure modes.  

 Ideal Background  

  •  This role is a strong fit for individuals who have worked as:  
  • ML Systems Engineer  
  • Distributed Systems Engineer  
  • AI Infrastructure Engineer  
  • HPC Engineer transitioning into ML  
  • Experience working with large language models or foundation models is a strong plus, but deep systems expertise is valued over pure model architecture experience.  

 Why This Role Matters  

 Without robust distributed training infrastructure, progress on large models stalls. This role directly enables:  

  • Larger models  
  • Faster iteration cycles  
  • More reliable research-to-production pipelines  

  

You will be building the foundation that makes large‑scale AI possible. 


What we offer 

We offer a generous compensation and benefits package (in addition to the base salary), including: 

·       Salary range $141,400.00 USD - $226,300.00 USD It is not typical for offers to be made at or near the top of the range. The actual salary will be determined based on experience and other job-related factors. 

·       Annual bonus opportunity 

·       Insurance coverage (medical, dental, vision, life, and disability) 

·       Paid time off 

·       Paid holidays 

·       Company contribution to the 401K

·       Equity awards for certain positions and levels 

·       Remote and/or hybrid work available depending on the position 

All compensation and benefits are subject to the terms and conditions of the underlying plans or programs, as applicable, and may be amended, terminated, or replaced from time to time. 


Cerence AI (NASDAQ: CRNC) helps people and intelligent systems work together naturally and reliably. Proven at global scale in automotive, Cerence provides the trusted interaction and action layer connecting people, machines, and intelligent systems. Its AI platform enables natural interaction and dependable action across complex real-world environments where reliability, safety, security, and performance are critical. Working with customers and partners worldwide, Cerence transforms advanced AI into trusted solutions that bridge the gap between AI potential and real-world performance.


For more information, visit www.cerence.ai.


EQUAL OPPORTUNITY EMPLOYER

Cerence is firmly committed to Equal Employment Opportunity (EEO) and to compliance with all federal, state and local laws that prohibit employment discrimination on the basis of age, race, color, gender, gender identity, gender expression, sex, sex stereotyping, pregnancy, national origin, ancestry, religion, physical or mental disability, medical condition, marital status, citizenship status, sexual orientation, protected military or veteran status, genetic information and other protected classifications. Cerence Equal Employment Opportunity Policy Statement.


All prospective and current Employees need to remain vigilant when it comes to executing security policies in the workplace. This includes:

- Following workplace security protocols and training programs to familiarize with the ways to maintain a safe workplace.
- Following security procedures to report any suspicious activity.
- Having respect for corporate security procedures to allow those procedures to be effective.
- Adhering to company's compliance and regulations.
- Encouraging to follow a zero tolerance for workplace violence.

- Basic knowledge of information security and data privacy requirements (e.g., how to protect data & how to be handling this data).

- Demonstrative knowledge of information security through internal training programs.

Similar Jobs

8 Days Ago
Remote
United States
220K-350K Annually
Senior level
220K-350K Annually
Senior level
Hardware • Information Technology • Software
Build analytical and simulation-based performance models for large-scale AI inference deployments on Positron hardware. Model parallelism, networking, batching, latency, throughput, and mixed accelerator configurations; validate projections against hardware measurements; calibrate models as systems evolve; and create customer-facing sizing and performance-estimation tools. Partner with customers, product, and chip-modeling teams to compare deployments, explain performance drivers, and communicate assumptions and confidence levels.
Top Skills: CC++Chip Performance SimulatorsData-Center NetworkingGpuHpcLarge Language ModelsPythonRust
One Month Ago
Remote or Hybrid
212K-407K Annually
Senior level
212K-407K Annually
Senior level
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Architect and engineer secure, scalable enterprise AI infrastructure for LLM training and inference, machine learning, agentic AI, and production platforms. Lead HPC, GPU, Kubernetes, cloud, networking, storage, reliability, observability, security, and automation strategies. Resolve complex performance and scalability challenges, establish engineering standards, evaluate emerging technologies, lead cross-functional initiatives, and mentor engineers in a regulated environment.
Top Skills: Accelerated ComputingAgentic AiCloud-Native PlatformsDistributed SystemsGpuHigh-Performance NetworkingHpcHybrid CloudInfrastructure AutomationKubernetesLlmsMachine LearningObservabilityPrivate CloudRack-Scale SystemsSreStorage
One Month Ago
Remote or Hybrid
New York, NY, USA
212K-407K Annually
Senior level
212K-407K Annually
Senior level
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Leads the architecture and engineering of secure, scalable enterprise AI infrastructure. Designs platforms for LLM training and inference, machine learning, agentic AI, and production workloads using HPC, GPUs, Kubernetes, hybrid cloud, networking, storage, and automation. Establishes standards for reliability, security, observability, governance, and compliance; resolves complex performance challenges; evaluates emerging technologies; leads cross-functional initiatives; and mentors engineers.
Top Skills: Accelerated ComputingAgentic AiDistributed SystemsGpu ComputingHigh-Performance NetworkingHigh-Performance StorageHpcHybrid CloudInfrastructure AutomationKubernetesLlmsMachine LearningObservabilityPrivate CloudRack-Scale SystemsResponsible AiSre

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account