Principal Software Engineer – AI Platform (Production Engineering / Reliability)

We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time.

• Own and evolve production operations strategy for AI/ML platforms and services
• Define SLOs, SLIs, and error budgets for AI systems (online & batch/inference pipelines)
• Lead root cause analysis (RCA) and drive systemic improvements post-incident
• Establish operational readiness standards for launching new AI capabilities
• Build frameworks for on-call excellence, incident response, and escalation

• Design and implement end-to-end observability systems across AI workloads:
  • Model performance monitoring
  • Data pipeline health
  • Infrastructure metrics
• Build and scale monitoring and alerting frameworks using modern tooling (e.g., Prometheus, Grafana, OpenTelemetry, Datadog, Azure Monitor, etc.)
• Define actionable, low-noise alerts tied to business and system impact
• Develop dashboards and telemetry standards for real-time visibility across services
• Drive adoption of golden signals (latency, errors, throughput, saturation) in AI systems

• Ensure reliable deployment and operation of:
  • Real-time inference services
  • Model pipelines (training, validation, deployment)
  • Data ingestion and feature pipelines
• Implement model observability (drift detection, data skew, performance degradation)
• Partner with ML engineers to improve production readiness of models
• Establish lifecycle standards for models in production environments

• Build internal platforms and tooling for:
  • Automated incident detection and response
  • Self-healing systems
  • Deployment validation and canarying
• Drive Infrastructure as Code (IaC) and policy automation
• Improve system resilience through chaos testing and fault injection

• Act as a trusted technical advisor across platform, ML, and product teams
• Set direction for operational excellence in AI systems at org scale
• Mentor senior engineers and influence cross-team architectural decisions
• Lead adoption of industry best practices in reliability engineering and observability

• 10+ years in software engineering, production engineering, or SRE roles
• Deep experience operating large-scale distributed systems in production
• Proven track record building monitoring, observability, and alerting systems
• Strong expertise in incident management and production support models
• Experience working with cloud platforms (Azure, AWS, GCP)

• Experience supporting AI/ML platforms or data-intensive systems
• Familiarity with model lifecycle management and MLOps practices
• Knowledge of:
  • OpenTelemetry, Prometheus, Grafana, Datadog
  • Kubernetes and containerized workloads
  • Streaming systems (Kafka, Event Hub, etc.)
• Experience defining and implementing SLO-driven engineering
• Background in high-availability, low-latency systems

• Systems thinking and ability to reason about complex, interdependent systems
• Strong bias for automation, scalability, and long-term solutions
• Exceptional debugging and incident management skills
• Ability to influence without authority across multiple teams
• Passion for operational excellence and reliability

Pay Range

The typical pay range for this role is:

This pay range represents the base hourly rate or base annual full-time salary for all positions in the job grade within which this position falls.  The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors.  This position is eligible for a CVS Health bonus, commission or short-term incentive program in addition to the base pay range listed above.  This position also includes an award target in the company’s equity award program. 
 

Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.

Great benefits for great people

We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families.

Additional details about available benefits are provided during the application process and on Benefits Moments.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.