Temporal Technologies Logo

Temporal Technologies

Senior Software Engineer, Reliability

Posted An Hour Ago
Be an Early Applicant
Remote
Hiring Remotely in United States
141K-231K Annually
Senior level
Remote
Hiring Remotely in United States
141K-231K Annually
Senior level
Own end-to-end reliability for Temporal Cloud by building an AI-powered Reliability Agent, improving incident response, defining SLOs and operational readiness practices, running gamedays and chaos testing, and leading load, performance, capacity, and observability initiatives. The role also includes post-incident reviews, reliability scorecards, cross-team collaboration, mentoring, and on-call participation.
The summary above was generated by AI
Role Summary

Join the Reliability team as a Senior Software Engineer, reporting to Tushar Roy, and help improve the reliability of operating Temporal Cloud end to end. You’ll build an AI-first reliability experience, including building a Reliability Agent from scratch that helps the SRE team triage common issues, surface useful context, and recommend next steps. You’ll also help make reliability knowledge and practices accessible across engineering so ownership is shared rather than concentrated with a small number of experts.

 
What You’ll Do
  • Own reliability outcomes for operating Temporal Cloud end to end, partnering across engineering, infrastructure, and product to drive measurable improvements.

  • Build and evolve the Reliability Agent, including workflows that gather context from telemetry, incidents, runbooks, deployments, and service history to support faster, more consistent triage.

  • Design safe, useful agent experiences for handling routine reliability questions and issues, with appropriate human oversight, escalation paths, auditability, and safeguards against incorrect actions.

  • Strengthen incident response for Temporal Cloud, from detection and triage through escalation, mitigation, and communication, and lead post-incident reviews that turn into durable corrective actions.

  • Define and evolve reliability practices, including service-level objectives, alerting thresholds, operational readiness criteria, and escalation paths.

  • Plan and run gamedays and safely expand chaos testing to validate failure response, evaluate agent-assisted operations, and reduce operational risk.

  • Build and maintain reliability scorecards that make service health, operational readiness, agent effectiveness, and improvement priorities visible.

  • Lead load, performance, and capacity testing, and improve observability standards across services.

 
What You’ll Bring
  • Strong computer science fundamentals, especially in distributed systems, concurrency, performance, and fault-tolerant design.

  • Experience designing, building, or operating complex systems under high load and partial failure.

  • Experience building AI-powered tools, agents, or production automation, with an understanding of tool use, evaluation, state, failure modes, and human-in-the-loop design.

  • A track record of driving reliability improvements across multiple services, systems, or engineering teams.

  • Hands-on experience with incident response and post-incident reviews, including leading incidents and following corrective actions through to completion.

  • Hands-on experience with one or more of gamedays, chaos testing, load testing, capacity planning, reliability scorecards, or observability programs.

  • Sound judgment in ambiguous situations, with the ability to balance reliability, agent autonomy, safety, customer impact, and delivery speed.

  • Clear communication skills and the ability to align stakeholders, share operational knowledge, mentor others, and participate in an on-call rotation. This is a remote United States role with no regular travel expected.

 
Nice to Haves
  • Experience operating multi-tenant systems and designing protections against noisy-neighbor behavior.

  • Experience building internal platforms or tooling that enables teams to meet reliability standards.

  • Familiarity with LLM observability, agent evaluation, retrieval-augmented generation, or workflow orchestration systems.

  • Open source contributions in infrastructure, distributed systems, AI, or reliability engineering.

Temporal Technologies is an Equal Opportunity Employer. Temporal Technologies does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status, or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need. We embrace and celebrate differences and diversity.

Temporal is committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, its services, programs, and activities. If you need to request a reasonable accommodation, please let your Recruiter know so we can assist.

Similar Jobs

4 Days Ago
Easy Apply
Remote or Hybrid
United States
Easy Apply
162K-290K Annually
Senior level
162K-290K Annually
Senior level
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Lead Samsara’s AI initiatives across the software development lifecycle, including developer agents and SRE AI. Set strategy, select experiments, make build-versus-buy decisions, control infrastructure and token costs, scale automated incident response, and build trust in AI-driven actions. Manage and grow a team of five to ten senior engineers while partnering directly with R&D leaders and developers.
Top Skills: Ai AgentsClaudeCloud InfrastructureCodexIncident Response AutomationIotModel Context Protocol (Mcp)Sre Tooling
One Month Ago
Easy Apply
Remote or Hybrid
2 Locations
Easy Apply
126K-248K Annually
Senior level
126K-248K Annually
Senior level
Big Data • Cloud • Software • Database
Build and operate security infrastructure for MongoDB Atlas across AWS, Azure, and Google Cloud. Responsibilities include hardening Linux hosts and containers, implementing cloud security controls, workload identity, vulnerability management, Infrastructure-as-Code, runtime threat detection, and monitoring systems. The role leads initiatives from threat modeling and architecture through deployment and long-term production reliability.
Top Skills: AnsibleApparmorAWSAzureCC++CgroupsCiliumCloudFormationEbpfFalcoGatekeeperGoGoogle Cloud PlatformKubernetesLinuxMongodb AtlasPythonRbacRustSeccompSelinuxTerraformTetragon
20 Days Ago
Remote
USA
131K-190K Annually
Senior level
131K-190K Annually
Senior level
Healthtech • Biotech
Build and operate site reliability engineering practices for regulated, cloud-based production environments. Responsibilities include observability, SLIs/SLOs, incident response, infrastructure automation, Kubernetes and cloud operations, disaster recovery exercises, compliant deployments, root-cause analysis, and reliability improvements. The role partners with engineering, product, laboratory, compliance, and quality teams while mentoring peers and helping establish the organization’s SRE culture.
Top Skills: AksArmAzureBicepCi/CdDatadogEksGCPGkeGoGrafanaIncident.IoKubernetesOpentelemetryOpsgeniePagerdutyPrometheusPulumiPythonServicenowTerraform

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account