Syllo Logo

Syllo

Staff Software Engineer, AI Inference

Reposted 3 Months Ago
Remote
Hiring Remotely in USA
190K-230K Annually
Senior level
Remote
Hiring Remotely in USA
190K-230K Annually
Senior level
Lead design and implementation of a production inference platform for LLMs. Define roadmap, integrate and optimize inference runtimes, manage GPU infrastructure, and build systems for deployment, autoscaling, observability, and operational excellence. Benchmark models/runtimes, partner with ML and product teams, make architectural tradeoffs, and mentor engineers to scale AI infrastructure.
The summary above was generated by AI

About Syllo 

Syllo is on a mission to transform litigation. Our product is a unified litigation platform that enables lawyers and paralegals to safely harness the power of language models and agentic AI throughout the litigation life cycle. Since going to market, we have gained a diverse group of enterprise customers, including some of the biggest law firms and corporations in the country, and we are quickly expanding. By reducing the expense of litigation industry-wide, we aim to improve access to high-quality representation and promote the alignment of legal outcomes with merit. 


About the Role 

As we continue to scale our AI platform, we're investing in our own inference stack to deliver best-in-class performance, reliability, cost efficiency, and flexibility across the latest generation of open-source language models.

We're looking for a Staff Software Engineer to spearhead this effort.

You'll define the architecture, evaluate emerging technologies, and build the systems that power model serving.  You'll partner closely with machine learning, infrastructure, and product engineering to establish the foundation for how AI models are deployed, optimized, monitored, and operated in production.

This is a highly hands-on technical role. You'll spend the majority of your time designing, building, and optimizing production systems while helping shape our long-term AI infrastructure strategy.

Responsibilities 

  • Lead the design and development of our production inference platform.
  • Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
  • Build and operate scalable, cost-effective systems for serving large language models in production.
  • Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
  • Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
  • Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
  • Partner with ML engineers to productionize new models and inference techniques.
  • Establish benchmarking methodologies to evaluate new models, runtimes, and hardware.
  • Make key architectural decisions around when to build internally versus leverage open-source or commercial solutions.
  • Mentor engineers as the team grows and help establish engineering best practices for AI infrastructure.

Qualifications

  • Significant experience designing and operating production AI inference systems.
  • Experience building or leading production LLM serving infrastructure.
  • Deep experience with one or more modern inference runtimes and frameworks such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, Hugging Face TGI, NVIDIA Dynamo, or comparable technologies.
  • Strong background in distributed systems, backend infrastructure, or high-performance platform engineering.
  • Experience optimizing inference performance across GPU workloads, including latency, throughput, batching, memory utilization, and serving efficiency.
  • Experience operating GPU infrastructure in production.
  • Strong proficiency in Python and at least one systems programming language (such as Go, Rust, or C++).
  • Proven ability to lead technical architecture for complex infrastructure initiatives.
  • Excellent communication skills and the ability to influence technical direction across engineering teams.

Salary Range ($190- $230K) plus health insurance and equity. 

United States - Remote Pay Range
$190,000—$230,000 USD

Similar Jobs

An Hour Ago
Remote or Hybrid
45K-85K Annually
Junior
45K-85K Annually
Junior
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Handles inbound calls and warm leads to consult customers on insurance needs, recommend appropriate Property and Casualty coverage, and convert prospects into policyholders. The role includes paid licensing and sales training, customer relationship building, persuasive communication, and compliance with state licensing requirements. Representatives work remotely on a weekday and weekend schedule and must maintain a dedicated workspace with high-speed wired internet.
Top Skills: PcWired High-Speed Internet
3 Hours Ago
Remote or Hybrid
Senior level
Senior level
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Designs, develops, and maintains AI-enabled automation solutions on the enterprise ServiceNow platform. The role builds intelligent workflows, agents, conversational experiences, and integrations across ITSM, ITOM, ITAM, HRSD, and CSM. Responsibilities include identifying automation opportunities, implementing Now Assist and generative AI capabilities, establishing AI governance and human oversight, ensuring security and compliance, monitoring solution effectiveness, and maintaining testing and audit documentation.
Top Skills: Ai Agent StudioAi AgentsFlow DesignerGenerative AiGlide ApisIntegrationhubJavaScriptLlmsNow AssistPredictive IntelligencePrompt EngineeringPythonRest ApisRobotic Process Automation (Rpa)ServicenowSoap ApisVirtual AgentWorkflow OrchestrationWorkflow Studio
4 Hours Ago
Remote or Hybrid
United States
150K-200K Annually
Senior level
150K-200K Annually
Senior level
Automotive • Big Data • Information Technology • Robotics • Software • Transportation • Manufacturing
Leads Cadillac sales performance across the Northeast zone by managing district teams, developing dealer relationships, improving market share and profitability, and executing customer, marketing, inventory, training, and aftersales strategies. Analyzes market and sales data, oversees dealer compliance and facility standards, coaches teams, manages contractual performance expectations, and drives adoption of digital marketing and dealership tools. This field-based role requires frequent travel and relocation or residence within the assigned territory.
Top Skills: Google Workspace

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account