.png){kind=logo}
Build and operate scalable data platforms, high-volume batch and streaming ingestion pipelines, data lake infrastructure, and distributed processing systems. Manage Kubernetes and KEDA autoscaling, provision infrastructure with Terraform, and support orchestration platforms such as Airflow or Prefect. Ensure data quality, governance, compliance, and performance while providing technical guidance on infrastructure, CI/CD, testing, and system design.
Description
We are looking for a highly skilled Senior Data Platform Engineer to join our Data Platform team and play a key role in designing, building, and evolving the company’s data infrastructure.
This is a hands-on engineering position focused on developing scalable data platforms, high-volume ingestion pipelines, and the foundational data systems that power analytics, product applications, AI workloads, and large-scale model execution. You will help architect our data lake, build reliable ingestion capabilities at scale, and shape the core infrastructure that supports the company’s future growth.
This is an exciting opportunity to work with cutting-edge technologies and have a defining impact on the architecture and evolution of our data platform.
What You’ll Do
- Build complex, high-volume batch and near-real-time ingestion services while helping evolve the platform’s streaming capabilities.
- Design and implement data solutions for all application requirements in a distributed microservices environment
- Develop and optimize large-scale data pipelines for batch and streaming use cases.
- Ensure data quality, compliance, and governance.
- Manage and optimize Kubernetes infrastructure, utilizing KEDA to dynamically scale resources for large-scale data processing workloads.
- Own the full lifecycle of our data stack using Terraform and Kubernetes, from provisioning to application-level frameworks.
- Provide technical guidance for the team, drive best practices around infrastructure, CI/CD, testing, and system design.
- Deep-dive into memory management and performance tuning to ensure our large-scale distributed systems are both cost-effective and lightning-fast.
- 5+ years of experience as a Data Platform/Infra Engineer
- Proven track record of building and operating scalable data infrastructure specifically within Data Lake architectures.
- Proven experience designing and operating large-scale data processing workloads on Kubernetes, including workload autoscaling with KEDA.
- Strong proficiency in Python and SQL.
- Hands-on experience with Infrastructure as Code, preferably Terraform.
- Experience with managing a data orchestration platform such as Airflow or Prefect
- Experience with Snowflake.
Similar Jobs
Healthtech
Develop and support production-grade backend platforms, APIs, data pipelines, orchestration frameworks, and machine learning deployment systems. Build scalable services for data ingestion, feature processing, algorithm execution, model deployment, and workflow automation. Improve system performance, reliability, and scalability while guiding testing, observability, and production readiness. Collaborate across software engineering, data science, product, and architecture teams, and provide technical leadership and mentorship as a hands-on engineer.
Top Skills:
APIsC++Ci/CdCloud-Native ArchitecturesJavaKubernetesMachine Learning DeploymentPython
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Leads the strategy, delivery, scaling, governance, and adoption of agentic AI products across Pfizer’s regulated manufacturing network. Owns product roadmaps, agent lifecycles, evaluation frameworks, responsible AI controls, value realization, and stakeholder engagement. Partners with manufacturing, engineering, data, quality, cybersecurity, and site leadership teams to transform operational knowledge into intelligent digital products and scale successful capabilities globally.
Top Skills:
Advanced AnalyticsAgentic AiAi AgentsAi OrchestrationCloud Data PlatformsDigital Operations Centre (Doc)HistoriansLimsLlmopsMesMlopsPlant ConnectivityScadaSerialization
Artificial Intelligence • Productivity • Software • Automation
Serve as a strategic People Business Partner for Data and Product leaders, owning organizational design, workforce planning, talent reviews, performance management, change management, and employee relations. Advise leaders on complex performance cases, exits, investigations, and employment matters across multiple EMEA jurisdictions. Drive people processes, partner with Centers of Excellence, support employee relations initiatives, and use AI-enabled tools to improve productivity and scalability in a fully remote environment.
Top Skills:
Ai ToolsAutomation
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
