The role involves designing and implementing data pipelines and systems for large datasets, supporting scientific processes and experimentation.
Hiring for multiple roles.
Traditional forecasting breaks under petabytes of features, megabytes of labels, and time-varying noise. Our breakthrough research is foundational in the field: generating industry-beating results (MIT News). Our predictive engines combine a wide variety of data through rigorous scientific process. We leverage alternative datasets (consumer transactions, geolocation, etc) along with private, public, external, and internal data.
The Data Platform is a critical component in our scientific approach to forecasting. The role will require everything from high-level architecture design to performant implementation.
Responsibilities include:
- Build distributed systems supporting complex transformations over petabyte-scale data
- Design, implement, and maintain production cloud data pipelines for complex, disparate datasets
- Work closely with our Quants and Applied Scientists to support rigorous scientific process, rapid experimentation, and new product initiatives
Required qualifications:
- Bachelor’s or advanced degree in Computer Science, math or equivalent
- 3+ years of experience in data engineering or related fields with a track record of building (coding) and scaling a data platform
- 2+ years of industry experience building ETL pipelines in a major cloud provider
- Highly proficient in python and SQL
- Advanced knowledge of databases, design patterns, OOP, and concurrency
- Knowledge of data management fundamentals and data storage principles
- Passion for intuitive data models
- Experience communicating with senior management as well as colleagues from engineering, analytics, and business backgrounds
- Experience with AWS, orchestration platforms, and modern tech stacks
- Strong software engineering fundamentals: system design, algorithms, data structures
Nice-to-haves:
- Experience building systems that integrate LLMs with large-scale data processing
- Working understanding of basic probability and statistics
- Experience implementing ML models
- Experience with DBT
- MS or PhD in STEM field
In-person at our NYC office strongly preferred. Remote candidates who can spend some time in-office (e.g., monthly) will also be considered.
Benefits:
- The salary range for this position is $145,000-$220,000
- Free health, vision, dental insurance
- 401k with match
- Unlimited PTO + 13 company holidays
- Daily Grubhub lunch stipend
- Pre-tax commuter benefits
Similar Jobs
Artificial Intelligence • Software
Own the architecture and implementation of production data ingestion systems for manufacturing data. Build scalable backend services and cloud infrastructure using Python, PostgreSQL, and Kubernetes; solve integrations across heterogeneous enterprise systems; establish validation, lineage, monitoring, and alerting; and partner with Engineering, Product, and customer-facing teams. Provide technical mentorship, influence system design, and balance rapid delivery with production-grade reliability.
Top Skills:
KubernetesPostgresPython
Information Technology • Consulting
Build and operate a modern data platform for network telemetry, including scalable ingestion pipelines, real-time event processing, workflow and rule engines, APIs, data models, lakehouse storage, observability, and lineage. Deploy services on Kubernetes using CI/CD, troubleshoot distributed systems, contribute to architecture and code quality, and collaborate with product, front-end, operations, and business stakeholders. Use AI-assisted development tools and participate in a light on-call rotation.
Top Skills:
Apache AirflowApache BeamApache FlinkApache IcebergApache KafkaApache Spark Structured StreamingAvroBigQueryCi/CdClaude CodeClickhouseCursorDagsterGithub CopilotJavaKafka StreamsKubernetesPrefectPrestoPythonSchema RegistrySnowflakeSpark SqlSQLStarrocksTrino
Social Media
Lead the strategy, technical direction, and long-term roadmap for Pinterest’s big data and AI infrastructure. Build and scale frameworks for compute, job management, resource management, scheduling, and remote shuffling across petabyte-scale datasets. Partner with internal customers on critical big data use cases and provide company-wide technical leadership on reliable, efficient data processing and storage.
Top Skills:
AWSFlinkGoJavaKubernetesPrestoPythonPyTorchRayScalaSparkTensorFlow
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
