The role involves designing and implementing data pipelines and systems for large datasets, supporting scientific processes and experimentation.
Hiring for multiple roles.
Traditional forecasting breaks under petabytes of features, megabytes of labels, and time-varying noise. Our breakthrough research is foundational in the field: generating industry-beating results (MIT News). Our predictive engines combine a wide variety of data through rigorous scientific process. We leverage alternative datasets (consumer transactions, geolocation, etc) along with private, public, external, and internal data.
The Data Platform is a critical component in our scientific approach to forecasting. The role will require everything from high-level architecture design to performant implementation.
Responsibilities include:
- Build distributed systems supporting complex transformations over petabyte-scale data
- Design, implement, and maintain production cloud data pipelines for complex, disparate datasets
- Work closely with our Quants and Applied Scientists to support rigorous scientific process, rapid experimentation, and new product initiatives
Required qualifications:
- Bachelor’s or advanced degree in Computer Science, math or equivalent
- 3+ years of experience in data engineering or related fields with a track record of building (coding) and scaling a data platform
- 2+ years of industry experience building ETL pipelines in a major cloud provider
- Highly proficient in python and SQL
- Advanced knowledge of databases, design patterns, OOP, and concurrency
- Knowledge of data management fundamentals and data storage principles
- Passion for intuitive data models
- Experience communicating with senior management as well as colleagues from engineering, analytics, and business backgrounds
- Experience with AWS, orchestration platforms, and modern tech stacks
- Strong software engineering fundamentals: system design, algorithms, data structures
Nice-to-haves:
- Experience building systems that integrate LLMs with large-scale data processing
- Working understanding of basic probability and statistics
- Experience implementing ML models
- Experience with DBT
- MS or PhD in STEM field
In-person at our NYC office strongly preferred. Remote candidates who can spend some time in-office (e.g., monthly) will also be considered.
Benefits:
- The salary range for this position is $145,000-$220,000
- Free health, vision, dental insurance
- 401k with match
- Unlimited PTO + 13 company holidays
- Daily Grubhub lunch stipend
- Pre-tax commuter benefits
Similar Jobs
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Design, build, and operate scalable, secure data platform infrastructure for ingesting, processing, cataloging, and accessing petabytes of data. Improve Spark/Databricks reliability and developer experience, build ingestion/replication systems, develop internal libraries and tooling (Go/Python), and collaborate with cross-functional teams to support analytics, ML, and customer-facing data products.
Top Skills:
AirflowAmundsenSparkAws RdsCloudFormationDagsterDatabricksDatahubDelta LakeDockerDynamoDBEcsFargateGoHive MetastoreHudiIcebergJavaKinesisKubernetesLambdaPrefectPythonS3ScalaSqsTerraformUnity Catalog
Fintech
Design, build, and operate Brex's data enablement platform (warehouse, streaming, replication, orchestration, observability, governance). Partner with product, data science, and analytics teams to deliver data-backed products, improve data competency, and maintain resilient, on-call data architectures.
Top Skills:
AirflowCdcDbtFlinkKafkaKotlinPythonSnowflake
Insurance
The Senior Software Engineer, Data Platform will design and maintain data infrastructure, build pipelines, ensure data quality, and collaborate across teams, mentoring others while driving projects to completion.
Top Skills:
BigQueryDbtFivetranGCPGitPythonSQLTableau
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
