Design and build data pipelines using AWS services and Python. Collaborate with data teams to ensure data quality and optimize performance.
About Us
At Qloo, we harness large-scale behavioral and catalog data to power recommendations and insights across entertainment, dining, travel, retail, and more. Our platform is built on a modern AWS data stack and supports analytics, APIs, and machine-learning models used by leading brands. We are looking for an experienced Data Engineer to help evolve and scale this platform.
Role Overview
As a Data Engineer at Qloo, you will design, build, and operate the pipelines that move data from external vendors, internal systems, and public sources into our S3-based data lake and downstream services. You’ll work across AWS Glue, EMR (Spark), Athena/Hive, and Airflow (MWAA) to ensure that our data is accurate, well-modeled, and efficiently accessible for analytics, indexing, and machine-learning workloads.
You should be comfortable owning end-to-end data flows, from ingestion and transformation to quality checks, monitoring, and performance tuning.
Responsibilities
- Design, develop, and maintain batch data pipelines using Python, Spark (EMR), and AWS Glue, loading data from S3, RDS, and external sources into Hive/Athena tables.
- Model datasets in our S3/Hive data lake to support analytics (Hex), API use cases, Elasticsearch indexes, and ML models.
- Implement and operate workflows in Airflow (MWAA), including dependency management, scheduling, retries, and alerting via Slack.
- Build robust data quality and validation checks (schema validation, freshness/volume checks, anomaly detection) and ensure issues are surfaced quickly with monitoring and alerts.
- Optimize jobs for cost and performance (partitioning, file formats, join strategies, proper use of EMR/Glue resources).
- Collaborate closely with data scientists, ML engineers, and application engineers to understand data requirements and design schemas and pipelines that serve multiple use cases.
- Contribute to internal tooling and shared libraries that make working with our data platform faster, safer, and more consistent.
- Document pipelines, datasets, and best practices so the broader team can easily understand and work with our data.
Qualifications
- Bachelor’s degree in Computer Science, Software Engineering, or a related field, or equivalent practical experience.
- Experience with Python and distributed data processing using Spark (PySpark) on EMR or a similar environment.
- Hands-on experience with core AWS data services, ideally including:
- S3 (data lake, partitioning, lifecycle management)
- AWS Glue (jobs, crawlers, catalogs)
- EMR or other managed Spark platforms
- Athena/Hive and SQL for querying large datasets
- Relational databases such as RDS (PostgreSQL/MySQL or similar)
- Experience building and operating workflows in Airflow (MWAA experience is a plus).
- Strong SQL skills and familiarity with data modeling concepts for analytics and APIs.
- Solid understanding of data quality practices (testing, validation frameworks, monitoring/observability).
- Comfortable working in a collaborative environment, managing multiple projects, and owning systems end-to-end.
We Offer
- Competitive salary and benefits package, including health insurance, retirement plan, and paid time off.
- The opportunity to shape a modern cloud-based data platform that powers real products and ML experiences.
- A collaborative, low-ego work environment where your ideas are valued and your contributions are visible.
- Flexible work arrangements (remote and hybrid options) and a healthy respect for work-life balance.
Top Skills
Airflow
Athena
Aws Glue
Emr
Hive
Python
Rds
S3
Spark
SQL
Qloo New York, New York, USA Office
195 Bowery 3rd Floor,, New York, NY, United States, 10002
Qloo New York, New York, USA Office
100 Crosby Street, New York, NY, United States, 10012
Similar Jobs
.png)
Cloud • Insurance • Payments • Software • Business Intelligence • App development • Big Data Analytics
The Data Engineer will build and maintain data solutions, optimize data architectures, and ensure data quality while collaborating with cross-functional teams.
Top Skills:
BigQueryGoogle Cloud PlatformPythonSQL
Fintech • Machine Learning • Payments • Software • Financial Services
The role involves leading data engineering initiatives focusing on data architecture, developing applications in AWS, mentoring talent, and driving technology adoption.
Top Skills:
AWSKafkaPythonScalaSnowflakeSQL
Artificial Intelligence • Cloud • Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
As a Senior Associate Data Engineer, you will build and maintain data pipelines, support analytics initiatives, and uphold data standards while collaborating with teams on data strategy and governance.
Top Skills:
Aws GlueDatabricksDbtGitKafkaPysparkSQL
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
