The Data Engineer will design, build, and maintain scalable data processing pipelines, collaborate with teams, and ensure data reliability and quality.
We are seeking an experienced Data Engineer to design, build, and maintain scalable data processing pipelines using modern big data technologies. The ideal candidate will have strong development skills and experience with distributed computing frameworks.
Required Skills & Qualifications:
Technical Requirements:
Preferred Qualifications:
Key Responsibilities:
Soft Skills:
The ideal candidate should have a proven track record of developing and maintaining production-grade data pipelines and be passionate about working with large-scale data systems.
Please note this position requires an active IRS MBI clearance.
Required Skills & Qualifications:
- Bachelor's degree in Computer Science, Engineering, or related technical field
- 5+ years of software development experience
- Strong proficiency in Python, Java or Scala programming languages
- Extensive hands-on experience with Apache Spark, including:
- Spark SQL
- Spark Streaming and its data sources KAFKA, CDC feed, etc
- Performance optimization and tuning
- Data transformation and processing
- Hands on Experience working with cloud-based Spark platforms (Databricks, AWS EMR, AWS Glue)
- Strong understanding of Hive, Unity catalog, Glue catalog
- Hands on Experience working with any data quality framework/tools
- Hands on Experience working with Observability and monitoring for data processing pipelines
- Strong understanding of distributed computing concepts
- Proficiency in version control systems (Github/GitLab/Bitbucket)
Technical Requirements:
- Experience building and maintaining high-volume data processing pipelines
- Knowledge of data modeling and ETL / ELT best practices
- Familiarity with SQL and NoSQL databases
- Understanding of data warehouse concepts and dimensional modeling
Preferred Qualifications:
- Experience with real-time data processing and streaming architectures
- Knowledge of Delta Lake or similar data lakehouse technologies
- Experience with CI/CD pipelines
- Cloud platform expertise (AWS/Azure/GCP)
- Contributions to open-source projects
Key Responsibilities:
- Design and implement scalable data pipelines
- Optimize existing data workflows for performance and reliability
- Collaborate with data scientists and analysts to support their data needs
- Implement data quality checks and monitoring
- Maintain documentation for data processes and architectures
Soft Skills:
- Strong problem-solving abilities
- Excellent communication skills
- Team collaboration
- Ability to work independently
- Strong attention to detail
The ideal candidate should have a proven track record of developing and maintaining production-grade data pipelines and be passionate about working with large-scale data systems.
Please note this position requires an active IRS MBI clearance.
Top Skills
Spark
Aws Emr
Aws Glue
Bitbucket
Databricks
Git
Gitlab
Hive
Java
Python
Scala
Similar Jobs
Fintech
The Data Engineer is responsible for designing and maintaining data pipelines, optimizing data systems, and collaborating with developers and analysts to ensure data quality and consistency.
Top Skills:
Apache AirflowC#/.NetLinuxPythonSQL
.png)
Cloud • Insurance • Payments • Software • Business Intelligence • App development • Big Data Analytics
The Data Engineer will build and maintain data solutions, optimize data architectures, and ensure data quality while collaborating with cross-functional teams.
Top Skills:
BigQueryGoogle Cloud PlatformPythonSQL
Fintech • Machine Learning • Payments • Software • Financial Services
As a Data Engineer at Capital One, you will design and implement cloud-based data solutions, collaborate with Agile teams, and utilize technologies such as Java, Scala, and SQL. You will ensure code quality through testing and performance tuning while mentoring less experienced engineers.
Top Skills:
AWSDatabricksEmrGCPHadoopHiveJavaKafkaMapreduceAzureMySQLPythonScalaSnowflakeSparkSQLUnix/Linux
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
