Build and maintain large-scale data processing pipelines using Apache Spark (PySpark/Scala) and Python. Design ETL workflows, implement distributed data processing, use version control (Git), and work with cloud platforms (GCP/AWS/Azure).
Job Title: Data Engineer (PySpark / Scala / Python)
Location: Remote
Job Description:We are hiring a Data Engineer with strong hands-on experience in PySpark, Scala, and Python. You must have solid expertise in Apache Spark, as it will be the core technology used for building and managing large-scale data processing pipelines.
Experience with cloud platforms like Google Cloud Platform (GCP), Microsoft Azure, or AWS is a plus.
Required Skills:Strong hands-on experience with Apache Spark
Proficient in PySpark
Experience in Scala and Python
Knowledge of ETL processes and data pipeline design
Understanding of distributed data processing
Familiarity with version control tools like Git
Basic knowledge of cloud platforms (GCP, AWS, or Azure)
Experience with cloud-native data tools (e.g., Dataproc, Glue, EMR, BigQuery)
Familiarity with workflow/orchestration tools like Airflow or Cloud Composer
Experience with CI/CD for data engineering
Exposure to both structured and unstructured data
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design, build, and maintain scalable batch and real-time data pipelines on an Azure data platform using Azure Data Factory, Databricks, and PySpark. Implement Medallion architecture (Bronze/Silver/Gold), enforce data quality and governance, optimize Spark jobs and cluster usage, monitor pipeline health, and collaborate with data science and reporting teams. Document lineage, handle CDC/incremental loads, and integrate streaming ingestion and validation checks to ensure reliable analytics and ML-ready datasets.
Top Skills:
SparkAzureAzure Data FactoryAzure Event HubsChange Data Capture (Cdc)DatabricksDelta LakeKafkaParquetPysparkPythonSpark Structured StreamingSQL
Digital Media • Information Technology • News + Entertainment
Design, build, and maintain large-scale data pipelines and lake architectures on AWS using Spark/Databricks. Ensure data quality, lineage, observability, and TDD-based testing. Support self-service analytics, semantic layers, and cross-functional stakeholders while providing production support and documentation.
Top Skills:
Apache IcebergSparkAWSDatabricksDatabricks Delta Live Tables (Dlt)Delta LakeKubernetesLookerPysparkPythonSnowflakeSQLTabTeradata
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Design, build, and maintain scalable data architecture and pipelines to support medical analytics and predictive models. Partner with product and data leads, prepare datasets, automate data delivery, mentor colleagues on standards, and evaluate tools to improve analytics productivity and stability.
Top Skills:
Adobe AnalyticsAdobe LaunchApache AirflowBrighttagCharles ProxyCSSData WarehousingDatabricksDataiku (Dss Data Science Studio)DrupalDynamic Tag ManagerEnsightenGoogle AnalyticsGoogle Tag ManagerHTMLInfrastructure As CodeJavaScriptJqueryKafkaMatomo/PiwikOmniture SitecatalystPiwik ProSignalTealium Iq
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
