Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Job Description :
Location - US (remote) . East coast preferable
Primary Skill :
Technical Lead -Data Engneering
- Azure databricks- Build Data Pipelines using ADB-Leveraging Python and Spark
- Build Data Transformation in ADB
- Convert Scala Pgms to Py Spark Pgms
- Understand the SCALA program in DataBricks
- ADF
- Secondary Skills:
- Azure data lake gen 2
- Synapse analytics (dedicated SQL pool)
- Exp in delta Lake
- Pyspark
Nice to Have : Tidal Scheduler , Exp in Healthcare Domain
Hexaware Technologies Iselin, New Jersey, USA Office
101 Wood Avenue South Suite, Iselin, United States, 08830
Hexaware Technologies Jamesburg, New Jersey, USA Office
1095 Cranbury S Rvr Rd St, Jamesburg, New Jersey, 08831-3411, United States, Jamesburg, United States, 08831-3411
Hexaware Technologies Jersey City, New Jersey, USA Office
Jersey City, United States
Hexaware Technologies New York, New York, USA Office
New York, United States, 0
Hexaware Technologies Secaucus, New Jersey, USA Office
Secaucus, United States
Similar Jobs
Information Technology • Consulting
Lead data engineer responsible for designing, building, and operating production data pipelines using Python, Airflow, Snowflake, DBT, and AWS. Automate ETL, implement CI/CD, testing and DBT validations, troubleshoot pipelines, and work client-facing to deliver reliable data solutions.
Top Skills:
Ai IdeApache AirflowAws EcsAws IamAws S3Ci/CdCsvDbtGitJinjaJSONPythonSnowflakeSQLXML
Information Technology • Consulting
Design, build, and optimize end-to-end ETL/ELT pipelines using Python and PySpark for large structured and unstructured datasets. Ensure data quality through validation and reconciliation, troubleshoot performance issues, implement CI/CD and code review practices, and collaborate with IT and business stakeholders to translate requirements into scalable data solutions.
Top Skills:
AWSAzureBitbucketCi/CdData WarehousingDistributed ComputingEtl/EltGCPGitHadoopMicrosoft Sql ServerPysparkPythonSparkSQL
Cloud • Consulting
Remote Big Data Tech Lead supporting AWS data analytics projects: assist technical supervision, perform code reviews, estimate scope/LOE, design and prioritize data solutions, manage technical delivery, and clarify business context to maximize impact.
Top Skills:
Amazon RedshiftSparkAws AthenaAws EmrAws GlueHadoopJavaPythonScalaSQL
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
