Design, build, and optimize end-to-end ETL/ELT pipelines using Python and PySpark for large structured and unstructured datasets. Ensure data quality through validation and reconciliation, troubleshoot performance issues, implement CI/CD and code review practices, and collaborate with IT and business stakeholders to translate requirements into scalable data solutions.
Responsibilities: • Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. • Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement. • Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle. • Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions. • Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability. • Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment. Skills: • Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred) • Experience with big data frameworks (Hadoop, Spark). • Experience with cloud platforms ( AWS, Azure or GCP) • Experience with Code versioning tools ( Bitbucket, Github ) • Experience with CI/CD and setting up pipelines. • Solid understanding of database design principles, data modelling, schemas and data warehousing solutions. • Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.
Hexaware Technologies Iselin, New Jersey, USA Office
101 Wood Avenue South Suite, Iselin, United States, 08830
Hexaware Technologies Jamesburg, New Jersey, USA Office
1095 Cranbury S Rvr Rd St, Jamesburg, New Jersey, 08831-3411, United States, Jamesburg, United States, 08831-3411
Hexaware Technologies Jersey City, New Jersey, USA Office
Jersey City, United States
Hexaware Technologies New York, New York, USA Office
New York, United States, 0
Hexaware Technologies Secaucus, New Jersey, USA Office
Secaucus, United States
Similar Jobs
Information Technology • Consulting
Design and maintain conceptual, logical, and physical data models for scalable data architectures. Implement Data Vault structures, including Hubs, Links, and Satellites, across enterprise data lakes and lakehouses. Deploy models using Azure data services, optimize performance, document data flows, and collaborate with engineers, scientists, analysts, and stakeholders. Apply data modeling best practices, integration techniques, security standards, and compliance requirements.
Top Skills:
AzureAzure Data LakeAzure Sql DatabaseAzure Synapse AnalyticsData VaultDatabricksEr/StudioErwinPower Designer
Information Technology • Software • Analytics • Business Intelligence
Leads teams developing and maintaining complex cloud-based big data platforms and pipelines. Owns architecture, delivery strategy, data modeling, governance, observability, reliability, security, and cost optimization. Translates business requirements into technical plans, oversees coding and production support, manages risks and client communication, establishes quality and CI/CD practices, and mentors engineers while providing cross-team architectural governance.
Top Skills:
AlertingCi/CdCloud ComputingData CatalogingData GovernanceData LineageData ModelingEvent-Driven PipelinesGCPGitKappa ArchitectureLambda ArchitectureLoggingMicroservicesMonitoringObservabilityPythonScrumSQL
Information Technology • Consulting
Leads hands-on big data processing, data engineering, and analytics workflows using Azure Databricks, Python, and SQL. Develops insights and reporting with Power BI, Tableau, and Excel. Analyzes complex datasets, communicates findings to business stakeholders, manages competing priorities, meets deadlines, and promotes continuous improvement.
Top Skills:
Azure DatabricksExcelPower BIPythonSQLTableau
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
