Designs, develops, and optimizes scalable data platforms, pipelines, ETL/ELT workflows, and Lakehouse architectures using Azure Databricks, Spark, and cloud services. Builds data models, quality and governance frameworks, integrations, and secure access controls. Optimizes workloads, supports streaming and analytics initiatives, troubleshoots pipelines, and contributes to CI/CD, DevOps, infrastructure automation, documentation, and technical leadership.
Job Summary
We are seeking a skilled Databricks Engineer with minimum of 5+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.
Job Description: Azure / Databricks Data Engineer (9-15 Years Experience)
Senior Azure Databricks Data Engineer
Experience
9-15 Years of IT Experience
Location
Hybrid
Job Summary
We are seeking a skilled Databricks Engineer with minimum of 5+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.
Key Responsibilities
- Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.
- Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.
- Develop data models and implement data quality, validation, and governance frameworks.
- Integrate data from multiple sources into a unified Lakehouse architecture.
- Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
- Implement security controls, access management, and data governance using Unity Catalog.
- Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.
- Monitor, troubleshoot, and resolve data pipeline issues.
- Support CI/CD, DevOps, and infrastructure automation practices.
- Maintain technical documentation and best practices.
Required Technical Skills Core Technologies
- Databricks Lakehouse Platform
- Apache Spark / PySpark
- Delta Lake
- SQL
- Python
Data Engineering
- ETL / ELT Development
- Data Modeling
- Data Warehousing
- Data Quality & Validation
- Streaming & Real-Time Processing
Governance & Security
- Unity Catalog
- Data Lineage
- Row-Level Security
- Access Control & Compliance
- Data Governance Frameworks
Cloud & DevOps
- Azure / AWS / GCP
- Terraform
- GitHub Actions / Azure DevOps
- CI/CD Pipelines
Analytics & AI
- Semantic Layers
- Data Products
- BI Platforms
- Machine Learning Support
- Generative AI & RAG Architectures
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.
- 9-15 years of experience in Data Engineering and Data Warehousing.
- Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.
- Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
- Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
- Experience in handling large-scale structured and unstructured datasets.
- Strong analytical, problem-solving, and troubleshooting skills.
Preferred Qualifications
- Microsoft Certified: Azure Data Engineer Associate (DP-203).
- Databricks Certified Data Engineer Associate/Professional.
- Experience with Snowflake, Power BI, or Microsoft Fabric.
- Experience in real-time streaming solutions using Kafka/Event Hubs.
- Exposure to Data Governance and Master Data Management initiatives.
Soft Skills
- Strong stakeholder management and communication skills.
- Ability to lead technical initiatives and drive architecture discussions.
- Experience working in Agile/Scrum environments.
- Excellent documentation and presentation skills.
- Strong mentoring and team leadership abilities.
Nice to Have
- Microsoft Fabric
- Power BI
- Azure Event Hubs
- Kafka
- Machine Learning data pipelines
- Data Governance tools such as Purview
ResponsibilitiesSupport critical and nonproduction databases in multitenant, highstress environments.
Support applications Installation, Configuration, Management, and Monitoring of databases in a SOX and PHI compliant environment
Installation, configuration, and integration of thirdparty applications and tools
Development of procedures for automated monitoring and proactive intervention, preventing customer impact QualificationsBachelors Degree/ Post Bachelor Degree 8 - 10+ Years
EXL New York, New York, USA Office
320 Park Avenue, 29th Floor, New York, NY, United States, 10022
EXL Jersey City, New Jersey, USA Office
Jersey City, United States, 0
EXL Newark, New Jersey, USA Office
Newark, United States
Similar Jobs
Professional Services • Real Estate • Consulting • Industrial
Design and build SOCOTEC's enterprise data platform: scalable Databricks/Spark/Delta Lake pipelines, data models, and MDM. Ingest from ERP/CRM, enable batch and near-real-time processing, ensure data governance, quality, lineage, and provide datasets for analytics and AI while driving architecture and observability.
Top Skills:
SparkAzureDatabricksDelta LakePythonSQL
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Own outbound product strategy for ServiceNow’s Workflow Data Fabric and RaptorDB Professional. Translate complex data-platform capabilities into customer value, positioning, messaging, go-to-market strategies, enablement materials, demos, and executive narratives. Partner with product, engineering, marketing, sales, solution consulting, and customer teams to drive launches, adoption, field readiness, competitive differentiation, and pipeline influence. Represent products in customer briefings, analyst sessions, webinars, conferences, and strategic deals while turning market feedback into roadmap and growth recommendations.
Top Skills:
Ai/Ml Data LayersAnalytical ProcessingAnalyticsCloud InfrastructureData PlatformsDatabasesDistributed SystemsEnterprise SoftwareIntegrations/IpaasPlatform ArchitectureRaptordbServicenowTransactional ProcessingWorkflow Data Fabric
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Leads payer back-office strategy, technology, and operations consulting engagements for health services clients. Responsibilities include modernizing payer operations, optimizing processes, analyzing business data, managing change, improving service delivery models, coaching teams, engaging stakeholders, and validating quality and compliance outcomes. The role leads large projects and develops strategic recommendations for operational excellence.
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
