EXL Logo

EXL

AVP - Lead Databricks Data Engineer

Posted Yesterday
Be an Early Applicant
Hybrid
New York, NY, USA
Expert/Leader
Hybrid
New York, NY, USA
Expert/Leader
Leads the architecture, development, modernization, and optimization of enterprise Databricks Lakehouse platforms. Designs scalable batch and real-time data pipelines, implements Spark, PySpark, Delta Lake, Unity Catalog, governance, data quality, CI/CD, and cloud-native architectures across Azure, AWS, or GCP. Provides technical leadership, mentors engineering teams, drives modernization programs, manages delivery and stakeholders, and supports analytics, AI/ML, and business intelligence initiatives.
The summary above was generated by AI
Job Description: Lead Databricks Data Engineer

Job Title: Lead Databricks Data Engineer
Experience: 10-12 Years
Location: US
Employment Type: Full-Time

 

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

 

Key ResponsibilitiesData Engineering & Architecture
  • Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
  • Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
  • Define enterprise data models, ingestion frameworks, and transformation strategies.
  • Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.
  • Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.
  • Drive cloud data platform modernization and migration initiatives.
Technical Leadership
  • Serve as the technical lead for Databricks and Data Engineering programs.
  • Establish engineering best practices, coding standards, and architecture guidelines.
  • Conduct architecture reviews, code reviews, and solution assessments.
  • Mentor and guide Data Engineers, Technical Leads, and junior team members.
  • Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
Cloud & Platform Engineering
  • Design cloud-native data solutions on Azure, AWS, or GCP.
  • Implement secure, scalable, and resilient data architectures.
  • Optimize costs, performance, reliability, and availability of data platforms.
  • Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
  • Define and implement monitoring, observability, and operational excellence practices.
Governance & Data Quality
  • Establish data governance, lineage, cataloging, and security controls.
  • Ensure compliance with organizational and regulatory requirements.
  • Implement data quality frameworks and monitoring solutions.
  • Collaborate with security and compliance teams to enforce data protection standards.
Stakeholder & Delivery Management
  • Partner with business leaders to understand strategic data requirements.
  • Translate business requirements into scalable technical solutions.
  • Drive delivery planning, risk management, and resource coordination.
  • Present architecture decisions and technical recommendations to leadership teams.

 

Required Technical SkillsDatabricks
  • Databricks Lakehouse Platform
  • Databricks Workflows
  • Delta Lake
  • Unity Catalog
  • Structured Streaming
  • Databricks SQL
  • Delta Live Tables (DLT)
  • Databricks Asset Bundles
Big Data Technologies
  • Apache Spark
  • PySpark
  • Spark SQL
  • Kafka
  • Real-Time Streaming Architectures
  • Data Lake & Lakehouse Architectures
Cloud Platforms (Strong Experience in One or More)Azure
  • Azure Databricks
  • Azure Data Factory (ADF)
  • ADLS Gen2
  • Azure Synapse Analytics
  • Azure Key Vault
  • Event Hub
AWS
  • S3
  • EMR
  • Glue
  • Redshift
  • Lambda
  • IAM
GCP
  • BigQuery
  • Cloud Storage
  • Dataflow
  • Pub/Sub
  • Composer
Programming
  • Python
  • SQL
  • PySpark
  • Scala (Preferred)
  • Shell Scripting
DevOps & Automation
  • Git
  • Azure DevOps
  • GitHub Actions
  • Jenkins
  • Terraform
  • Docker
  • Kubernetes
Data Warehousing
  • Dimensional Modeling
  • Star & Snowflake Schema
  • Data Vault
  • Master Data Management
  • Data Governance Frameworks

 

Required Experience
  • 12-15 years of overall IT experience.
  • Minimum 6+ years of hands-on Databricks experience.
  • 7+ years of experience in Spark/PySpark-based data engineering.
  • Experience designing enterprise-scale Data Lake and Lakehouse solutions.
  • Strong experience leading large data transformation and modernization programs.
  • Experience handling structured, semi-structured, and unstructured data processing.
  • Experience supporting analytics, AI/ML, and reporting ecosystems.
  • Proven experience managing technical teams and driving architectural decisions.

 

Preferred Qualifications
  • Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
  • Databricks Certified Data Engineer Professional.
  • Azure Data Engineer Associate.
  • AWS Data Analytics Specialty.
  • Google Professional Data Engineer Certification.
  • TOGAF or Cloud Architecture certifications are a plus.

 

Nice to Have
  • MLOps and AI Engineering exposure.
  • Snowflake and Databricks integration experience.
  • Microsoft Fabric exposure.
  • Data Mesh implementation experience.
  • Data Governance tools such as Collibra, Alation, or Informatica.
  • Expertise in Data Quality frameworks such as Great Expectations or Deequ.

 

Key Competencies
  • Solution Architecture
  • Technical Leadership
  • Stakeholder Management
  • Program Delivery
  • Data Strategy & Governance
  • Team Mentoring & Coaching
  • Problem Solving & Decision Making
  • Communication & Executive Presentation Skills
  • Performance Optimization
  • Innovation & Continuous Improvement
Technology Classification

Technology Tower: Data & AI
Practice Area: Data Engineering
Primary Skill: Databricks Lakehouse Engineering
Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

ResponsibilitiesJob Description: Lead Databricks Data Engineer

Job Title: Lead Databricks Data Engineer
Experience: 10-12 Years
Location: US
Employment Type: Full-Time

 

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

 

Key ResponsibilitiesData Engineering & Architecture
  • Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
  • Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
  • Define enterprise data models, ingestion frameworks, and transformation strategies.
  • Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.
  • Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.
  • Drive cloud data platform modernization and migration initiatives.
Technical Leadership
  • Serve as the technical lead for Databricks and Data Engineering programs.
  • Establish engineering best practices, coding standards, and architecture guidelines.
  • Conduct architecture reviews, code reviews, and solution assessments.
  • Mentor and guide Data Engineers, Technical Leads, and junior team members.
  • Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
Cloud & Platform Engineering
  • Design cloud-native data solutions on Azure, AWS, or GCP.
  • Implement secure, scalable, and resilient data architectures.
  • Optimize costs, performance, reliability, and availability of data platforms.
  • Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
  • Define and implement monitoring, observability, and operational excellence practices.
Governance & Data Quality
  • Establish data governance, lineage, cataloging, and security controls.
  • Ensure compliance with organizational and regulatory requirements.
  • Implement data quality frameworks and monitoring solutions.
  • Collaborate with security and compliance teams to enforce data protection standards.
Stakeholder & Delivery Management
  • Partner with business leaders to understand strategic data requirements.
  • Translate business requirements into scalable technical solutions.
  • Drive delivery planning, risk management, and resource coordination.
  • Present architecture decisions and technical recommendations to leadership teams.

 

Required Technical SkillsDatabricks
  • Databricks Lakehouse Platform
  • Databricks Workflows
  • Delta Lake
  • Unity Catalog
  • Structured Streaming
  • Databricks SQL
  • Delta Live Tables (DLT)
  • Databricks Asset Bundles
Big Data Technologies
  • Apache Spark
  • PySpark
  • Spark SQL
  • Kafka
  • Real-Time Streaming Architectures
  • Data Lake & Lakehouse Architectures
Cloud Platforms (Strong Experience in One or More)Azure
  • Azure Databricks
  • Azure Data Factory (ADF)
  • ADLS Gen2
  • Azure Synapse Analytics
  • Azure Key Vault
  • Event Hub
AWS
  • S3
  • EMR
  • Glue
  • Redshift
  • Lambda
  • IAM
GCP
  • BigQuery
  • Cloud Storage
  • Dataflow
  • Pub/Sub
  • Composer
Programming
  • Python
  • SQL
  • PySpark
  • Scala (Preferred)
  • Shell Scripting
DevOps & Automation
  • Git
  • Azure DevOps
  • GitHub Actions
  • Jenkins
  • Terraform
  • Docker
  • Kubernetes
Data Warehousing
  • Dimensional Modeling
  • Star & Snowflake Schema
  • Data Vault
  • Master Data Management
  • Data Governance Frameworks

 

Required Experience
  • 12-15 years of overall IT experience.
  • Minimum 6+ years of hands-on Databricks experience.
  • 7+ years of experience in Spark/PySpark-based data engineering.
  • Experience designing enterprise-scale Data Lake and Lakehouse solutions.
  • Strong experience leading large data transformation and modernization programs.
  • Experience handling structured, semi-structured, and unstructured data processing.
  • Experience supporting analytics, AI/ML, and reporting ecosystems.
  • Proven experience managing technical teams and driving architectural decisions.

 

Preferred Qualifications
  • Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
  • Databricks Certified Data Engineer Professional.
  • Azure Data Engineer Associate.
  • AWS Data Analytics Specialty.
  • Google Professional Data Engineer Certification.
  • TOGAF or Cloud Architecture certifications are a plus.

 

Nice to Have
  • MLOps and AI Engineering exposure.
  • Snowflake and Databricks integration experience.
  • Microsoft Fabric exposure.
  • Data Mesh implementation experience.
  • Data Governance tools such as Collibra, Alation, or Informatica.
  • Expertise in Data Quality frameworks such as Great Expectations or Deequ.

 

Key Competencies
  • Solution Architecture
  • Technical Leadership
  • Stakeholder Management
  • Program Delivery
  • Data Strategy & Governance
  • Team Mentoring & Coaching
  • Problem Solving & Decision Making
  • Communication & Executive Presentation Skills
  • Performance Optimization
  • Innovation & Continuous Improvement
Technology Classification

Technology Tower: Data & AI
Practice Area: Data Engineering
Primary Skill: Databricks Lakehouse Engineering
Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

QualificationsJob Description: Lead Databricks Data Engineer

Job Title: Lead Databricks Data Engineer
Experience: 10-12 Years
Location: US
Employment Type: Full-Time

 

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

 

Key ResponsibilitiesData Engineering & Architecture
  • Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
  • Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
  • Define enterprise data models, ingestion frameworks, and transformation strategies.
  • Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.
  • Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.
  • Drive cloud data platform modernization and migration initiatives.
Technical Leadership
  • Serve as the technical lead for Databricks and Data Engineering programs.
  • Establish engineering best practices, coding standards, and architecture guidelines.
  • Conduct architecture reviews, code reviews, and solution assessments.
  • Mentor and guide Data Engineers, Technical Leads, and junior team members.
  • Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
Cloud & Platform Engineering
  • Design cloud-native data solutions on Azure, AWS, or GCP.
  • Implement secure, scalable, and resilient data architectures.
  • Optimize costs, performance, reliability, and availability of data platforms.
  • Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
  • Define and implement monitoring, observability, and operational excellence practices.
Governance & Data Quality
  • Establish data governance, lineage, cataloging, and security controls.
  • Ensure compliance with organizational and regulatory requirements.
  • Implement data quality frameworks and monitoring solutions.
  • Collaborate with security and compliance teams to enforce data protection standards.
Stakeholder & Delivery Management
  • Partner with business leaders to understand strategic data requirements.
  • Translate business requirements into scalable technical solutions.
  • Drive delivery planning, risk management, and resource coordination.
  • Present architecture decisions and technical recommendations to leadership teams.

 

Required Technical SkillsDatabricks
  • Databricks Lakehouse Platform
  • Databricks Workflows
  • Delta Lake
  • Unity Catalog
  • Structured Streaming
  • Databricks SQL
  • Delta Live Tables (DLT)
  • Databricks Asset Bundles
Big Data Technologies
  • Apache Spark
  • PySpark
  • Spark SQL
  • Kafka
  • Real-Time Streaming Architectures
  • Data Lake & Lakehouse Architectures
Cloud Platforms (Strong Experience in One or More)Azure
  • Azure Databricks
  • Azure Data Factory (ADF)
  • ADLS Gen2
  • Azure Synapse Analytics
  • Azure Key Vault
  • Event Hub
AWS
  • S3
  • EMR
  • Glue
  • Redshift
  • Lambda
  • IAM
GCP
  • BigQuery
  • Cloud Storage
  • Dataflow
  • Pub/Sub
  • Composer
Programming
  • Python
  • SQL
  • PySpark
  • Scala (Preferred)
  • Shell Scripting
DevOps & Automation
  • Git
  • Azure DevOps
  • GitHub Actions
  • Jenkins
  • Terraform
  • Docker
  • Kubernetes
Data Warehousing
  • Dimensional Modeling
  • Star & Snowflake Schema
  • Data Vault
  • Master Data Management
  • Data Governance Frameworks

 

Required Experience
  • 12-15 years of overall IT experience.
  • Minimum 6+ years of hands-on Databricks experience.
  • 7+ years of experience in Spark/PySpark-based data engineering.
  • Experience designing enterprise-scale Data Lake and Lakehouse solutions.
  • Strong experience leading large data transformation and modernization programs.
  • Experience handling structured, semi-structured, and unstructured data processing.
  • Experience supporting analytics, AI/ML, and reporting ecosystems.
  • Proven experience managing technical teams and driving architectural decisions.

 

Preferred Qualifications
  • Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
  • Databricks Certified Data Engineer Professional.
  • Azure Data Engineer Associate.
  • AWS Data Analytics Specialty.
  • Google Professional Data Engineer Certification.
  • TOGAF or Cloud Architecture certifications are a plus.

 

Nice to Have
  • MLOps and AI Engineering exposure.
  • Snowflake and Databricks integration experience.
  • Microsoft Fabric exposure.
  • Data Mesh implementation experience.
  • Data Governance tools such as Collibra, Alation, or Informatica.
  • Expertise in Data Quality frameworks such as Great Expectations or Deequ.

 

Key Competencies
  • Solution Architecture
  • Technical Leadership
  • Stakeholder Management
  • Program Delivery
  • Data Strategy & Governance
  • Team Mentoring & Coaching
  • Problem Solving & Decision Making
  • Communication & Executive Presentation Skills
  • Performance Optimization
  • Innovation & Continuous Improvement
Technology Classification

Technology Tower: Data & AI
Practice Area: Data Engineering
Primary Skill: Databricks Lakehouse Engineering
Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

HQ

EXL New York, New York, USA Office

320 Park Avenue, 29th Floor, New York, NY, United States, 10022

EXL Jersey City, New Jersey, USA Office

Jersey City, United States, 0

EXL Newark, New Jersey, USA Office

Newark, United States

Similar Jobs

An Hour Ago
In-Office
New York, NY, USA
157K-210K Annually
Senior level
157K-210K Annually
Senior level
Cloud • Information Technology • Machine Learning
Own and mature CoreWeave’s security risk management programs, including AI-assisted exception workflows, KRI dashboards, enterprise risk assessments, executive reporting, and M&A security integrations. Partner with Assurance Engineering and cross-functional stakeholders to define requirements, drive adoption, manage risk reviews, establish operating processes, resolve stalled decisions, and improve integration playbooks. The role requires strong security risk fluency, program ownership, judgment, and effective use of AI tools.
Top Skills: AIGrc ToolingKey Risk Indicator (Kri) Dashboards
An Hour Ago
In-Office
New York, NY, USA
157K-210K Annually
Senior level
157K-210K Annually
Senior level
Cloud • Information Technology • Machine Learning
Lead security program operations by coordinating operating cadences, collecting status and risks, maintaining Jira standards and OKR records, preparing executive and technical reporting, and tracking action items through completion. Partner with TPMs and security program owners to keep epics, dependencies, dashboards, workflows, and automation accurate and on schedule across multiple teams.
Top Skills: APIsApps ScriptGoogle SheetsJIRAJira AutomationJqlLlmsLookerPythonTableau
An Hour Ago
In-Office
New York, NY, USA
157K-210K Annually
Senior level
157K-210K Annually
Senior level
Cloud • Information Technology • Machine Learning
Lead the company-wide Production IAM program, deploying SPIFFE/SPIRE and related workload identity, authentication, PKI, certificate, and secrets-management services. Coordinate security, platform, and infrastructure engineering teams through architecture, rollout, adoption, operational handoff, and migration. Drive authenticated service-to-service communication across Kubernetes, bare-metal, and multi-region environments, establish onboarding and migration tooling, manage risks and dependencies, and report adoption and security metrics to leadership.
Top Skills: Aws Private CaCert-ManagerEnvoyHashicorp VaultIstioJwtKubernetesLinkerdMtlsOauth 2.0OidcPkiSpiffeSpireX.509

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account