EXL Logo

EXL

Databricks Platform Engineer

Posted 6 Days Ago
Be an Early Applicant
Hybrid
New York, NY, USA
Expert/Leader
Hybrid
New York, NY, USA
Expert/Leader
Owns the enterprise Databricks platform on AWS, including workspace architecture, administration, security, identity, governance, cluster management, upgrades, monitoring, troubleshooting, CI/CD automation, Terraform infrastructure as code, compliance, and cost optimization. Partners with data engineering, analytics, DevOps, and security teams to maintain scalable, reliable, secure, and high-performing production and nonproduction environments.
The summary above was generated by AI

Databricks Platform Manager / Engineer (AWS)


Experience

8–10 years of overall IT experience, including 5+ years of hands-on experience administering and managing Databricks platforms in production environments on AWS.


Job Summary

We are looking for a Databricks Platform Manager/Engineer responsible for designing, administering, securing, and optimizing the enterprise Databricks platform on AWS. The ideal candidate will own the platform lifecycle, ensuring scalability, security, governance, cost optimization, and operational excellence while enabling data engineering, analytics, and AI teams to work efficiently.


This role focuses on platform engineering rather than data engineering or analytics development.

Key Responsibilities

Platform Administration

  • Strong hands-on experience in System Design and Architecture, including making architecture decisions.
  • Strong AWS and DevOps experience.
  • Hands-on experience with Terraform and CI/CD pipelines.
  • Strong understanding of Cloud Networking, Security, Scalability, and High Availability (HA).
  • Good production troubleshooting and problem-solving skills.
  • Design, deploy, and manage Databricks workspaces across Development, Test, UAT, and Production environments.
  • Establish workspace standards, networking, and environment isolation.
  • Manage platform upgrades, including Databricks Runtime (DBR) version planning, testing, and rollout.
  • Maintain platform availability, reliability, and performance.
  • Compute & Cluster Management
  • Create and manage cluster policies to enforce organizational standards.
  • Configure instance pools and job clusters.
  • Prevent oversized or non-compliant clusters.
  • Standardize cluster configurations across teams.
  • Optimize cluster startup times and resource utilization.
  • Security & Identity Management
  • Integrate Databricks with AWS IAM.
  • Configure and manage authentication and authorization.
  • Manage service principles, personal access tokens (PATs), and secret scopes.
  • Implement least-privilege access controls.
  • Support enterprise identity management and SSO.
  • Governance & Data Access
  • Administer Unity Catalog.
  • Manage catalogs, schemas, tables, and permissions.
  • Implement role-based access control (RBAC).
  • Enforce data governance and access policies.
  • Support audit and compliance requirements.
  • DevOps & CI/CD
  • Configure Git integration with Databricks Repos.
  • Support CI/CD pipelines for notebooks, workflows, and infrastructure.
  • Collaborate with DevOps teams on automated deployments.
  • Enable Infrastructure as Code (IaC) using Terraform where applicable.

Monitoring & Operations

  • Monitor workspace health, jobs, clusters, and platform performance.
  • Respond to incidents and perform root cause analysis.
  • Maintain operational dashboards and alerts.
  • Support production releases and platform maintenance activities.
  • Cost Optimization
  • Monitor Databricks usage and cloud spending.
  • Recommend cluster sizing and auto-scaling strategies.
  • Optimize job scheduling and compute utilization.
  • Track cost trends and recommend savings opportunities.
  • Compliance & Audit
  • Configure audit logging.
  • Ensure platform complies with enterprise security policies.
  • Support internal and external audits.
  • Maintain operational documentation and platform standards

Required Skills

  • Databricks Platform
  • Workspace administration
  • Cluster management
  • Cluster policies
  • Instance pools
  • Job clusters
  • Databricks Runtime management
  • Workflows
  • Databricks Repos
  • Unity Catalog
  • Secret Scopes
  • Service Principals
  • Personal Access Tokens (PATs)
  • AWS
  • IAM
  • VPC fundamentals
  • EC2 concepts
  • S3
  • CloudWatch
  • KMS
  • Security Groups
  • Networking fundamentals
  • DevOps
  • Git
  • CI/CD
  • Terraform
  • Infrastructure as Code
  • Azure DevOps, GitHub Actions, or Jenkins
  • Monitoring
  • Platform monitoring
  • Incident management
  • Log analysis
  • Performance tuning
  • Cost optimization
  • Required Experience

Candidates should have hands-on experience with:

  • Managing enterprise Databricks platforms
  • Setting up multiple Databricks workspaces
  • Configuring Unity Catalog
  • AWS IAM integration
  • Cluster policy creation and governance
  • Runtime version upgrades
  • CI/CD implementation
  • Git integration
  • Platform monitoring and troubleshooting
  • Secrets management
  • Audit logging
  • Good Cloud, DevOps, and system Design Experience.
  • Hands-on terraform and CI/CD.
  • Strong understanding of cloud networking, security , scalability, and HA
  • Good production troubleshooting skills.
  • Production support
  • Cost optimization initiatives
  • Preferred Qualifications
  • Experience managing large-scale Databricks environments (100+ users).
  • Experience supporting multiple business units or enterprise data platforms.
  • Hands-on experience with Terraform for Databricks infrastructure provisioning.
  • Knowledge of Lakehouse architecture and Delta Lake.
  • Experience with FinOps practices for cloud cost optimization.
  • Familiarity with data governance and security frameworks.

Nice to Have

  • Experience with AWS Organizations and Control Tower.
  • Knowledge of networking concepts for private connectivity (PrivateLink, VPC endpoints).
  • Experience with monitoring tools such as CloudWatch, Datadog, or Splunk.
  • Exposure to Apache Spark internals.
  • Experience supporting AI/ML workloads on Databricks.

Education

  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field. 
  • Certifications (Preferred)
  • Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).
  • AWS Certified Solutions Architect – Associate/Professional.
  • AWS Certified SysOps Administrator.
  • Terraform Associate certification.
  • Ideal Candidate Profile

The ideal candidate has a strong platform engineering and cloud infrastructure background with deep expertise in Databricks administration on AWS. They are comfortable owning the operational health, security, governance, automation, and cost optimization of the Databricks platform while partnering closely with data engineering, analytics, DevOps, and security teams. Experience with Terraform is highly desirable, as many enterprise Databricks environments are provisioned and managed through Infrastructure as Code.

ResponsibilitiesSupport critical and nonproduction databases in multitenant, highstress environments.
Support applications Installation, Configuration, Management, and Monitoring of databases in a SOX and PHI compliant environment
Installation, configuration, and integration of thirdparty applications and tools
Development of procedures for automated monitoring and proactive intervention, preventing customer impact QualificationsBachelors Degree/ Post Bachelor Degree 8 - 10+ Years
HQ

EXL New York, New York, USA Office

320 Park Avenue, 29th Floor, New York, NY, United States, 10022

EXL Jersey City, New Jersey, USA Office

Jersey City, United States, 0

EXL Newark, New Jersey, USA Office

Newark, United States

Similar Jobs

24 Days Ago
In-Office or Remote
2 Locations
102K-204K Annually
Senior level
102K-204K Annually
Senior level
Fitness • Healthtech • Retail • Pharmaceutical
Build and operate an Azure Databricks data platform using Python frameworks, configuration-driven pipelines, CI/CD, Terraform, and infrastructure as code. Support production reliability through monitoring, incident response, root cause analysis, runbooks, disaster recovery, and automation. Maintain data security and governance with Unity Catalog, enable domain teams to onboard data sources, and improve self-service platform capabilities in a regulated healthcare environment.
Top Skills: Adls Gen2SparkAzureAzure Key VaultCi/CdDatabricksDatabricks Asset BundlesDelta LakeDelta Live TablesGitGithub ActionsInfrastructure As CodeLakeflowMicrosoft Entra IdPysparkPytestPythonSQLTerraformUnity Catalog
One Month Ago
In-Office
New York, NY, USA
130K-180K Annually
Senior level
130K-180K Annually
Senior level
Artificial Intelligence • Software • Analytics • Financial Services
Build and operate Curinos’ Databricks-native AI and ML platform. Responsibilities include developing deployment tooling, CI/CD pipelines, infrastructure-as-code, observability, governance, model monitoring, LLM evaluation frameworks, and scalable inference systems. The role partners with data scientists and engineers to productionize ML models and AI applications while supporting reliability, incident response, reproducibility, auditability, and operational excellence in a regulated FinTech environment.
Top Skills: Agentic AiAPIsCi/CdDatabricksDatabricks Asset BundlesDatabricks WorkflowsDelta LakeDelta Live TablesInfrastructure As CodeLlmsMcp ServersMlflowMlopsModel ServingPysparkPythonSpark SqlUnity Catalog
An Hour Ago
In-Office
New York, NY, USA
157K-210K Annually
Senior level
157K-210K Annually
Senior level
Cloud • Information Technology • Machine Learning
Own and mature CoreWeave’s security risk management programs, including AI-assisted exception workflows, KRI dashboards, enterprise risk assessments, executive reporting, and M&A security integrations. Partner with Assurance Engineering and cross-functional stakeholders to define requirements, drive adoption, manage risk reviews, establish operating processes, resolve stalled decisions, and improve integration playbooks. The role requires strong security risk fluency, program ownership, judgment, and effective use of AI tools.
Top Skills: AIGrc ToolingKey Risk Indicator (Kri) Dashboards

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account