We are looking for a Databricks Platform Manager/Engineer responsible for designing, administering, securing, and optimizing the enterprise Databricks platform on AWS. The ideal candidate will own the platform lifecycle, ensuring scalability, security, governance, cost optimization, and operational excellence while enabling data engineering, analytics, and AI teams to work efficiently.
This role focuses on platform engineering rather than data engineering or analytics development.
ResponsibilitiesPlatform Administration
Design, deploy, and manage Databricks workspaces across Development, Test, UAT, and Production environments.
Establish workspace standards, networking, and environment isolation.
Manage platform upgrades, including Databricks Runtime (DBR) version planning, testing, and rollout.
Maintain platform availability, reliability, and performance.
Compute & Cluster Management
Create and manage cluster policies to enforce organizational standards.
Configure instance pools and job clusters.
Prevent oversized or non-compliant clusters.
Standardize cluster configurations across teams.
Optimize cluster startup times and resource utilization.
Security & Identity Management
Integrate Databricks with AWS IAM.
Configure and manage authentication and authorization.
Manage service principals, personal access tokens (PATs), and secret scopes.
Implement least-privilege access controls.
Support enterprise identity management and SSO.
Governance & Data Access
Administer Unity Catalog.
Manage catalogs, schemas, tables, and permissions.
Implement role-based access control (RBAC).
Enforce data governance and access policies.
Support audit and compliance requirements.
DevOps & CI/CD
Configure Git integration with Databricks Repos.
Support CI/CD pipelines for notebooks, workflows, and infrastructure.
Collaborate with DevOps teams on automated deployments.
Enable Infrastructure as Code (IaC) using Terraform where applicable.
Monitoring & Operations
Monitor workspace health, jobs, clusters, and platform performance.
Respond to incidents and perform root cause analysis.
Maintain operational dashboards and alerts.
Support production releases and platform maintenance activities.
Cost Optimization
Monitor Databricks usage and cloud spending.
Recommend cluster sizing and auto-scaling strategies.
Optimize job scheduling and compute utilization.
Track cost trends and recommend savings opportunities.
Compliance & Audit
Configure audit logging.
Ensure platform complies with enterprise security policies.
Support internal and external audits.
Maintain operational documentation and platform standards.
QualificationsRequired Skills
Databricks Platform
Workspace administration
Cluster management
Cluster policies
Instance pools
Job clusters
Databricks Runtime management
Workflows
Databricks Repos
Unity Catalog
Secret Scopes
Service Principals
Personal Access Tokens (PATs)
AWS
IAM
VPC fundamentals
EC2 concepts
S3
CloudWatch
KMS
Security Groups
Networking fundamentals
DevOps
Git
CI/CD
Terraform
Infrastructure as Code
Azure DevOps, GitHub Actions, or Jenkins
Monitoring
Platform monitoring
Incident management
Log analysis
Performance tuning
Cost optimization
Required Experience
Candidates should have hands-on experience with:
Managing enterprise Databricks platforms
Setting up multiple Databricks workspaces
Configuring Unity Catalog
AWS IAM integration
Cluster policy creation and governance
Runtime version upgrades
CI/CD implementation
Git integration
Platform monitoring and troubleshooting
Secrets management
Audit logging
Production support
Cost optimization initiatives
Preferred Qualifications
Experience managing large-scale Databricks environments (100+ users).
Experience supporting multiple business units or enterprise data platforms.
Hands-on experience with Terraform for Databricks infrastructure provisioning.
Knowledge of Lakehouse architecture and Delta Lake.
Experience with FinOps practices for cloud cost optimization.
Familiarity with data governance and security frameworks.
Nice to Have
Experience with AWS Organizations and Control Tower.
Knowledge of networking concepts for private connectivity (PrivateLink, VPC endpoints).
Experience with monitoring tools such as CloudWatch, Datadog, or Splunk.
Exposure to Apache Spark internals.
Experience supporting AI/ML workloads on Databricks.
Education
Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
Certifications (Preferred)
Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).
AWS Certified Solutions Architect – Associate/Professional.
AWS Certified SysOps Administrator.
Terraform Associate certification.
Ideal Candidate Profile
EXL New York, New York, USA Office
320 Park Avenue, 29th Floor, New York, NY, United States, 10022
EXL Jersey City, New Jersey, USA Office
Jersey City, United States, 0
EXL Newark, New Jersey, USA Office
Newark, United States
Similar Jobs
What you need to know about the NYC Tech Scene
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
