This is a remote position.
KATBOTZ is seeking an experienced Databricks SME to support enterprise-level data engineering and analytics initiatives. The ideal candidate will have deep expertise in the Databricks ecosystem, big data technologies, cloud platforms, and modern data architecture. The candidate will work closely with cross-functional teams to design, optimize, and implement scalable data solutions for global clients.
Lead the design, development, and optimization of Databricks-based data platforms and pipelines.
Provide technical expertise on Databricks architecture, performance tuning, and best practices.
Build scalable ETL/ELT workflows using Apache Spark and Databricks.
Collaborate with business stakeholders, architects, and engineering teams to gather requirements and deliver solutions.
Implement Delta Lake, data governance, and data quality frameworks.
Support migration of legacy data platforms to Databricks.
Optimize data processing workflows for performance and cost efficiency.
Mentor development teams on Databricks, Spark, and cloud-native solutions.
Troubleshoot production issues and provide timely resolutions.
Ensure adherence to security, compliance, and coding standards.
10+ years of experience in Data Engineering and Big Data technologies.
Strong hands-on expertise in Databricks and Apache Spark.
Experience with Python, PySpark, SQL, and Scala.
Expertise in Delta Lake, Unity Catalog, and Databricks Workflows.
Strong understanding of data warehousing and modern lakehouse architecture.
Experience with cloud platforms such as Microsoft Azure, Amazon AWS, or Google GCP.
Experience integrating Databricks with BI and reporting tools.
Knowledge of CI/CD pipelines and DevOps practices.
Strong analytical and problem-solving skills.
Excellent communication and stakeholder management abilities.
Preferred Qualifications
Databricks Certified Professional certification.
Experience in real-time streaming using Kafka or Structured Streaming.
Exposure to AI/ML workflows within Databricks.
Prior experience working with global remote teams.
- Competitive compensation package
- Opportunities for professional development and career advancement.
- Flexible working conditions, with remote options available.
- Dynamic and supportive work environment.
KATBOTZ LLC is an Equal Opportunity Employer. We provide equal employment opportunities to all qualified individuals, regardless of race, religion, gender, gender identity, age, marital status, national origin, sexual orientation, citizenship status, veteran status, disability, or any other legally protected status. As an organization, we are unwavering in our commitment to maintaining a discrimination-free work environment, and fostering a culture of inclusivity, belonging and equal opportunity for all employees and applicants.
Similar Jobs
What you need to know about the NYC Tech Scene
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
