Build scalable ETL/ELT pipelines in Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, Workflows, and Unity Catalog. Ingest batch and streaming data across medallion architecture layers, develop transformations and data quality rules, and implement orchestration, monitoring, alerting, and automation. The role requires cloud data engineering, data modeling, performance tuning, CI/CD, and strong knowledge of lakehouse architecture, governance, and distributed systems.
DATAECONOMY is one of the fastest-growing Data & Analytics company with global presence. We are well-differentiated and are known for our Thought leadership, out-of-the-box products, cutting-edge solutions, accelerators, innovative use cases, and cost-effective service offerings.
We offer products and solutions in Cloud, Data Engineering, Data Governance, AI/ML, DevOps and Blockchain to large corporates across the globe. Strategic Partners with AWS, Collibra, cloudera, neo4j, DataRobot, Global IDs, tableau, MuleSoft and Talend.
Senior/Lead Data Engineer — Databricks
Rutherford, NJ/ Jersey City, NJ
Full-time
- Build scalable, production-grade ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Live Tables, Workflows).
- Ingest structured, semi-structured, and streaming data into Bronze, Silver, and Gold layers.
- Develop optimized transformations, data quality rules, and reusable framework components.
- Implement best practices for job orchestration, monitoring, alerting, and automation.
- Hands-on experience: Spark, Delta Lake, Workflows, Unity Catalog.
- Strong SQL programming and performance tuning skills.
- Experience with cloud environments (AWS/Azure/GCP).
- Experience with modern data lakehouse concepts and distributed systems.
- Strong understanding of Lakeflow Connect, LSDP/Lakehouse, Medallion Architecture, Data Validations, Genie, and Agent Bricks/RAG use cases.
- Should be able to explain these concepts using real project examples and architecture decisions.
- Knowledge of medallion architecture, DLT and unity catalog within Databricks.
Requirements
- Strong Python (PySpark) and SQL programming
- Databricks — Spark, Delta Lake, Workflows, Unity Catalog
- ETL/ELT pipeline development — Medallion Architecture (Bronze/Silver/Gold)
- Delta Live Tables, Auto-Loader, Structured Streaming
- Data modeling — dimensional (star/snowflake), normalization/denormalization
- CI/CD, Git, job orchestration
- Cloud experience — AWS, Azure, or GCP
- 7–10+ years in data engineering
- Knowledge of medallion architecture, DLT and unity catalog within Databricks.
Nice-to-Have Skills
- Lakeflow Connect, LSDP/Lakehouse, Genie, Agent Bricks/RAG use cases
- Data governance, metadata management, Unity Catalog advanced features
- Airflow, dbt, or similar orchestration tools
- Data security, compliance, and access models
- Cost optimization and performance tuning in cloud environments
- Corporate/enterprise data warehousing background
Similar Jobs
Aerospace • Information Technology • Security • Cybersecurity • Defense
Lead database engineering for a federal species-data integration program in a hybrid AWS cloud environment. Responsibilities include harmonizing schemas, documenting source-to-target mappings and lineage, enforcing metadata and data-quality standards, implementing encryption and NIST-aligned security controls, building authoritative source registries, optimizing performance, and leading production migration, validation, monitoring, and operational documentation. The role requires Databricks or equivalent lakehouse expertise, ETL programming, federal governance knowledge, and database migration leadership.
Top Skills:
SparkAws GovcloudDatabricksDelta LakePythonScala
Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software
Lead operational reliability and platform enablement for Databricks: build monitoring, CI/CD, deployment standards, compute and job policies, observability, runbooks, and governance to support secure, cost-aware, production data workloads across regulated environments. Mentor engineers and align platform with cloud/infrastructure and compliance requirements.
Top Skills:
Ci/CdDatabricksDatabricks Asset BundlesDatabricks WorkflowsDelta LakeInfrastructure-As-CodeService PrincipalsUnity CatalogVersion Control (Git)
Software
Design and build modern GCP lakehouse architectures, including Python and Spark pipelines, BigQuery ingestion, Kafka CDC, Delta Lake and Iceberg UniForm integrations, Delta Sharing endpoints, Snowflake and Databricks connectivity, data lineage, governed access controls, and semantic-layer alignment. Collaborate cross-functionally, troubleshoot full-stack issues, and deliver reusable data-sharing adapters and end-to-end data features.
Top Skills:
Apache IcebergSparkBigQueryChange Data Capture (Cdc)DatabricksDatabricks Unity CatalogDelta LakeDelta SharingGCPGoogle Cloud StorageIceberg UniformKafkaLookmlPythonSnowflakeSnowflake Horizon Catalog
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
