Senior Data Engineer — Databricks Lakehouse & Streaming Ingestion

• Location: Remote – Open to candidates across the MENA region with a preference of Palestine Based candidates.
  

• Role Type: Full-time, Remote Contractor
  

• Reports to: Data Engineering Lead / CTO 
  

• Timezone Overlap: Minimum 4 hours of daily overlap with US Mountain Time (MT).
  

Our client is looking for a Senior Data Engineer to join their team and help build and maintain the data infrastructure powering their healthcare AI platform. The company develops AI agents for specialty medical practices, including voice agents that interact with patients and document agents that process incoming medical documents.

In this role, you will be responsible for building reliable data ingestion and processing pipelines, working with Databricks, Spark, PySpark, Delta Lake, and streaming technologies. You'll also help ensure data accuracy, maintain data quality, and troubleshoot inconsistencies across multiple data sources.

This is a hands-on opportunity for an experienced Data Engineer who can independently own production data pipelines and build reliable data solutions. If your instinct when a dashboard number looks wrong is to trace it back to the raw event stream and prove where the discrepancy entered, this role is for you.

Our client is an early-stage healthcare technology company rethinking the way specialty medical practices operate. Their AI-powered tools help automate repetitive, time-consuming administrative tasks, allowing healthcare teams to spend less time on paperwork and more time focusing on patients.

They're a small, fast-moving team passionate about building intelligent solutions to real-world challenges in healthcare.

• Build and maintain resilient ingestion from third-party vendor REST APIs (voice-AI observability and telephony platforms), handling divergent pagination schemes (offset/limit vs. page/cursor), row and rate caps (time-windowing with adaptive bisection), and schema drift — with contract and column-presence checks that alert when a field is renamed or moved rather than silently propagating nulls.
  

• Own streaming ingestion from Azure Event Hubs into Databricks using Structured Streaming / Auto Loader — checkpoints and offset management, watermarking, at-least-once dedup — and run source-parity reconciliation (row counts, dropped/duplicated event detection, late-data and schema-mismatch root-causing) when ingested data doesn't match production.
  

• Develop and maintain Delta Lake pipelines in a medallion architecture (bronze → silver → gold) using Spark SQL and PySpark: idempotent MERGE upserts, Delta Live Tables and materialized-view constraints (CREATE OR REFRESH, live. references, DLT-vs-job execution contexts).
  

• Build entity resolution over dirty free-text — normalizing practice, provider, and payer names via regex, canonical dictionaries, fuzzy matching, confidence-scored crosswalks, and override tables.
  

• Maintain the semantic/metrics layer with rigor: precise metric definitions (denominators, grain, cohort boundaries), correct handling of non-additive aggregates (medians and percentiles that break aggregate-awareness), so every number is reproducible.
  

• Instrument data quality end-to-end: freshness, parity, contract checks, and alerting that catches problems before a dashboard does.