Prove Logo

Prove

Lead Data Engineer

Posted Yesterday
Be an Early Applicant
In-Office
New York, NY, USA
175K-200K Annually
Senior level
In-Office
New York, NY, USA
175K-200K Annually
Senior level
Lead the architecture, development, and maintenance of high-throughput data pipelines supporting real-time and batch inference. Translate complex domain logic into modular DAGs, establish data governance and ontology standards, optimize distributed processing for performance and cost, and collaborate with data science, informatics, and product engineering teams. Champion engineering quality through code reviews, testing, architectural discussions, and CI/CD practices.
The summary above was generated by AI
About Prove 

As the world moves to a mobile-first economy, businesses need to modernize how they acquire, engage with and enable consumers. Prove’s phone-centric identity tokenization and passive cryptographic authentication solutions reduce friction, enhance security and privacy across all digital channels, and accelerate revenues while reducing operating expenses and fraud losses. Over 1,000 enterprise customers use Prove’s platform to process 20 billion customer requests annually across industries, including banking, lending, healthcare, gaming, crypto, e-commerce, marketplaces, and payments. For the latest updates from Prove, follow us on LinkedIn.

Prove is driving the future of digital identity. We are looking for Provers who know how to make an impact. We’re talking self-starting professionals who thrive in a fast-paced environment, process information quickly, and make intelligent decisions. The work is challenging and requires not only smart but natural curiosity and tenacity. Teamwork is also important to us – we work together and play together.   

Prove has big plans, and we’re excited about the future. If this sounds like the place for you – come join our team! 

Title: Lead Data Engineer

Department: Data Science

Reports To: Director of Informatics

FLSA Status: Exempt

Location: Hybrid - New York, NY

Job Summary

As a Lead Data Engineer on Prove’s Data Science team, you will build the technical backbone that translates complex identity data into verified digital trust. Prove is fundamentally a data company: our identity solutions and analytical products are built on extracts spanning all tiers of data sophistication. In this role, every pipeline and architecture you design connects directly to revenue-generating products that secure transactions and prevent fraud for more than 1,000 global enterprise customers.

You will spearhead the development and maintenance of high-throughput data pipelines that power our core inference systems, taking raw intelligence and refining it into reliable scientific statements. Acting as a bridge across disciplines, you will translate multi-domain logic into resilient, highly performant directed acyclic graphs (DAGs), while proactively establishing data governance and ontology standards. This is an environment built for experimentation, offering you the autonomy to deploy modern paradigms and tools to solve intricate problems in identity resolution.

Key Responsibilities:

  • Architect, build, and maintain resilient data pipelines that power real-time and batch inference, moving data from raw intelligence to verifiable scientific statements.
  • Translate complex, multidisciplinary domain logic into efficient, maintainable, and modular directed acyclic graphs (DAGs) and execution workflows.
  • Proactively implement and advocate for robust data governance frameworks, metadata management, and semantic data ontologies.
  • Partner closely with data science, informatics, and product engineering teams to serve high-integrity data extracts that directly power revenue-generating products.
  • Continuously evaluate, benchmark, and adopt cutting-edge data engineering tools, frameworks, and storage paradigms.
  • Optimize distributed data processing pipelines for high availability, computational efficiency, and cost governance.
  • Champion engineering excellence across the team through code reviews, architectural discussions, automated testing, and CI/CD practices.
  • Promote, maintain and enhance our cultural values of humility, passion, inclusion, and leadership.
Qualifications and Experience:
  • Strong passion for learning about our products and markets through in-house and external training.
  • 7+ years of professional experience in data engineering or developing complex, data-driven software products.
  • Deep, first-principles understanding of data systems, distributed compute engines, data modeling, and modern pipeline architecture.
  • Demonstrated track record of collaborating across technical and scientific functions, with the ability to translate cross-disciplinary logic into concrete data systems.
  • High intellectual curiosity and a proactive mindset toward experimentation and technological innovation.
  • Hands-on experience with modern data orchestration (e.g., Airflow, Dagster, Prefect) and large-scale distributed frameworks.

Preferred Qualifications:

  • Advanced degree (Master’s or Ph.D.) in Computer Science, Mathematics, Statistics, Physics, or a related quantitative field.
  • Experience architecting creative solutions for identity resolution, graph data structures, or fraud prevention systems.
  • Fluent and terminal-native in Linux/Unix environments, command-line utilities, and shell scripting.
  • Experience in high-growth/pre-IPO Technology companies.

The anticipated salary pay range for Metro 2 areas for this role is $175,000 - $200,000 plus company bonus.

Prove follows a market driven compensation philosophy based on geographic location and respective market rates. Job offers will be aligned to location. Please speak with your recruiter if you have questions. Prove defines:

  • Metro 2 - NYC metro area, Seattle metro area, Los Angeles metro area, and the Miami metro area.
  • Metro 3 - all other cities across the domestic United States, with the exception of the San Francisco Bay Area.
Benefits & Perks for FTE Provers:
  • Competitive salaries & Bonus Plan (for eligible roles) and Equity Plan
  • Modern Health for financial, mental, and physical wellness
  • 401(k) Retirement Plan & Match (US Offices) and Local Country Pension (International Offices)
  • Unlimited Vacation and Flexible hours
  • Comprehensive medical benefits for you and your family ❤️
  • Emotional & Physical Wellness – Access to wellness services (EAP & Prove Well-Being Reimbursement)
  • Bottomless snacks & beverages for certain office locations
  • Daily GrubHub stipend for lunch if coming into the office (US Offices)
  • A great place to work and connect with other talented Provers like yourself!

Don’t meet every single requirement? Studies have shown that women and people of color are less likely to apply to jobs unless they meet every single qualification. At Prove we are dedicated to building a diverse, inclusive and authentic workplace, so if you’re excited about this role but your past experience doesn’t align perfectly with every qualification in the job description, we encourage you to apply anyways. You may be just the right candidate for this or other roles.

Equal Opportunity Employment:
Prove is an equal opportunity employer committed to providing equal employment opportunity for all people regardless of race, color, religion, gender or sexual orientation, age, marital status, national origin, citizenship status, disability, veteran status or other personal characteristics 

Privacy & Data Protection:
When you are applying for a job at Prove, we collect and use your personal information in the job application process. To understand more about how Prove uses your personal information, please see our Recruitment Privacy Policy on our website.

HQ

Prove New York, New York, USA Office

245 5th Avenue, New York, NY, United States, 10016

Similar Jobs

6 Days Ago
Hybrid
New York, NY, USA
112K-200K Annually
Senior level
112K-200K Annually
Senior level
Financial Services
Lead the design, development, and support of scalable enterprise data pipelines and platforms. Build Snowflake-based solutions, Python components, and Airflow workflows while ensuring data quality, governance, security, monitoring, performance, and reliability. Partner with business and technology stakeholders on architecture and modernization, troubleshoot production issues, conduct code reviews, and provide technical guidance to data engineering teams in an Agile enterprise environment.
Top Skills: Apache AirflowSparkAWSAzure Data FactoryDatabricksGCPAzurePythonSnowflakeSQL
12 Days Ago
In-Office
116K-194K Annually
Senior level
116K-194K Annually
Senior level
Fintech
Leads the design, development, deployment, and support of complex data pipelines and data products across cloud, warehouse, and lakehouse platforms. Builds scalable batch and streaming solutions using Snowflake, Databricks, SQL, Python, dbt, Spark, and related technologies. Ensures data quality, security, observability, reliability, and cost efficiency while troubleshooting production issues. Provides technical leadership, mentors engineers, establishes reusable standards, and collaborates with architecture, governance, cybersecurity, product, and platform teams.
Top Skills: Apache AirflowApache IcebergApache KafkaSparkAvroAzure Data FactoryCi/CdCsvDatabricksDatabricks WorkflowsDbtGitJSONAzureParquetPysparkPythonSnowflakeSQLTerraformXML
15 Days Ago
In-Office or Remote
40 Locations
105K-244K Annually
Senior level
105K-244K Annually
Senior level
Aerospace • Security • Cybersecurity • Industrial
Leads the design, development, governance, and productization of enterprise data models, pipelines, data products, and ontologies. Builds machine-learning-ready datasets, implements security and lineage controls, optimizes Databricks and Delta Lake environments, establishes data-product standards, collaborates with architects and data scientists, and mentors engineering talent. The role also defines product roadmaps, service levels, usage metrics, and continuous-improvement processes for enterprise data assets supporting AI agents, analytics, and applications.
Top Skills: Apache AirflowApache NifiSparkCloud IamDatabricksDelta LakeFeature StoresGenerative AiMachine Learning PipelinesPysparkPythonScalaSQLUnity Catalog

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account