Software Engineer, Data Platform

About Middesk

Middesk makes it easier for businesses to work together. Since 2018, we’ve been transforming business identity verification, replacing slow, manual processes with seamless access to complete, up-to-date data. Our platform helps companies across industries confidently verify business identities, onboard customers faster, and reduce risk at every stage of the customer lifecycle.

Middesk came out of Y Combinator, is backed by Sequoia Capital and Accel Partners, and was recently named to Forbes Fintech 50 List and cited as an industry leader in business verification by digital identity strategy firm, Liminal.

"Velocity" is the rate at which we effect realized value for our customers, not the rate at which we ship code. We believe that great products result when technical excellence and craft is combined with deep understanding of our customers and the problems we can help them solve. We’re a humble, self-motivated team that seeks to ship early and ship often; we pride ourselves with being willing to roll up our sleeves to solve even the messiest problems our customers present to us.  Middesk Engineering is customer-first engineering.

Middesk is, at its core, a data company. We live and die by the quality of our data asset and the engine that powers it. We’re on a mission to build a comprehensive and complete business dataset for every business in the world. As part of the Foundation team at Middesk, you’ll collaborate with Data Science, Infrastructure and Data Engineering to build and maintain our own proprietary Entity Resolution system used to power the Middesk business identity platform, scaling our system to resolve millions of business identities across hundreds of data sources and thousands of distinct data sets. You’ll often work with and support product engineers looking to launch new products and features.

The ideal candidate is ready to learn relentlessly, eager to learn a wide-breadth of technologies and software stacks,  build and scale our data infrastructure and data pipelines and connect them into our core business verification products. 

We follow a hybrid work model, and for this role, there is an expectation of 2 days per week in our SF or NYC office. Candidates should be based within a commutable distance, as we believe in the value of in-person collaboration and building strong team connections while also supporting flexibility where possible.

• Build the batch and streaming data pipelines critical to Middesk’s data infrastructure using Airflow, BigQuery, Dataflow, Dataproc and VertexAI.

• Build reliable, scalable, maintainable, and cost-efficient systems across the stack.

• Build, scale and extend existing web scraping platforms and capabilities used for real-time data acquisition.

• Build, update and maintain our interface layer between our application and data platform stack to improve performance and as we continue to scale our platform and datasets. 

• 5+ years of experience working in a data engineering or backend engineering role

• Strong programming skills in at least one backend language and web application framework

• Knowledge of SQL and experience with RDMS like postgresql, mySQL, etc.

• Fluency with Ruby, Python or Go

• Experience building and designing collections stored on ElasticSearch 

• Experience working with Airflow or other orchestration platforms 

• Experience working on data pipelines and data streaming with tools like Spark or Kafka

• Experience building machine learning systems

• Experience with Terraform, Datadog, or Kubernetes

• Experience with scraper technologies, including agentic AI