Oh Snap!
This job is no longer active - but you can still view the details below.

Senior Data Engineer

| Greater NYC Area
Apply now


Custora exists to help our customers improve the relationships they have with their own customers. We do this by ingesting data about every interaction a company has with each of their customers, and then making predictions using that data about how those customers will behave in the future. Our customers then use these predictions for everything from major strategic decision making to tailoring their communications with each customer.

Data engineers at Custora work on the pipelines that sit at the core of this architecture. We work to make these pipelines faster, more fault tolerant and to expand their scope.

The volume of data is large: we're working with 7 of the top 20 largest retailers in the world (+ many more not in the top 20), and are ingesting data from them both in a regular batch and in near-real time.

We've carefully selected the types of data to ingest to favor high signal data, so we care deeply about maintaining the correctness and completeness of the data being ingested as that directly affect the output of our models, and therefore the output of our product. Your work directly impacts both the predictions we’re able to make, and the day to day performance our customers experience when using our product.

Getting more specific, you will:

  • Design and build complex data pipelines on the Spark platform, ingesting both batch and real time datasets
  • Optimize the performance of data pipelines, getting into the internal of Spark and potentially other technologies to make gains
  • Spec and build the infrastructure that the data pipelines run on
  • Work with our data science team to deploy predictive models at scale
  • Build tools to continuously validate incoming data and proactively identify and communicate data anomalies before they manifest into problems.

We’re a small team, so you’ll be working on (and be able to meaningfully contribute to) high impact projects from your first day. 
What’s the day-to-day like?
Inspired by Basecamp, we work in ~8 week product cycles. First, we work together (engineering + product) to identify the projects we think will have the biggest impact on our company goals. Here’s an example of a recent project we conceptualized and delivered over one of these cycles:

Build out entity resolution functionality

Someone may make a purchase both online and in a store, to the retailer this may look like two people. The same person could have used a different email address when signing up for the retailer’s mailing list compared to when they bought something online. To get the best input to our models, we need to be able to identify that the purchases and the email list sign up came from the same person.

A feature to link multiple customer identifiers to the same person and remove duplicate customers by rule came about in two phases. The first was working with a retailer that could provide us with mappings of their online and offline purchases. A few months later, a separate retailer needed a rules based approach to deduplicating their customers. The challenge here is being able to ingest mapping information on customer ids and apply rules to deduplicate customer records at the same time. If we deduplicate by rule first, we may lose ids that should map to each other, but if we have multiple entries for a customer, we may generate incorrect mappings.*

It speaks to the flexibility of our pipeline that we were able to combine these two processes into a single step in our pipeline. Once the entity resolution process was deployed at scale on our Spark cluster, retailers could take advantage of one or both of these features. It has made a significant impact in our ability to find high value customers - customers that typically make many purchases in multiple channels and has multiple touchpoints with the retailer.

Qualifications:

  • 5 or more years of experience as a software engineer.
  • Degree in Computer Science or a deep competency achieved via other means.
  • Familiarity with Spark, Ruby on Rails, AWS, and SQL-based databases
  • High standards for code quality and maintainability.

Nice to Have's:

  • Experience with Scala, R, and/or Chef.
  • Consistent record of delivering significant features or building out platforms and services.
  • Experience working in e-commerce.

What it’s like to work here:

  • On Monday we eat and meet as a team to chat projects and progress.
  • We’re 70ish genuinely nice people. We work together and experiment with how to do things. It’s an amazing environment to both explore your professional interests and shape your career trajectory.
  • We move quickly. You build something and the next day it comes to life. You see and feel an immediate impact with the collective efforts of the team.
  • We’re building a company and a team we love. We’re in it for the long run.
  • Read more about what makes us, us here.
  • Find out more here or here.

The perks

  • We’re a flexible work environment
  • Competitive salary and meaningful equity
  • Health, dental and vision insurance (100% covered)
  • Free lunch every day, plus free water — hot and cold!
  • Unlimited vacation: take as much time as you need (we recommend at least 3 weeks)
  • Monthly unlimited MetroCard

*A bit of technical detail on how the algorithm was implemented:

We solved for transitivity in the id mappings and the rules based deduplication by using a graph based approach. We represent user ids as nodes in a graph and ids that map to the same person is an edge that links two nodes. By computing disjoint subsets, each subset of ids represents a unique customer, the list of ids and how they map to each other is then ingested.


Custora is an equal opportunity employer. We value diversity. We don’t discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, marital status, veteran status, disability status, or socioeconomic status.

Read Full Job Description
Apply now
loading ...
Emailed

Technology we use

  • Engineering
  • Product
  • Sales & Marketing
    • JavascriptLanguages
    • RLanguages
    • RubyLanguages
    • ScalaLanguages
    • ReactLibraries
    • HadoopFrameworks
    • Ruby on RailsFrameworks
    • SparkFrameworks
    • HiveDatabases
    • MySQLDatabases
    • RedisDatabases
    • Google AnalyticsAnalytics
    • SketchDesign
    • WordpressCMS
    • HubSpotCRM
    • SalesforceCRM

Location

We're a skip from Bryant Park in the heart of the Garment District, with easy access to all the trains and an outdoor terrace overlooking Manhattan.

An Insider's view of Custora

How has your career grown since starting at the company?

I joined Custora after having spent time in finance and consulting. While I joined on Success team, just a year in, I was encouraged to follow my passion for product development. The ability to map your own career trajectory and have such a big impact makes Custora an amazing place to come to work every day!

Nitika

Product Manager

What unique initiatives do you have that encourage innovation?

Our product development process at Custora encourages innovation across multiple teams, not just engineering and product. Team members across the org have the ability to pitch product and development initiatives which we get to vote on and decide as team what gets built and what get prioritized.

Jon

Co-Founder, CTO

What are Custora Perks + Benefits

Custora Benefits Overview

We pride ourselves on making the Custora work environment one that people feel supported and appreciated. We offer generous benefits across the board and incorporate them into our compensation packages in a holistic way.

Culture
Friends outside of work
Eat lunch together
Intracompany committees
Team owned deliverables
Team based strategic planning
Group brainstorming sessions
Open office floor plan
Health Insurance & Wellness Benefits
Dental Benefits
100% coverage
Vision Benefits
100% coverage
Health Insurance Benefits
100% coverage
Retirement & Stock Options Benefits
401(K)
Company Equity
Child Care & Parental Leave Benefits
Generous Parental Leave
Flexible Work Schedule
Return-to-work program post parental leave
Vacation & Time Off Benefits
Unlimited Vacation Policy
Perks & Discounts
Casual Dress
Commuter Benefits
Company Outings
Free Daily Meals
Stocked Kitchen
Some Meals Provided
Lunch provided daily!
Happy Hours
Often offsite and more often at our office bar and terrace
Pet Friendly
Recreational Clubs
Fitness Subsidies
Professional Development Benefits
Job Training & Conferences
We provide an annual stipend for personal & professional development workshops and classes.
Lunch and learns
Acme Co. hosts lunch and learn meetings once per month.
Continuing Education stipend
We offer $500 annually for continuing education.
Apply now
Save job
loading ...
Emailed
View Custora's full profileSee more Custora jobs