Senior Data Engineer at Transfix
What you'll do:
- Design and plan the implementation of data-based architectural frameworks that power logistics business from a data-driven perspective
- Build algorithms and software against his designs and architecture documents.
- Assure the quality and performance of data products by using unit tests, integration tests, and monitoring and alerting tools to catch in-flight errors and exceptions.
- Design, implement, and scale the architecture of Transfix’s data warehouse, including associated data marts.
- Build and maintain a variety of scalable ETL workflows and event pipelines for pushing data from our production applications and online databases into the warehouse.
- Write scripts that read from, and write to, external APIs and disparate sources of data (e.g. S3, scrapes of market data, customer APIs, and vendor APIs like Salesforce and Netsuite).
- Design and maintain appropriate development and testing environments and Continuous Integration / Continuous Delivery (CI/CD) processes that increase safety of changes to our production environment.
- Manage backup and recovery schedules for production databases.
- Detect, triage, and resolve issues with production data warehouse and ETLs.
- Like all engineers on the team, you take part in on-call rotations for your domain area.
- Bachelor’s degree in mathematics or a related field plus 4 years of experience in data engineering.
- 4 years of experience with data storage systems and big data processing frameworks.
- 4 years of experience with agile software development methodologies.
- 3 years of data engineering experience designing data warehouses and data pipelines using database replication, ETL, event streaming, and reporting tools.
- 6 months of administering reporting/analytics tools such as Looker or Tableau.
- 3 years of experience with Python, Ruby, SQL
- 6 months of experience with Snowflake and Cloud Based development.
- 6 months of experience establishing CI/CD processes for automated testing, making frequent changes to production, and managing schema migrations.
- Must successfully complete competency-based interview.