Data Engineer
We connect 1.5 billion active shoppers with the things they need and love. Our client’s technology takes an algorithmic approach to predict what user we show an ad to, when, and for what products. The dataset is about 50 petabytes in Hadoop (more than 120 TB extra per day) and we take less than 10ms to respond to an ad request. This is truly big data and machine learning without the buzzwords. If scale and complexity excite you, join us!
You’ll spend time on the following:
- Work with analysts to collect and define analytics data needs, build technical specifications
- Ensure we design a consistent and global data model suitable for analytics
- Translate the needs into SQL, build optimal data pipelines that meet business owners needs and requirements , and validate the data
- Maintain the most critical data sets with high quality monitoring
- Develop tools and processes to collect and monitor Data Quality metrics throughout our stack
- Investigate Data quality issue and propose innovative solutions to avoid recurrences
- Participate in building the long term vision on data governance problematic
Here’s what we’re looking for:
- Practical experience writing SQL queries
- Practical experience on data modelling and ETL
- Understanding of high performance and large Hadoop clusters
- Analytical mind - You like to question the information you have and understand the big picture and the real problems that should be solved
- Scalability - You like working with problems involving huge amounts of data, provide data insights, with good reliability practices
- Accountable - you have a real sense of ownership and feel responsible for the service your team provides to multiple clients
- Passionate - You are a problem solver, a fixer, and a creative technologist. We believe coding is a talent and a passion, not just a skill.
- Team Oriented - You need to be a great team worker and a good communicator.
Nice to have:
- You are comfortable with Java or Scala or a similar language
- Practical experience writing Map/Reduce, Scalding/Spark or Hive/Presto jobs
- Previous experience as an analyst, exploring large data-sets to extract business insights-
- Experience on Data Governance questions
- Experience working with product owners to understand and implement business requirements
We proudly, passionately and actively strive to make both ThoughtWorks and our industry more representative of the communities we serve. We promote diversity in all its forms and reject discrimination and inequality.
Our diversity and award winning culture inspires our thought leaders and serves to nurture and develop amazing ideas. We believe this makes us a world leading destination of choice for all technologists.
We’re also passionate about delivering quality by ensuring the most valuable use of our talents and experience. We aim to support different working patterns to ensure a diverse collective of people can call ThoughtWorks their home so if you’re looking to work with high profile clients, delivering digital transformation and innovation, get in touch and chat to us about working flexibly!