Engineers are some of the hardest workers in tech — but many say it’s also one of the most rewarding roles the industry has to offer. Solving complex problems, developing new products, and transforming ideas into code that people use daily can be as fun as it is challenging.
Built In NYC recently connected with seven of New York City’s best and brightest engineers to learn about a recent project that inspired them to think in new and exciting ways. Their answers were, to say the least, an inspiration.
Shutterstock is constantly pushing the limits of their platform, and Developer Stefan Hayden has been on the front lines of some of their most acclaimed technologies. He credits Shutterstock’s singular vision for the unique success the company continues to find.
Describe your favorite project, what were the goals and how were they achieved?
Shutterstock Editor is by far my favorite project and I feel fortunate to have worked on it over the last two years. Our team was tasked with helping users create a finished design with the images they find on Shutterstock, and I’m pleased to say the user feedback has been amazing. It's energizing to interview users and hear the enthusiasm in their voices for how we have enhanced their experience.
How did your company support your deadline-driven work?
Few companies attempt to achieve what Shutterstock has with Editor, and few developers have had the opportunity to build such a full-featured design tool for a browser. While we set goals and timelines for this initiative, we also allowed ourselves the space to learn about the problem we were trying to solve and the tools we should use to solve it. It took a few attempts before we settled on an overall architecture, and we continue to improve the modules and code quality every day. Many companies might have lost faith on the path to where we are today, but Shutterstock had a vision and stuck to it.
Describe something you learned while working on this project.
Two years ago, I moved from a PHP LAMP project that only used jQuery to work on Editor, which fully adopted the new stack of node/express/react/webpack. The list of new skills I’ve had the ability to learn is endless, and I couldn't have done it without Shutterstock's support.
For Carlos Orrego, a software engineer at digital advertising buying firm The Trade Desk, new and exciting challenges are one of the perks of the job. Fortunately, his company provides both resources and mentorship — anything an engineer might need is merely an ask away.
Describe your favorite project, what were the goals and how were they achieved?
My favorite project so far involved processing billions of user data rows in order to generate look-alike models for our customers. This is a very memory and disk-intensive task and required offloading processing onto a very large Hadoop cluster. But more hardware is not always the answer, and so we also developed some clever tricks to speed up processing these vast amounts of data.
How did your company support your deadline-driven work?
I received all of the resources I needed right away. Access to the big data platform, provisioning of test clusters, access to support and documentation. Our principal engineers were always willing to give advice and help with any issues. I had great design sessions with other engineers where I explained the problem and they offered suggestions. Everyone made it clear that if I needed help, I just had to ask.
Describe something you learned while working on this project.
Up until now, I had worked on other areas of our system where I could see the expansiveness of our platform. For example, we’re now processing 10 million requests per second. But I hadn’t had a chance firsthand to see the massive amounts of data we work with every day. It’s hard to imagine until you actually have to process all of it. This is what big data really means.
Because DigitalOcean works to create the best cloud computing software for developers, engineers no doubt hold themselves to the highest possible standard. Senior Software Engineer Vaibhav Bhembre explained that this is an all-hands on deck process, and input from all is valued equally.
Describe your favorite project, what were the goals and how were they achieved?
Spaces, DigitalOcean's object storage product, would undoubtedly be my favorite project. The goal for Spaces is to take the scalable and fault-tolerant storage infrastructure we had experience building to the next level by providing cloud storage service directly to all developers and their users in the world, regardless of whether they were using Droplets or not.
How did your company support your deadline-driven work?
Building and shipping new products and services at DigitalOcean has always been a cross-team effort, be it internal or external. The open access and collaboration that DigitalOcean fosters helped ensure that dependencies between engineering teams were timely and accounted for as we worked on implementing, testing and benchmarking the new product. As we neared the launch date, multiple teams including community, product and marketing, pitched in to ensure that the deployment was not only on time but successfully received by our users.
Describe something you learned while working on this project.
DigitalOcean puts a lot of thought into providing a seamless user experience for all products we build. The "DO Simple" mantra is central to everything we do here. Meeting with teams that drove design for our Spaces control panel was quite a learning experience for me. The control panel not only simplified object access but also managed object and spaces permissions easily. We were able to borrow a few of those learnings and employ them, leading to truly simple workflow chains from top to bottom of our engineering stack.
Though Codecademy excels at teaching the complicated world of software engineering to students, they also have skilled engineers, like Lydia Stepanek, who keep things running smoothly behind the scenes of the classroom.
Describe your favorite project, what were the goals and how were they achieved?
I worked on building a tool for our operations team that helps us manage a huge number of incoming projects that need to be reviewed. The tool automatically assigns each project to a code advisor who provides feedback on the project, saving operations from having to manually assign every project, which takes many hours of work each week (and weekends!).
How did your company support your deadline-driven work?
The tool actually crashed at one point, due to a lot of extra database calls that were being made from my code. I organized a postmortem and a ton of my teammates came to the meeting and supported me. They showed me commands to test performance, and we created action items that would avoid a similar problem from happening in the future. I learned a lot from the team.
At IEX Group, a U.S. stock exchange, engineers are responsible for changes happening at a millisecond level. Quantitative Researcher Allison Bishop says her scientific background helps her excel in this challenging, fast-paced environment, and that IEX’s support enables her research to thrive.
Describe your favorite project, what were the goals and how were they achieved?
I work on the IEX Signal, a model that is designed to predict moments of price instability at a millisecond level. In combination with the350-microsecond speed bump into our market, the signal helps us protect institutional investors from latency arbitrage.
How did your company support your deadline-driven work?
I am proud to work for a company that is very supportive of research to improve our market, even when that means filtering out bad trades (which in the short term actually costs us money). IEX provides the time and tools to do my research thoroughly, so that each time we roll out a model, I can be sure we are getting it right and really improving the experience. Since I come from an academic background, I consider this level of rigor to be a hallmark of my scientific work, and I am happy that IEX supports my work to maintain that standard.
Describe something you learned while working on this project.
I learned a lot about trading dynamics at the millisecond and microsecond time scales, which is a fascinating ecosystem of activity. I also learned that simple models can often be as effective as complex ones, and can capture what’s happening in a more intuitive and communicable way.
Turbonomic is at the forefront of one of the hottest topics in tech. The company develops automated cloud management solutions that help businesses control costs and maintain compliance. Amruta Karmarkar, senior manager of public cloud engineering at Turbonomic, explained the challenges of managing something as complex as the cloud, and the joys of working within a supportive company.
Describe your favorite project, what were the goals and how were they achieved?
I joined Turbonomic in February, and my first task was to help the engineering team get Public Cloud Support out the door. This was a great “first” project for me at Turbo: solving the extremely difficult problem of automated workload optimization and placement in the public cloud; not to mention the challenging timeline and the opportunity to build something truly differentiating and impactful for the company. We were able to overcome this immense challenge with the help of a dedicated team, working at an incredible speed, while dealing with every obstacle that came their way.
How did your company support your deadline-driven work?
Turbonomic is a great place to work. Everyone here is incredible. Even the founders are hands-on, technical and willing to dive into the details and pitch in to help whenever needed. That is something I haven’t seen at other companies. Being new to Turbonomic I had to ramp up very quickly on the Turbo core technology. Our amazing and talented engineers and management team helped every step of the way and enabled me and the team to be successful.
Describe something you learned while working on this project.
I learned the complexity of managing the price to performance in the Public Cloud, which is extremely challenging! Without a deep understanding of how to place and optimize application workloads in the cloud, enterprises more often than not see their IT budgets balloon out of control. I got hands on with technologies such as AWS and learned more than I could have imagined along the way. If you like the idea of learning every day and figuring out challenging integrations and technologies, Turbonomic has plenty to go around.
On an e-commerce platform such as Spring, there are many moving parts one must keep track of. Senior Software Engineer Julie Qiu has helped streamline this process, and says she couldn’t have done it without the support and positivity of her team.
Describe your favorite project, what were the goals and how were they achieved?
This year at Spring, I led the effort to incorporate natural language processing into our search engine, which led to a 20 percent improvement in search-to-order conversion rate for complex searches. To do this, we partnered with Twiggle, a startup that specializes in machine learning-based search. The project entailed indexing our entire product catalog against Twiggle's product ontology to import their product attributes, and sending every search query that came through Spring to Twiggle.
How did your company support your deadline-driven work?
One of my favorite parts of this project was the great group of people involved. My product manager was incredibly proactive about communicating with the Twiggle team and helping coordinate two visits for the Twiggle team to come to our office (they are based in Israel). The engineers on my team were also great to work with and took ownership of different pieces of the project. Additionally, I had the opportunity to coordinate with our engineering team in Krakow, who built the first version of our search engine. Overall, this project was a great experience because of the people.
Describe something you learned while working on this project.
This was the first time that I worked with Elasticsearch, which helped me learn about data stores, how to optimize search queries, and how machines process the English language. I am actually going to speak at Elastic{ON} 2018 about search at Spring, so that is another new and exciting opportunity for me as a result of this project.
