DevOps has been steadily growing in popularity among engineering teams, boasting the ability to break down silos and facilitate cross-collaboration. But in order to harness the power of DevOps, teams must first decide on the tools they should use to get the job done.
With so many different tools to choose from, it can be difficult to make a decision. That’s why MayStreet DevOps Engineer Mike LaRocca recommends considering core objectives before diving into tool selection.
LaRocca and his team examined several key factors when choosing their tools: strong use case, stability and support, and cost-benefit analysis.
For LaRocca, it’s important to be continually tinkering with his team’s toolchain, as adaptation becomes the key to driving growth.
“We create space for exploring new tools and ideas,” he said. “This openness is crucial to identifying a variety of possibilities for future innovation.”
Built In NYC caught up with LaRocca and another local engineer to discover how their teams have built out their DevOps toolchains and how these tools have helped them grow.
MayStreet’s platform is designed to provide organizations with deeper insights into global capital markets.
Give us a brief glimpse into your DevOps toolchain. What are a few of your favorite tools your team is currently using?
MayStreet embraces tools driven by core DevOps and site reliability engineering principles like infrastructure as code, continuous integration and deployment, and monitoring and observability.
Terraform gives us the instant ability to see any cloud environment or infrastructure system laid out in code. The nuts and bolts that UI and command-line interface tools abstract away are exposed, apparent and reproducible. This enables simpler onboarding, change tracking and the execution of repeatable release pipelines. We embrace modular development, and we strive to create abstractions that follow the single-responsibility principle so we can reuse the modules as needed.
Ansible is highly versatile. We leverage playbooks in continuous integration to automate processes and deploy to Kubernetes. We also have a system of EC2 hosts that serve our users, all provisioned and ensured through playbooks.
We leverage Prometheus and Grafana for monitoring and alerts via Opsgenie. We host code on GitLab and develop pipelines in GitLab CI. We’ve also integrated Teleport for simple and secure host access management. Looking ahead, Terraform’s compliance framework will help us develop enforceable patterns and rules for infrastructure sets. Drifctl will help us integrate pieces of the infrastructure that are not yet in Terraform.
What were some of the key considerations when evaluating and choosing the tools your team would use?
The first consideration is a strong use case. Does the tool improve a process, reduce the chance of error or enhance visibility? Working with unnecessary tools is inefficient, as fun as proof of concept exercises and exploration can be.
Next is the support network. We seek to adopt tools that are growing, whether in terms of feature set or stability. If we encounter a bug, we need to know the issue will be addressed promptly. It’s even better if the tool is open source and we can address it ourselves. Another factor is maturity, meaning the tool must have established a certain level of usage or stability. Again, there is room for trying new approaches, but when bringing technology into the core process, it needs to be stable and supported.
Lastly, the cost-benefit analysis is crucial. Prometheus and Grafana have been great for us, but as we scale, does it make sense to use a paid tool like Datadog instead? We are constantly asking ourselves these questions, as they have huge implications on resource allocation. Adding a third-party tool can make sense when it gives the DevOps team more time to dedicate to other aspects of the system, is reasonably priced and provides an equivalent or better solution.
Questioning the status quo is vital to growth.”
What has been the key benefit your team has seen from building out and maintaining a healthy toolchain?
We have a few key traits that drive evolution. The first is our communication with the community. We create space for exploring new tools and ideas. This openness is crucial to identifying a variety of possibilities for future innovation.
MayStreet has a long history in the cloud. We had apps running on EC2s back when there was no identity and access management or virtual private cloud. As AWS grew, our team scaled the infrastructure to enhance networking, security and availability. Amazon’s Elastic Kubernetes Service is a great example of AWS introducing a valuable, low-cost managed service. It eases the migration of certain apps to k8s or containers, providing a less resource-intensive path than managing our own k8s clusters on EC2s. Another example is infrastructure as code. As Ansible launched and built out its solutions from there, we began to integrate it more heavily. Eventually, tools like Terraform evolved into more targeted solutions and we have integrated that as well.
There is also a desire among our team to push the envelope. Can we automate more in certain areas? Can we enhance security or visibility? Questioning the status quo is vital to growth.
Talkspace’s platform offers people 24/7 online access to certified therapists specializing in areas such as depression, anxiety and trauma.
Give us a brief glimpse into your DevOps toolchain. What are a few of your favorite tools your team is currently using?
We leverage a powerful set of DevOps tools to streamline various portions of our systems development life cycle. We are using Selenium with Cucumber for automated UI testing. We are using Mocha and Jest for testing on our Node.js and TypeScript applications. We handle AWS infrastructure deployment using CloudFormation stacks written and maintained using AWS’ Cloud Development Kit (CDK). CDK gives what we believe to be the truest form of infrastructure as code and allows us to directly integrate the infrastructure code for an app with the source code. All of this is running on our on-premises CircleCI cluster, which the team manages internally.
We employ tools such as Dome9 from Check Point and Cloudflare WAF to monitor and enforce network and security rules. Dome9 also allows us to run continuous compliance checks on our AWS infrastructure to ensure compliance with the federal law restricting release of medical information. We are utilizing New Relic and AWS CloudWatch for monitoring and alerting on many metrics throughout our platform. All of this information is aggregated and escalated through Opsgenie.
What were some of the key considerations when evaluating and choosing the tools your team would use?
After security and privacy, the most important thing is reusability. We are growing quickly, which means that operations need to be able to support new systems and services coming down the pipeline. Our DevOps tools need to behave in such a way that we do not need to reinvent the wheel each time a new service is added to the monitoring platform or a new repository needs to be deployed.
We tend to determine this reusability by deciding how much of the tooling setup can be done by code, configuration files or some other manual process. It’s always a good sign if the tool has a robust API and great documentation.
After security and privacy, the most important thing is reusability.”
How has your DevOps toolchain evolved over time?
As we have experienced growth, our toolchain has had to adapt in many ways. We needed to onboard an alert aggregation tool like Opsgenie to organize and centralize incoming alerts. We also needed to add more tooling around security like AWS GuardDuty. The logistics of operating out of one AWS account versus operating out of a dozen presents different security and compliance challenges that require different tools. Utilizing AWS Organizations tools such as service control policies allows us to consistently grow our footprint with peace of mind.
