HashiCorp Consul: Because Dynamic Workloads Call for Dynamic Service Networking

HashiCorp is taking multi-cloud infrastructure automation by storm with their formidable suite of open-source products. Their approach elevates workflows over technology.

Their offerings span the infrastructure layer with Terraform and Packer; the security layer with Vault and Boundary; the application layer with Nomad, Waypoint, and Vagrant; and the networking layer with Consul.

In this article, we’ll zoom in on Consul.

What is HashiCorp Consul?

HashiCorp Consul, featured on TechRadar by Devoteam, is a microservices networking solution with service mesh capabilities. It solves some of the headaches that come with managing applications in a distributed system. 

As Armon Dadgar says in his Consul explainer video, there’s no such thing as a free lunch. 

In other words, while the trend to microservice architecture offers advantages—like efficient development, faster time to market, and scalability—these advantages come at a new “cost.” Splitting an application into separate microservice components introduces a new set of operational challenges.

The challenges of scaling microservices

At a high level, here are some of the gnarly problems that come with scaling microservices-style applications:

• High number of load balancers needed across all the microservices
• Load balancers introducing a single point of failure for each corresponding microservice
• Network latency as services talk to one another through the load balancers between them
• Security concerns, as you no longer have these functions running within the same application using the same ecosystem, but data being transferred from one component to another across a network.
• Traffic patterns become messy

A service networking solution like Consul allows you to overcome these challenges in an orchestrated, automated fashion.

HashiCorp Consul explained

The way HashiCorp Consul approaches these service networking challenges is through what we call a “registry.” Essentially, this is a common registry “where information about the health and location of all of your services can be managed in real-time,” HashiCorp explains. It introduces an efficient way for discovery and traffic routing between the various services.

This service registry tracks services using their “identities.” 

So instead of using load balancers, microservice instances boot and get registered as part of this central registry. And when one microservice wants to talk to another microservice, it queries the common registry, which removes the need to go through a load balancer. So you get failure detection and load levelling across multiple instances.

Consul lists four functionalities at its core:

1. Service discovery—a central registry governing the way services should communicate with one another
2. Automated networking—reducing the burden on operations by doing manual tasks repeatedly 
3. Secured networking—through authentication, authorization, and encryption
4. Controlled access—a way to control access to services and centralising traffic management

And now, with a better way to govern discovery, configuration, and segmentation, you can scale your microservices application in a simple, organised way. 

Consul is designed to be “highly secure, highly scalable, and fault tolerant.” It can connect 50,000+ microservices.

Does every microservice architecture need a service mesh?

No. It depends.

As cloud consultant Ned Bellavance explains in a thought-provoking video, you might even have to step back and consider whether breaking up your application into microservices is even the ideal model for your case. A monolithic application can still make sense in certain situations, so it’s important not to rule it out just because the trend is pushing away from it. There are so many factors to consider.

And if a microservices application architecture is the right way to go, introducing a service mesh in the network layer is not something you need to jump into blindly. You really need to weigh the benefits of a service mesh like Consul against (1) the current challenges you’re facing, (2) the total cost of ownership of the solution, and (3) the return on investment. At the end of the day, the solution you pick needs to be as simple as possible while meeting the requirements of your complex problem. That way you’re not indefinitely adding a disproportionate amount of new complexities every time you add a new solution to your distributed system tech stack.

If your microservices applications have evolved to the point where “new features are typically introduced as additional external services” and “the distribution of your applications continues to grow,” that’s where you’ll hit a wall, George Miranda explains in an O’Reilly article. And that’s where introducing HashiCorp Consul to streamline your service networking is an excellent move.

How well does Consul integrate with other tools?

Integration is the name of the game. You can use Consul across various platforms and technologies. Connecting your workloads across multiple data centres and clouds—well, that’s exactly what it was designed to do. Currently, it supports 30+ technology partner integrations.

Jake Lundberg, HashiCorp’s Field CTO, says, “Consul is system and platform agnostic, which means that it easily integrates with any of your existing toolsets. If you’re running anywhere from your mainframe systems to bare metal, VMs, containers—or even into serverless networks—Consul will work for you. It easily integrates with existing application scheduling platforms like HashiCorp Nomad, and Kubernetes.

So let’s dive into some use cases.

HashiCorp Consul use cases

Use cases span from common to advanced and include:

• Discovering, tracking, and monitoring microservices in a dynamic, ephemeral environment
• Service registry and health monitoring with a real-time directory of all services and their health status
• Network middleware automation for dynamic reconfiguration as services scale up, down, or move
• Network infrastructure automating (NIA) with “Consul–Terraform–Sync”
• Combining Consul’s API Gateway with Consul’s service mesh for a control plane to manage both east-west and north-south traffic
• Built-in dynamic load balancing (but integration with other load balancers is possible too)
• Multi-platform service mesh to support a multi-cloud strategy
• Zero trust network with service mesh to secure service-to-service traffic with identity-based security policies and encrypted traffic with Mutual-TLS
• Centralised visualisation that provides observability of decentralised applications
• Automated GEO failover

Consul is often used in conjunction with Kubernetes for automating, securing, and observing connections between pods and clusters. HashiCorp offers tutorials and documentation on this.

The Consul website features enterprise case studies so you can see what’s possible. Clients include Mercedes-Benz, Stripe, Bloomberg, Workday, and Criteo.

How do I get started with Consul?

HashiCorp is transparent about its pricing tiers. You can opt for Consul’s self-managed open-source download (free), cloud-managed version, or self-managed enterprise offering.

How can I learn more?

This article is part of a greater series centred around the technologies and themes found within the first edition of the TechRadar by Devoteam . To read further into these topics, please explore TechRadar by Devoteam .