Service Discovery with Consul

Resolve service addresses with Consul DNS.

2 years ago, I wrote an article about Service Discovery with Go. The solution outlined in this post is limited and this post is my new take on Service Discovery.

The limitations of the previous post included:

• Works only on Linux.
• Shells out to Linux commands, instead of using only Go code.

Per Carlos Otero Barros suggestion, I’m going to have a look at HashiCorp Consul. Consul can work on other operating systems and it has a Go library to interact with it.

George Dagerotip — unsplash.com

In this post, I’ll explore resolving the address of a service using DNS, instead of mDNS. To achieve this, I’ll utilize the DNS service offered by Consul.

Launching Consul

I’ll be using Docker Compose to download and launch Consul. The configuration for this container will be unorthodox because I’ll be setting the its network_mode setting to host. This means that the container will have full access to my computer’s network.

I’m setting it as such so that the agent can perform health checks on services running outside of Docker.

I’ll also be launching the Consul agent in dev mode, another thing to not do in production. Here is the docker compose configuration:

services:
  consul:
    image: hashicorp/consul
    container_name: consul
    command: "consul agent -dev -client=0.0.0.0"  # Start Consul in development mode
    volumes:
      - consul-data:/consul/data
    network_mode: host
    restart: unless-stopped

volumes:
  consul-data:  # Volumes to persist Consul data
    driver: local

By default, Consul UI will run on port 8500 and the DNS service on 8600. With the file in place, I’ll change the terminal working directory to the compose file’s directory and run the following command to launch the agent:

docker compose up

Now that Consul is up, it’s time to write a self-announcing service.

Building a Service with Go

To add a service to Consul, I’ll need to do the following:

• Launch an HTTP server with, at least, a handler registered at endpoint GET /health. This handler should return status code OK ( 200 ).
• Register the service on Consul.
• De-register the service on shutdown. Only if I’m running one instance of my service.

For the HTTP server, I’ll add a function that will register the health check handler to the default HTTP request multiplexer and launch a server with the specified port. Here is the code of said function:

// startHTTPServer starts a simple HTTP server with a health check endpoint
func startHTTPServer(port int) {
   http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
      w.WriteHeader(http.StatusOK)
      fmt.Fprintln(w, "OK")
   })
  
   log.Printf("Starting HTTP server on :%d...\n", port)
   // launch server and passing nil as
   // mux value to use default multiplexer (mux)
   err := http.ListenAndServe(fmt.Sprintf(":%d", port), nil)

   if err != nil {
      // exit program because
      // server will be launched as 
      // goroutine.
      log.Fatalf("Failed to start HTTP server: %v", err)
   }
}

With the server code in place, I’ll need a function that will register this service on Consul. To do this, my function will need the following parameters:

• serviceID (string): A unique identifier for the service in Consul's registry.
• serviceName (string): The name of the service to be registered.
• serviceAddress (string): The IP address or hostname where the service is accessed at.
• servicePort (int): The port on which my service listens for incoming requests.
• consulClient (*api.Client): The Consul client used to interact with the Consul API for service registration. I’m passing it because this will be the same client I’ll use to de-register the service on shutdown.

A health check is a mechanism used to monitor the availability and performance of a service, application, or system to ensure it is functioning correctly. The Consul API has a type named AgentServiceCheck and it is used to:

• Define which URL/ hostname or IP address to request during a health check.
• How often to perform the health check.
• How long to wait for a response during the health check.

For this post, I’m going to add a health check that will request my service’s GET /health endpoint. During the request, Consul will wait 5 seconds before deeming the server unreachable and perform a check every 30 seconds. Here is the code constructing AgentServiceCheck:

api.AgentServiceCheck{
   HTTP:     fmt.Sprintf("http://%s:%d/health", serviceAddress, servicePort),
   Interval: "10s",
   Timeout:  "5s",  
}

To register the service, I’ll create an AgentServiceRegistration instance using the parameters passed to the service registration function, then call the Consul API ServiceRegister function with it. Below is the entire registration function:

// registerServiceWithConsul registers the service with Consul
func registerServiceWithConsul(
   serviceID,
   serviceName,
   serviceAddress string,
   servicePort int,
   consulClient *api.Client,
) error {
 
   // Define the service registration
   registration := &api.AgentServiceRegistration{
      ID:      serviceID,
      Name:    serviceName,
      Address: serviceAddress,
      Port:    servicePort,
      Check: &api.AgentServiceCheck{
         HTTP:     fmt.Sprintf("http://%s:%d/health", serviceAddress, servicePort),
         Interval: "10s",
         Timeout:  "5s",
      },
   }
  
   // Register the service
   err := consulClient.Agent().ServiceRegister(registration)
   if err != nil {
      return fmt.Errorf("failed to register service: %v", err)
   }
  
   return nil
}

With the HTTP server and register function in place, I can begin writing the main function for this service.

Putting it together

To test out DNS resolving, I’ll need a main function that will:

• Launch the HTTP server on a Goroutine.
• Register the service on Consul.
• (Optional, if service is one instance) De-register service on consul on shutdown.

I’ll start by declaring the variables required by functions: startHTTPServer and registerServiceWithConsul. Here is the code performing this:

func main() {
   // Service configuration
   serviceID := "example-service-1"
   // serviceName will become prefix in 
   // dns query: example-service.service.consul
   serviceName := "example-service"
   // used by startHTTP and service registration
   // function.
   servicePort := 8080
   // the server will be running locally
   serviceAddress := "127.0.0.1"
   // address of consul service launched earlier
   consulAddress := "http://localhost:8500"
   
   // Specify custom config.
   config := &api.Config{
      Address: consulAddress, // Consul address
   }
  
   client, err := api.NewClient(config)
   if err != nil {
      log.Fatalf("Failed to create Consul client: %v", err)
   }
}

Next, I’ll need a Go routine to launch the actual HTTP service. Here is the code performing this:

func main() {

   ... variables

   go startHTTPServer(servicePort)

   ...
}

Once the service is launched, I’ll call the registerServiceWithConsul function and pass the variables declared at the beginning of the main function. Here is the code performing this:

func main(){
    
    ...

   // Register the service with Consul
   err = registerServiceWithConsul(serviceID, serviceName, serviceAddress, servicePort, client)
   if err != nil {
    log.Fatalf("Failed to register service with Consul: %v", err)
   }
  
   fmt.Printf("Service %s is running on %s:%d and registered with Consul.\n", serviceName, serviceAddress, servicePort)
  
   ...
}

Lastly, I’ll listen for termination signals and once one is received, I’ll call the Consul API ServiceDeregister function to remove the service from the Consul registry. Here is the code performing this:

func main(){
    
     ...

     sigs := make(chan os.Signal, 1)
     // define signals to listen for
     signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
    
     // Wait for termination signal
     <-sigs
    
     // Deregister the service upon shutdown.
     err = client.Agent().ServiceDeregister(serviceID)
     if err != nil {
      log.Fatalf("Failed to deregister service: %v", err)
     }

     fmt.Println("Service deregistered from Consul")
}

With the service’s main function complete, I can begin the test.

The Test

In the previous post, I wrote a function that would use avahi CLI to return a list of services through mDNS. In this post, I’m going launch the service and try to resolve it’s physical address with the dig command.

The dig (Domain Information Groper) command will enable me to query the Consul DNS server and retrieve the physical address of my service. I’ll run the dig command with the following arguments:

• @127.0.0.1: Specifies the DNS server to query; in this case, the local Consul agent running at 127.0.0.1.
• -p 8600: Specifies the port number to use for the query; here, it is 8600, which is Consul's default DNS service port.
• example-service.service.consul: The fully qualified domain name (FQDN) of the service being queried, where example-service is the service name registered in Consul. Unlike mDNS, I’ll need to know the service name ahead of time.
• +short: Outputs the results in a concise format, typically just the IP addresses, without additional query details.

Here is the command I plan on using:

dig @127.0.0.1 -p 8600 example-service.service.consul +short

After running the command, I should have the physical address of the service, assuming my service is running. Here is the output of the command:

Conclusion

I prefer this approach to service discovery over the one detailed in the previous post because it is platform-agnostic and works seamlessly across different programming languages.

However, unlike mDNS, this method requires services to explicitly register themselves at launch or another designated time, which adds a bit of overhead. Despite this minor drawback, the solution is highly scalable and better suited for enterprise environments.

The Consul HTTP API also allows you to query the list of services in the registry. This is essentially service discovery. However, Unlike mDNS, it involves less “magic” since you are responsible for explicitly registering the services yourself.

Thank you for reading!

Sources

• Consul API Docs — https://developer.hashicorp.com/consul/api-docs
• Go API client — https://pkg.go.dev/github.com/hashicorp/consul/api
• Full project source code: https://github.com/cheikhsimsol/go_proj/tree/main/consul/sv1
• Docker Compose — https://docs.docker.com/compose/
• Building an API Gateway with Consul and Go — https://cheikhhseck.medium.com/request-routing-with-consul-95df78aed9e5