#Healthcheck for #golang #microservices

One of the important paradigm change in moving away from monolith application to a cloud of micro-services is how to keep all that interconnected could functioning.

A monolith enterprise application usually runs in an application container so a lot of time you are using the container monitoring features to make assumptions about the heath of the environment.

In case of micro-services you need to move also the health-check at the micro-service level. Lucky for us in the docker based environments the orchestration/ management part is part of the high availability orchestration offered by docker swarm or kubernets. I will touch briefly on that later.

A GO micro-service at its core can be always designed as a small HTTP server that does something. This way we will see that it is very easy to design a health-check for the micro-service.

One of the nicest libraries I found for this is https://github.com/InVisionApp/go-health

As the maintainers describe it:

A library that enables async dependency health checking for services running on an orchestrated container platform such as kubernetes or mesos.

 

The usage is very straight forward:

Step 1: Define a simple implementation of a health checker (basic check if the database on which we depend is accessible)
ServiceCheck.go

package service 
import (
"database/sql"
"fmt"
)
 
type ServiceCheck struct {
   Database *sql.DB
} 
 
func (c *ServiceCheck ) Status() (interface{}, error) {
  // ping postgres
  err := c.Database.Ping()
  if err != nil {
     return nil, fmt.Errorf("Cannot ping database")
  } else {
     return nil, nil
} 

Note that we just have to define a struct that implements a Status method.

Step 2: Add a HTTP /healthcheck handler to the micro-service

Not to have a trivial example let’s consider that our service is doing some operations with a database and we want to be sure we are connected to the database.

First declare the imports

package main

import (
	"database/sql"
	"fmt"
	health "github.com/InVisionApp/go-health"
	handlers "github.com/InVisionApp/go-health/handlers"
	_ "github.com/lib/pq"
	"log"
	"net/http"
	"service"
	"time"
)

Then declare the global variables:

var Db *sql.DB
var H *health.Health

Declare the main function, connect to the database

func main() {
	// connect to the database
	// build connection string
	psqlInfo := fmt.Sprintf("host=%s port=%s user=%s "+
		"password=%s dbname=%s sslmode=disable",
		service.DbHost, service.DbPort, service.DbUser, service.DbPassword, service.DbName)
	// open connection
	var err error
	Db, err = sql.Open("postgres", psqlInfo)
	if err != nil {
		log.Fatal("Cannot Open DB : " + err.Error())
	} else {
		log.Println("Database connection opened")
	}

Create a new health instance and add to it an instance of our service checker.

	H = health.New()

	// instantiate service checker
	pchk := &service.ServiceCheck{
		Database: Db,
	}

	H.AddChecks([]*health.Config{
		{
			Name:     "service-check",
			Checker:  pchk,
			Interval: time.Duration(5) * time.Second,
			Fatal:    true,
		},
	})

Start the health check instance

	if err := H.Start(); err != nil {
		log.Fatal("Unable to start healthcheck ", err.Error())
	}

Add the HTTP /healthcheck and /DefaultHandler handlers and start the HTTP server.

// register handlers
	http.HandleFunc("/healthcheck", handlers.NewJSONHandlerFunc(H, nil))
	http.HandleFunc("/", DefaultHandler)

	log.Println("Start Server...")
	err := http.ListenAndServe(":"+sevice.ServicePort, nil)
	log.Fatal(err)
}

// Default Request Handler
func DefaultHandler(w http.ResponseWriter, r *http.Request) {
	...
}

Step 3: Add the healthcheck call to docker-compose.yml

Make sure curl is installed in the docker image of our service. If using alpine an “apt-get install curl” in Dockerfile of our micro-service is all what is needed.

In docker-compose.yml file then when specifying our service entry just add a curl call to /healthcheck.

service:
  myservice:
   image: serviceImage:latest  
   healthcheck:
       test: ["CMD","curl","-f","http://localhost/healthcheck"]
       interval: 1m
       timeout: 10s
       retries: 3  
...

Now you can monitor the health of the service directly using docker-compose ps.

The advantage is that if running in a cluster environment like Docker Swarm or Kubernets the service will auto-start the service on failure.

If not running in a cluster environment you can still benefit from this by using https://hub.docker.com/r/willfarrell/autoheal/

The above feature was supposed to be standard functionality in docker by it looks like was dropped as it is considered to be a cluster feature.

Autoheal: Monitor and restart unhealthy docker containers. This functionality was proposed to be included with the addition of HEALTHCHECK, however didn’t make the cut.