Graceful Shutdown

Graceful Shutdowns for Reliability

Every running web application eventually needs to stop— either due to a critical, unrecoverable error or to rollout new features. As a result, the service must undergo a controlled restart operation. Abrupt termination of a running service diminishes its reliability and can disrupt system integrity.

"Graceful Shutdown" is a widely used concept in software engineering, referring to a set of steps that cleanly stop a service without causing distruptions. Fortunately, Golang provides built-in support to implement graceful shutdowns for HTTP services.

Let's explore how to implement a graceful shutdown with an example:

    // main.go

    var cfg ServerConfig

    shutdown := make(chan os.Signal, 1)
    signal.Notify(shutdown, syscall.SIGINT, syscall.SIGTERM)

    var mux *http.ServeMux

    server:= http.Server{
        Addr: cfg.Addr,
        Handler: mux,
    }

We create a buffered channel of size 1 to handle shutdown events. This event might be triggered, for example, when Kubernetes instructs the service to stop to deploy a new version of the program.

    // main.go

    serverErros := make(chan error, 1)

    go func() {
        serverErrors <- server.ListenAndServe()
    }()

Another buffered channel of size 1 is created to capture any errors that occur during server startup. To aovid stalling the main goroutine, the blocking ListenAndServe method is called inside a separate goroutine.

    // main.go

    select {
    case err := <- serverErrors:
        fmt.Println(err)
    case <- shutdown:
        ctx, cancel := context.WithTimeout(context.Background(), cfg.ShutdownTimeout)
        defer cancel()

        if err := server.Shutdown(ctx); err != nil {
            server.Close()
        }
    }

At this point, the main goroutine is blocked at the select statement, waiting for either a shutdown signal or any errors that occur during startup. Meanwhile, incoming requests are served concurrently in a separate goroutine.

When a shutdown signal is received, the Shutdown method is called with a graceful termination timeout. This timeout allows in-progress operations to complete before the server shuts down.

The Shutdown method first closes all open listeners, ensuring the server no more accepts any new incoming connections. It then waits for the duration specified by cfg.ShutdownTimeout. If this timeout expires before the shutdown completes, the Close method is called to forcefully terminate the server.

This process represents the set of steps required to implement what's known as a "Graceful Shutdown" in a Golang application.