Boost Docker Container Application Performance with One Change

Boost Docker Container Application Performance with One Change

Is your application running in a Docker container slower than you expect?

I once developed and supported a deployment of 3500 or so Docker containers running instances of the same software. After working through scaling issues, we still found that the workload had unacceptable performance even though the CPU, memory, and disk utilization were all fine. The individual application instances were taking 3-4 seconds to process requests when they should have taken 200-300ms based on local testing.

What can cause slow application performance in Docker containers?

After some in-depth investigating, I found the problem actually was slow Domain Name System (DNS) lookups that the application was doing inside the Docker containers. If you are unfamiliar with DNS, applications send a request to a DNS nameserver to resolve (that is, convert) a domain name like example.com into an IP address for a server. Docker containers by default use the Docker service itself to resolve domain names to IP addresses. While the Docker service can provide this functionality, using Docker service for DNS lookups is not as good as using a real nameserver. When I configured my Docker containers to use a real nameserver for DNS lookups instead of the Docker service, requests that took 3-4 seconds (that is, 3000-4000 milliseconds) to complete took only 20-30 milliseconds to complete. A single, simple configuration change decreased the latency of the application by 99%! I knew DNS lookups were a normal part of running any workload, but I was shocked this change made such a difference.

How do you boost Docker container application performance?

If you think your applications also suffer from slow DNS lookups, you can easily verify the performance impact of slow DNS lookups by configuring your Docker containers to use a different, specific DNS nameserver to handle DNS lookups instead of relying on Docker’s built-in functionality.

To configure your Docker container to use a specific DNS nameserver, set the dns option when deploying your Docker container. You have to provide one or more IP addresses for DNS nameservers to Docker. There are organizations that provide free-to-use DNS nameservers for the public Internet. For example, Google provide free-to-use DNS nameservers with IP addresses 8.8.8.8 and 8.8.4.4, and Cloudflare provides a free-to-use DNS nameserver with IP address 1.1.1.1.

Use the command line to configure the DNS nameservers for your Docker container

If you are using the command line to manage Docker containers, add --dns to your options for your container with a list of IP addresses for one or more DNS nameservers. For example, to use a Google DNS nameserver add --dns 8.8.8.8. If you want to use more than one nameserver, add the --dns option multiple times with a single IP address after each --dns, like --dns 8.8.8.8 --dns 1.1.1.1.

Here is an example Docker CLI command running a the hello-world Docker image with custom DNS nameservers:

docker run --name dns-test --dns 1.1.1.1 --dns 8.8.8.8 hello-world

This command will create a Docker container from the hello-world image, configure the Docker container to use 1.1.1.1 and 8.8.8.8 for DNS nameservers, and run the hello-world application.

If you run docker inspect dns-test you should see the IP addresses 1.1.1.1, and 8.8.8.8 in the Docker container configuration. The docker inspect command prints a large amount of information that can be difficult to read through. If you would like to see just the DNS nameserver configuration in particular, you can run the following command:

docker inspect --format '{{.HostConfig.Dns}}' dns-test

This command prints the container configuration and filters the output down to just the DNS nameserver configuration using Docker’s Go template syntax for output filtering. If you run that command you should see the following output:

[1.1.1.1 8.8.8.8]

This confirms that the Docker container is using 1.1.1.1 and 8.8.8.8 as DNS nameservers. You can remove the dns-test Docker container at this point using docker rm dns-test.

Use Docker Compose to configure the DNS nameservers for your Docker container

If you are deploying your Docker containers using Docker Compose, you can easily add DNS nameservers to the Docker container configuration by adding the dns key to your service configuration. Here is an example service configuration with the dns key using the hello-world image:

services:
  example:
    image: hello-world
    dns:
      - 1.1.1.1
      - 8.8.8.8

When you run docker compose up to deploy a service with the dns key, the Docker containers will use the DNS nameservers you provided. In the example configuration above, the Docker containers will use 1.1.1.1, 8.8.8.8. You can verify this by using the docker inspect command to review the configuration for the Docker containers making up the Docker Compose services.

If DNS lookups are your problem, you should see better performance

Any time your code makes some kind of connection using a domain name as the target address, you have to make a DNS lookup to map the domain name on to an IP address.

If you configure your Docker containers to use specific DNS nameservers instead of Docker’s built-in DNS lookup functionality you should see an improvement in performance if DNS lookups are slowing your applications down. This is especially likely for applications that make many HTTP requests or retrieve data from remote resources as part of their normal operation.