Docker

For information on how to install Docker, see the official installation information.

Taking a high-level view, Docker consists of the following major components:

  • the Docker daemon, which is responsible for creating, running, and monitoring containers, as well as building and storing images;
  • the Docker client, which is used to talk to the Docker; by default, this happens over a Unix domain socket, but it can also use a TCP socket to enable remote clients or a file descriptor for systemd-managed sockets;
  • Docker registries, which store and distribute images; the default registry is the Docker Hub, which hosts public images, however private registries can also be used to store sensitive images as well as to locally cache images.

At a more fundamental level, Docker is a containerisation system which uses a few key features of the Linux kernel:

  • cgroups, which are responsible for managing resources used by a container; they are also responsible for freezing and unfreezing containers, as used in the docker pause functionality;
  • namespaces are responsible for isolating containers, making sure that a container's filesystem, hostname, users, networking, and processes are separated from the rest of the system;
  • the Union File System (UFS), used to store the layers for containers; the UFS is provided by one of several storage drivers, either AUFS, devicemapper, BTRFS, or Overlay.

The following 'supporting technologies' are included with Docker and serve to extend or expand upon its basic functionality:

  • Docker Machine installs and configures Docker hosts on local or remote resources. Machine also configures the Docker client, making it easy to swap between environments;
  • Swarm, Docker's clustering solution; swarm can group together several Docker hosts, allowing the user to treat them as a unified resource; and
  • Compose, a tool for building and running applications composed of multiple Docker containers. It is primarily used in development and testing rather than production.


With reference to the container life cycle (with the above illustration originally from here):

From [[http://docker-saigon.github.io/post/Docker-Internals|here]]

Command Flag(s) Description
run IMAGE CMD Run a command in a new container
-h, --hostname string Sets the container's hostname
--name string Sets the container's name
-d, --detach Run container in background and print container ID
--device list Add a host device to the container
-p, --publish list Publish a container's port(s) to the host; list is of the form [HOST_IP:]HOST_PORT:CONTAINER_PORT
-P, --publish-all Publish all exposed ports to random ports; use docker port ID CONTAINER_PORT to discover the host-side port allocated by Docker.
--link list Add link to another container; list is of the form EXTERNAL_CONTAINER_NAME:DESIRED_HOSTNAME, where the former refers to a (pre-existing) container, and the latter refers to the hostname that can be used within the to-be-created container.
-e, --env list Sets environment variables; list is of the form VAR=VAL, and this flag may be invoked multiple times to set multiple variables
--env-file list Read in a file of environment variables
-v, --volume list Bind mount a volume. list is of the form HOST_DIR:CONTAINER_DIR; if no host directory is specified, it will be created under $DOCKER_HOME
-i -t, --interactive --tty Creates an interactive session
--rm Automatically remove the container when it exits
pause CONTAINER Pause all processes within one or more containers
unpause CONTAINER Unpause all processes within one or more containers
stop CONTAINER Stop one or more running containers
start CONTAINER Start one or more stopped containers
restart CONTAINER Restart one or more containers
kill CONTAINER
rm CONTAINER Remove one or more containers
-f, --force Force the removal of a running container (uses SIGKILL)
-v, --volumes Remove the volumes associated with the container
Command Flag(s) Description
attach CONTAINER Attach local standard input, output, and error streams to a running container; note that, by default, sending any signal sent to the attaching process / terminal will be forwarded to the container's process, so sending a ^C will stop the container. The correct way to detach is via a ^P ^Q.
--sig-proxy bool Proxy all received signals to the process (default true)
exec CONTAINER CMD Run a command in a running container
-i -t, --interactive --tty Creates an interactive session; when used with a /bin/sh-like command, useful for debugging / interactive image generation tasks
inspect CONTAINER Return low-level information on Docker objects
-f, --format template Format the output using the given Go template; an example template is {{.NetworkSettings.IPAddress}}
logs CONTAINER Fetch the logs of a container
-f, --follow Follow log output
ps List containers
-a, --all Show all containers (default shows just running)
-f, --filter filter Filter output based on conditions provided; an example filter is status=exited
-q, --quiet Only display numeric IDs
stats Display a live stream of container(s) resource usage statistics
top CONTAINER Display the running processes of a container; docker ps options may optionally be appended after the container name.
Command Flag(s) Description
build PATH Build an image from a Dockerfile, using PATH as the build context; a .dockerignore file can be used to specify which files / paths to not send to the Docker daemon during the build process
-t, --tag list Name and optionally tag the build; list is of the form [REGISTRY/]REPOSITORY[:TAG]
history IMAGE Show the history of an image
diff CONTAINER Inspect changes to files or directories on a container's filesystem
commit CONTAINER Create a new image from a container's changes; optionally, [REPOSITORY[:TAG]] may be added after CONTAINER
pull IMAGE Pull an image or a repository from a registry
push NAME[:TAG] Push an image or a repository to a registry
save CONTAINER Save one or more images to a tar archive (streamed to STDOUT by default)
load CONTAINER Load an image from a tar archive or STDIN
export CONTAINER Export a container's filesystem as a tar archive
import CONTAINER Import the contents from a tarball to create a filesystem image
image Manage images
ls [-f, --filter filter] Subcommand to list images
rm [-f, --force] Subcommand to remove one or more images
prune [-f, --force | --filter] Subcommand to remove unused images
images Synonym for image ls
rmi Synonym for image rm


Docker volumes are directories that are not part of the container's UFS — they are just normal directories on the host that are 'bind mounted' into the container.

There are three different ways to initialise volumes.

In the first, volumes can be defined at runtime with the -v CONTAINER_PATH flag. Any files the image held inside the specified directory will be copied into the volume; the docker inspect -f {{.Mounts}} CONTAINER_NAME command can be used to find out where the volume lives on the host computer.

The second way to set up a volume is by using the VOLUME instruction in a Dockerfile. If permissions and ownership on a volume or its files need to be set, it is important to note that any instruction executed after the VOLUME instruction in a Dockerfile will not be able to make changes to that volume.

The third way is to extend the -v argument to docker run with an explicit directory to bind to on the host using the format -v HOST_DIR:CONTAINER_DIR. This can't be done from a Dockerfile, as it would be non-portable and could pose a security risk.


The -v HOST_DIR:CONTAINER_DIR syntax is very useful for sharing files between the host and one or more containers. We can also share data between containers by using the --volumes-from CONTAINER argument with docker run.

A common practice is to create data containers — containers whose sole purpose is to share data between other containers. The main benefit of this approach is that it provides a handy namespace for volumes that can be easily loaded using the --volumes-from command.

For example, we can create a data container for a PostgreSQL database with the following command:

$ docker run --name dbdata postgres echo “Data-only container for postgres”

This will create a container from the postgres image and initialise any volumes defined in the image before running the echo command and exiting. (There's no need to leave data containers running, since doing so would just be a waste of resources.)

We can then use this volume from other containers with the --volumes-from argument. For example:

$ docker run -d --volumes-from dbdata --name db1 postgres

There's normally no need to use a 'minimal image' such as busybox or scratch for the data container. Just use the same image that is used for the container consuming the data. Using the same image doesn't take up any extra space — you must already have downloaded or created the image for the consumer; it also gives the image a chance to seed the container with any initial data and ensures permissions are set up correctly.


From https://docs.docker.com/network, Docker ships, by default, with the following core networking drivers:

  • bridge, the default network driver. If you don’t specify a driver, this is the type of network you are creating; bridge networks are usually used when your applications run in standalone containers that need to communicate
  • host, for standalone containers. Removes network isolation between the container and the Docker host, and allows the container to use the host's networking directly
  • overlay, which connects multiple Docker daemons together and enable swarm services to communicate with each other; it can also be used to facilitate communication between a swarm service and a standalone container, or between two standalone containers on different Docker daemons. This strategy removes the need to do OS-level routing between these containers
  • ipvlan, which gives users total control over both IPv4 and IPv6 addressing; the VLAN driver builds on top of that in giving operators complete control of layer 2 VLAN tagging and even IPvlan L3 routing
  • macvlan, which allows MAC addresses to be assigned to a container, making it appear as a physical device on the network; the Docker daemon routes traffic to containers by their MAC addresses. The macvlan driver is sometimes the best choice when dealing with legacy applications that expect to be directly connected to the physical network, rather than routed through the Docker host's network stack
  • none, which does exactly what it says on the tin

Rather than going into the specifics of the different network drivers here, it's probably enough to note the following, and to save a more in-depth look at specific networking considerations for when we look at Docker Compose, Docker Swarm and/or the containerisation of system emulators:

  • User- and/or Compose-defined bridge networks are best used when multiple containers need to communicate with each other on the same Docker host;
  • overlay networks are best used when containers running on different Docker hosts need to communicate with each other, or when multiple applications work together using swarm services; and
  • macvlan networks are best used when migrating from a VM setup, or when containers need to look like physical hosts on a network, each with a unique MAC address.


When working with Docker, we have the following terminology regarding images:

  • REGISTRY: a service responsible for hosting and distributing images; the default registry is the Docker Hub
  • REPOSITORY: a collection of related images, usually providing different versions of the same application or service
  • TAG: an alpha-numeric identifier attached to images within a repository

There are also three namespaces, when it comes to Docker images:

  • 'Official' namespaces contain neither prefixes nor any '/'s; these belong to the 'root' namespace, which is controller by Docker Inc.
  • 'User' namespaces contain a 'username/' prefix
  • Names prefixed with a hostname or IP address refer to images hosted on 'private' repositories
Dockerfile
FROM image:tag
MAINTAINER Maintainer Name <maintainer@email.address>
 
# copy files from the build-host into the container's FS
COPY hostfile containerfile
 
# since each RUN directive results in the creation of a new FS layer, they are often combined together
RUN  cmd_1 && cmd_2 && cmd_3 && ...
 
# specifies the container's default entry point; in order to handle CL arguments, a script is often used
# which might have the form:
#   #!/bin/bash
#   if [ $# -eq 0 ]; then
#     # default action(s) to be invoked in the absence of CL args
#   else
#     # process the args
#   fi
ENTRYPOINT ["/path/to/entry_cmd"]


ADD

Copies files from the build context or remote URLs into the image. If an archive file is added from a local path, it will automatically be unpacked.

CMD

Runs the given instruction when the container is started. If an ENTRYPOINT has been defined, the instruction will be interpreted as an argument to the ENTRY POINT (in this case, make sure you use the exec format). The CMD instruction is overridden by any arguments to docker run after the image name. Only the last CMD instruction will have an effect, and any previous CMD instructions will be overridden (including those in base images).

COPY

Used to copy files from the build context into the image. It has two forms, COPY src dest and COPY [“src”, “dest”], both of which copy the file or directory at src in the build context to dest inside the container. The JSON array format is required if the paths have spaces in them.

ENTRYPOINT

Sets an executable (and default arguments) to be run when the container starts. Any CMD instructions or arguments to docker run after the image name will be passed as parameters to the executable. ENTRYPOINT instructions are often used to provide “starter” scripts that initialise variables and services before interpreting any given arguments.

ENV

Sets environment variables inside the image. These can be referred to in subsequent instructions. The variables will also be available inside the image.

EXPOSE

Indicates to Docker that the container will have a process listening on the given port or ports. This information is used by Docker when linking containers or publishing ports by supplying the -P argument to docker run; by itself, EXPOSE will not affect networking.

FROM

The FROM directive specifies which image will serve as the basis of this new one. It is also possible to go ultra minimal and ship images with only binaries; to do this, once can inherit from the special scratch image (a completely blank filesystem) and simply copies one's binaries in and set an appropriate CMD instruction.

Another interesting choice of base image is phusion/baseimage-docker, which was created in reaction to the official Ubuntu image, which they claim is missing several essential services, including the init, cron and ssh daemons.

MAINTAINER

Sets the 'Author' metadata on the image to the given string. You can retrieve this with docker inspect -f {{.Author}} IMAGE.

ONBUILD

Specifies an instruction to be executed later, when the image is used as the base layer to another image. This can be useful for processing data that will be added in a child image.

RUN

Runs the given instruction inside the container, and commits the result.

USER

Sets the user (by name or UID) to use in any subsequent RUN, CMD, or ENTRYPOINT instructions. Note that UIDs are the same between the host and container, but usernames may be assigned to different UIDs, which can make things tricky when setting permissions.

VOLUME

Declares the specified file or directory to be a volume. If the file or directory already exists in the image, it will copied into the volume when the container is started. If multiple arguments are given, they are interpreted as multiple volume statements. You cannot specify the host directory for a volume inside a Dockerfile for portability and security reasons.

WORKDIR

Sets the working directory for any subsequent RUN, CMD, ENTRYPOINT, ADD, or COPY instructions. Can be used multiple times. Relative paths may be used and are resolved relative to the previous WORKDIR.

From docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact:

Both CMD and ENTRYPOINT instructions define what command gets executed when running a container. There are few rules that describe their co-operation:

  1. Dockerfile should specify at least one of CMD or ENTRYPOINT commands
  2. ENTRYPOINT should be defined when using the container as an executable
  3. CMD should be used as a way of defining default arguments for an ENTRYPOINT command or for executing an ad-hoc command in a container
  4. CMD will be overridden when running the container with alternative arguments

The table below shows what command is executed for different ENTRYPOINT / CMD combinations:

No ENTRYPOINT ENTRYPOINT exec_entry p1_entry ENTRYPOINT [“exec_entry”, “p1_entry”]
No CMD error, not allowed /bin/sh -c exec_entry p1_entry exec_entry p1_entry
CMD [“exec_cmd”, “p1_cmd”] exec_cmd p1_cmd /bin/sh -c exec_entry p1_entry exec_entry p1_entry exec_cmd p1_cmd
CMD [“p1_cmd”, “p2_cmd”] p1_cmd p2_cmd /bin/sh -c exec_entry p1_entry exec_entry p1_entry p1_cmd p2_cmd
CMD exec_cmd p1_cmd /bin/sh -c exec_cmd p1_cmd /bin/sh -c exec_entry p1_entry exec_entry p1_entry /bin/sh -c exec_cmd p1_cmd

Note: if CMD is defined from the base image, setting ENTRYPOINT will reset CMD to an empty value; in this scenario, CMD must be defined in the current image to have a value.

By default, when you docker push an image to a repository, it will overwrite any pre-existing images with the specified tag, even if it was generated for a different architecture. Apparently docker buildx can streamline the process of generating multi-architecture images, however a more manual process (which definitely doesn't require a cross-platform build environment) involves the use of docker manifest.

# AMD64
$ docker build -t user/repository:tag-amd64 --build-arg ARCH=amd64/ .
$ docker push     user/repository:tag-amd64

# ARMV5
$ docker build -t user/repository:tag-armv5 --build-arg ARCH=armv5/ .
$ docker push     user/repository:tag-armv5

# ARMV7
$ docker build -t user/repository:tag-armv7 --build-arg ARCH=armv7/ .
$ docker push     user/repository:tag-armv7

$ docker manifest create user/repository:tag \
--amend user/repository:tag-amd64 \
--amend user/repository:tag-armv5 \
--amend user/repository:tag-armv7
$ docker manifest push user/repository:tag


From https://docs.docker.com/compose/compose-file/compose-file-v3/, the Compose file is a YAML file defining services, networks and volumes. The default path for a Compose file is ./docker-compose.yml.

A service definition contains configuration that is applied to each container started for that service, much like passing command-line parameters to docker run; likewise, network and volume definitions are analogous to docker network create and docker volume create.

As with docker run, options specified in the Dockerfile, such as CMD, EXPOSE, VOLUME, ENV, are respected by default — you don’t need to specify them again in docker-compose.yml.

You can use environment variables in configuration values with a Bash-like ${VARIABLE} syntax, too.

docker-compose.yml
version: '3'
services:
  container_1:
    image: image_2_name:tag
    ports:
     - host_port:container_port
    environment:
     - ENV_VAR_1=val1
     - ENV_VAR_2=val2 
  container_2:
    image: image_2_name:tag
    ports:
     - host_port:container_port
    environment:
     - ENV_VAR_1=val1

For the definitive list of Compose-supported commands, see https://docs.docker.com/compose/compose-file/compose-file-v3, however what follows is a brief overview of the main configuration options:


build

Configuration options that are applied at build time; sub-options include context, dockerfile, args, labels, target and network, the last of which determines the network containers connect to for the RUN instructions during build.

cap_add and cap_drop

Add or drop container capabilities. See man 7 capabilities for a full list.

container_name

Specify a custom container name, rather than a generated default name. Because Docker container names must be unique, you cannot scale a service beyond one container if you have specified a custom name; attempting to do so results in an error. The container_name option is ignored when deploying a stack in swarm mode.

depends_on

Express dependency between services. Service dependencies cause the following behaviours:

  • docker-compose up starts services in dependency order;
  • docker-compose up SERVICE automatically includes SERVICE's dependencies;
  • docker-compose stop stops services in dependency order.

Note that depends_on does not actually wait for the containerised services to be ready, just for their encapsulating containers to have started.

deploy

Specify configuration related to the deployment and running of services. This only takes effect when deploying to a swarm with docker stack deploy, and is ignored by docker-compose up and docker-compose run.

Sub-options include endpoint_mode (being vip for virtual IP or dnsrr for DNS round-robin), labels, mode (either global for exactly one container per swarm node, or replicated for a specified number of containers), placement and resources (to specify constrains on allowed nodes), replicas and max_replicas_per_node, restart_policy

devices

List of device mappings. Uses the same format as the --device docker client create option.

Custom DNS servers and search domains. Can be a single value or a list.

entrypoint

Override the default entrypoint; this maye also be a list, in a manner similar to Dockerfile

env_file

Add environment variables from a file; can be a single value or a list. If you have specified a Compose file with docker-compose -f FILE, paths in env_file are relative to the directory that file is in.

environment

Add environment variables; can be either an array or a dictionary. Any boolean values (true, false, yes, no) need to be enclosed in quotes to ensure they are not converted to True or False by the YML parser.

Environment variables with only a key are resolved to their values on the machine Compose is running on, which can be helpful for secret or host-specific values.

expose

Expose ports without publishing them to the host machine — they’ll only be accessible to linked services. Only the internal port can be specified.

Link to containers started outside this docker-compose.yml or even outside of Compose, especially for containers that provide shared or common services. external_links follow semantics similar to the legacy option links when specifying both the container name and the link alias.

extra_hosts

Add hostname mappings; use the same values as the docker client --add-host parameter.

healthcheck

Configure a check that's run to determine whether or not containers for this service are 'healthy'.

image

Specify the image to start the container from; can either be a repository/tag or a partial image ID.

init

Run an init inside the container that forwards signals and reaps processes; set this option to true to enable this feature for the service.

labels

Add metadata to containers using Docker labels; you can use either an array or a dictionary. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software.

Link to containers in another service; either specify both the service name and a link alias (“SERVICE:ALIAS”), or just the service name.

logging

Logging configuration for the service.

network_mode

Network mode; use the same values as the docker client --network parameter, plus the special form service:[service name]. This option is ignored when deploying a stack in swarm mode.

networks

Networks to join, referencing entries under the top-level networks key.

ports

Expose ports. Note: port mapping is incompatible with network_mode: host, and docker-compose run ignores ports unless you include --service-ports.

profiles

profiles defines a list of named profiles for the service to be enabled under. When not set, the service is always enabled; for the services that make up your core application you should omit profiles so they will always be started.

restart

no is the default restart policy, and it does not restart a container under any circumstance. When always is specified, the container always restarts. The on-failure policy restarts a container if the exit code indicates an on-failure error. unless-stopped always restarts a container, except when the container is stopped (manually or otherwise).

The restart option is ignored when deploying a stack in swarm mode.

secrets

Grant access to secrets on a per-service basis using the per-service secrets configuration.

The short syntax variant only specifies the secret name; this grants the container access to the secret and mounts it at /run/secrets/<secret_name> within the container.

The long syntax provides more granularity in how the secret is created within the service's task containers, and its options include source and target, and uid, gid and mode.

security_opt

Override the default labelling scheme for each container.

sysctls

Kernel parameters to set in the container; may be either an array or a dictionary.

ulimits

Override the default ulimits for a container; may be specified either a single limit, or as soft/hard limits.

volumes

Mount host paths or named volumes, specified as sub-options to a service.

The short syntax uses the generic [SOURCE:]TARGET[:MODE] format, where SOURCE can be either a host path or volume name. TARGET is the container path where the volume is mounted. Standard modes are ro for read-only and rw for read-write (default).

The long form syntax allows the configuration of additional fields that can't be expressed in the short form. Options include type, source and target, read_only, bind, volume and tmpfs

The following commands are commonly used when working with Compose. Most are self-explanatory and have direct Docker equivalents, but it's worth being aware of them. Note that the following flags can be specified before the docker-compose commands to override the default expected compose filename and project paths:

  • -f, --file FILE : specify an alternate compose file (default: docker-compose.yml)
  • -p, --project-name NAME : specify an alternate project name (default: directory name)


Command Description
up Starts all the containers defined in the Compose file and aggregates the log output; normally you will want to use the -d argument to run Compose in the background.
build Rebuilds any images created from Dockerfiles; the up command will not build an image unless it doesn't exist, so use this command whenever you need to update an image.
ps Provides information on the status of containers managed by Compose
run Spins up a container to run a one-off command. This will also spin up any linked containers unless the --no-deps argument is given.
logs Outputs coloured and aggregated logs for the Compose-managed containers
stop Stops containers without removing them
rm Removes stopped containers; Remember to use the -v argument to remove any Docker-managed volumes

A normal workflow begins with calling docker-compose up -d to start the application. The docker-compose logs and ps commands can be used to verify the status of the application and help debugging.

After changes to the code, call docker-compose build followed by docker-compose up -d. This will build the new image and replace the running container. Note that Compose will preserve any old volumes from the original containers, which means that databases and caches persist over containers (this can be confusing, so be careful).

If you don't need a new image but have modified the Compose YAML, calling up -d will cause Compose to replace the container with one with the new settings. If you want to force Compose to stop and recreate all the containers, use the --force-recreate flag.

When you're finished with the application, calling docker-compose stop will halt the application. The same containers will be restarted if docker-compose start or up is called, assuming no code has changed. Use docker-compose rm to get rid of them completely.


  • Last modified: 2022-06-16 07:17
  • by Peter