Docker

For information on how to install Docker, see the official installation information.

Taking a high-level view, Docker consists of the following major components:

• the Docker daemon, which is responsible for creating, running, and monitoring containers, as well as building and storing images;
• the Docker client, which is used to talk to the Docker; by default, this happens over a Unix domain socket, but it can also use a TCP socket to enable remote clients or a file descriptor for systemd-managed sockets;
• Docker registries, which store and distribute images; the default registry is the Docker Hub, which hosts public images, however private registries can also be used to store sensitive images as well as to locally cache images.

At a more fundamental level, Docker is a containerisation system which uses a few key features of the Linux kernel:

• cgroups, which are responsible for managing resources used by a container; they are also responsible for freezing and unfreezing containers, as used in the docker pause functionality;
• namespaces are responsible for isolating containers, making sure that a container's filesystem, hostname, users, networking, and processes are separated from the rest of the system;
• the Union File System (UFS), used to store the layers for containers; the UFS is provided by one of several storage drivers, either AUFS, devicemapper, BTRFS, or Overlay.

The following 'supporting technologies' are included with Docker and serve to extend or expand upon its basic functionality:

• Docker Machine installs and configures Docker hosts on local or remote resources. Machine also configures the Docker client, making it easy to swap between environments;
• Swarm, Docker's clustering solution; swarm can group together several Docker hosts, allowing the user to treat them as a unified resource; and
• Compose, a tool for building and running applications composed of multiple Docker containers. It is primarily used in development and testing rather than production.

With reference to the container life cycle (with the above illustration originally from here):

Docker volumes are directories that are not part of the container's UFS — they are just normal directories on the host that are 'bind mounted' into the container.

There are three different ways to initialise volumes.

In the first, volumes can be defined at runtime with the -v CONTAINER_PATH flag. Any files the image held inside the specified directory will be copied into the volume; the docker inspect -f {{.Mounts}} CONTAINER_NAME command can be used to find out where the volume lives on the host computer.

The second way to set up a volume is by using the VOLUME instruction in a Dockerfile. If permissions and ownership on a volume or its files need to be set, it is important to note that any instruction executed after the VOLUME instruction in a Dockerfile will not be able to make changes to that volume.

The third way is to extend the -v argument to docker run with an explicit directory to bind to on the host using the format -v HOST_DIR:CONTAINER_DIR. This can't be done from a Dockerfile, as it would be non-portable and could pose a security risk.

The -v HOST_DIR:CONTAINER_DIR syntax is very useful for sharing files between the host and one or more containers. We can also share data between containers by using the --volumes-from CONTAINER argument with docker run.

A common practice is to create data containers — containers whose sole purpose is to share data between other containers. The main benefit of this approach is that it provides a handy namespace for volumes that can be easily loaded using the --volumes-from command.

For example, we can create a data container for a PostgreSQL database with the following command:

$ docker run --name dbdata postgres echo “Data-only container for postgres”

This will create a container from the postgres image and initialise any volumes defined in the image before running the echo command and exiting. (There's no need to leave data containers running, since doing so would just be a waste of resources.)

We can then use this volume from other containers with the --volumes-from argument. For example:

$ docker run -d --volumes-from dbdata --name db1 postgres

There's normally no need to use a 'minimal image' such as busybox or scratch for the data container. Just use the same image that is used for the container consuming the data. Using the same image doesn't take up any extra space — you must already have downloaded or created the image for the consumer; it also gives the image a chance to seed the container with any initial data and ensures permissions are set up correctly.

From https://docs.docker.com/network, Docker ships, by default, with the following core networking drivers:

• bridge, the default network driver. If you don’t specify a driver, this is the type of network you are creating; bridge networks are usually used when your applications run in standalone containers that need to communicate
• host, for standalone containers. Removes network isolation between the container and the Docker host, and allows the container to use the host's networking directly
• overlay, which connects multiple Docker daemons together and enable swarm services to communicate with each other; it can also be used to facilitate communication between a swarm service and a standalone container, or between two standalone containers on different Docker daemons. This strategy removes the need to do OS-level routing between these containers
• ipvlan, which gives users total control over both IPv4 and IPv6 addressing; the VLAN driver builds on top of that in giving operators complete control of layer 2 VLAN tagging and even IPvlan L3 routing
• macvlan, which allows MAC addresses to be assigned to a container, making it appear as a physical device on the network; the Docker daemon routes traffic to containers by their MAC addresses. The macvlan driver is sometimes the best choice when dealing with legacy applications that expect to be directly connected to the physical network, rather than routed through the Docker host's network stack
• none, which does exactly what it says on the tin

Rather than going into the specifics of the different network drivers here, it's probably enough to note the following, and to save a more in-depth look at specific networking considerations for when we look at Docker Compose, Docker Swarm and/or the containerisation of system emulators:

• User- and/or Compose-defined bridge networks are best used when multiple containers need to communicate with each other on the same Docker host;
• overlay networks are best used when containers running on different Docker hosts need to communicate with each other, or when multiple applications work together using swarm services; and
• macvlan networks are best used when migrating from a VM setup, or when containers need to look like physical hosts on a network, each with a unique MAC address.

When working with Docker, we have the following terminology regarding images:

• REGISTRY: a service responsible for hosting and distributing images; the default registry is the Docker Hub
• REPOSITORY: a collection of related images, usually providing different versions of the same application or service
• TAG: an alpha-numeric identifier attached to images within a repository

There are also three namespaces, when it comes to Docker images:

• 'Official' namespaces contain neither prefixes nor any '/'s; these belong to the 'root' namespace, which is controller by Docker Inc.
• 'User' namespaces contain a 'username/' prefix
• Names prefixed with a hostname or IP address refer to images hosted on 'private' repositories

Dockerfile

FROM image:tag
MAINTAINER Maintainer Name <maintainer@email.address>
 
# copy files from the build-host into the container's FS
COPY hostfile containerfile
 
# since each RUN directive results in the creation of a new FS layer, they are often combined together
RUN  cmd_1 && cmd_2 && cmd_3 && ...
 
# specifies the container's default entry point; in order to handle CL arguments, a script is often used
# which might have the form:
#   #!/bin/bash
#   if [ $# -eq 0 ]; then
#     # default action(s) to be invoked in the absence of CL args
#   else
#     # process the args
#   fi
ENTRYPOINT ["/path/to/entry_cmd"]

Copies files from the build context or remote URLs into the image. If an archive file is added from a local path, it will automatically be unpacked.

Runs the given instruction when the container is started. If an ENTRYPOINT has been defined, the instruction will be interpreted as an argument to the ENTRY POINT (in this case, make sure you use the exec format). The CMD instruction is overridden by any arguments to docker run after the image name. Only the last CMD instruction will have an effect, and any previous CMD instructions will be overridden (including those in base images).

Used to copy files from the build context into the image. It has two forms, COPY src dest and COPY [“src”, “dest”], both of which copy the file or directory at src in the build context to dest inside the container. The JSON array format is required if the paths have spaces in them.

Sets an executable (and default arguments) to be run when the container starts. Any CMD instructions or arguments to docker run after the image name will be passed as parameters to the executable. ENTRYPOINT instructions are often used to provide “starter” scripts that initialise variables and services before interpreting any given arguments.

Sets environment variables inside the image. These can be referred to in subsequent instructions. The variables will also be available inside the image.

Indicates to Docker that the container will have a process listening on the given port or ports. This information is used by Docker when linking containers or publishing ports by supplying the -P argument to docker run; by itself, EXPOSE will not affect networking.

The FROM directive specifies which image will serve as the basis of this new one. It is also possible to go ultra minimal and ship images with only binaries; to do this, once can inherit from the special scratch image (a completely blank filesystem) and simply copies one's binaries in and set an appropriate CMD instruction.

Another interesting choice of base image is phusion/baseimage-docker, which was created in reaction to the official Ubuntu image, which they claim is missing several essential services, including the init, cron and ssh daemons.

Sets the 'Author' metadata on the image to the given string. You can retrieve this with docker inspect -f {{.Author}} IMAGE.

Specifies an instruction to be executed later, when the image is used as the base layer to another image. This can be useful for processing data that will be added in a child image.

Runs the given instruction inside the container, and commits the result.

Sets the user (by name or UID) to use in any subsequent RUN, CMD, or ENTRYPOINT instructions. Note that UIDs are the same between the host and container, but usernames may be assigned to different UIDs, which can make things tricky when setting permissions.

Declares the specified file or directory to be a volume. If the file or directory already exists in the image, it will copied into the volume when the container is started. If multiple arguments are given, they are interpreted as multiple volume statements. You cannot specify the host directory for a volume inside a Dockerfile for portability and security reasons.

Sets the working directory for any subsequent RUN, CMD, ENTRYPOINT, ADD, or COPY instructions. Can be used multiple times. Relative paths may be used and are resolved relative to the previous WORKDIR.

From docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact:

Both CMD and ENTRYPOINT instructions define what command gets executed when running a container. There are few rules that describe their co-operation:

1. Dockerfile should specify at least one of CMD or ENTRYPOINT commands
2. ENTRYPOINT should be defined when using the container as an executable
3. CMD should be used as a way of defining default arguments for an ENTRYPOINT command or for executing an ad-hoc command in a container
4. CMD will be overridden when running the container with alternative arguments

The table below shows what command is executed for different ENTRYPOINT / CMD combinations:

Note: if CMD is defined from the base image, setting ENTRYPOINT will reset CMD to an empty value; in this scenario, CMD must be defined in the current image to have a value.

By default, when you docker push an image to a repository, it will overwrite any pre-existing images with the specified tag, even if it was generated for a different architecture. Apparently docker buildx can streamline the process of generating multi-architecture images, however a more manual process (which definitely doesn't require a cross-platform build environment) involves the use of docker manifest.

# AMD64
$ docker build -t user/repository:tag-amd64 --build-arg ARCH=amd64/ .
$ docker push     user/repository:tag-amd64

# ARMV5
$ docker build -t user/repository:tag-armv5 --build-arg ARCH=armv5/ .
$ docker push     user/repository:tag-armv5

# ARMV7
$ docker build -t user/repository:tag-armv7 --build-arg ARCH=armv7/ .
$ docker push     user/repository:tag-armv7

$ docker manifest create user/repository:tag \
--amend user/repository:tag-amd64 \
--amend user/repository:tag-armv5 \
--amend user/repository:tag-armv7
$ docker manifest push user/repository:tag

From https://docs.docker.com/compose/compose-file/compose-file-v3/, the Compose file is a YAML file defining services, networks and volumes. The default path for a Compose file is ./docker-compose.yml.

A service definition contains configuration that is applied to each container started for that service, much like passing command-line parameters to docker run; likewise, network and volume definitions are analogous to docker network create and docker volume create.

As with docker run, options specified in the Dockerfile, such as CMD, EXPOSE, VOLUME, ENV, are respected by default — you don’t need to specify them again in docker-compose.yml.

You can use environment variables in configuration values with a Bash-like ${VARIABLE} syntax, too.

docker-compose.yml

version: '3'
services:
  container_1:
    image: image_2_name:tag
    ports:
     - host_port:container_port
    environment:
     - ENV_VAR_1=val1
     - ENV_VAR_2=val2 
  container_2:
    image: image_2_name:tag
    ports:
     - host_port:container_port
    environment:
     - ENV_VAR_1=val1

For the definitive list of Compose-supported commands, see https://docs.docker.com/compose/compose-file/compose-file-v3, however what follows is a brief overview of the main configuration options:

Configuration options that are applied at build time; sub-options include context, dockerfile, args, labels, target and network, the last of which determines the network containers connect to for the RUN instructions during build.

Add or drop container capabilities. See man 7 capabilities for a full list.

Specify a custom container name, rather than a generated default name. Because Docker container names must be unique, you cannot scale a service beyond one container if you have specified a custom name; attempting to do so results in an error. The container_name option is ignored when deploying a stack in swarm mode.

Express dependency between services. Service dependencies cause the following behaviours:

• docker-compose up starts services in dependency order;
• docker-compose up SERVICE automatically includes SERVICE's dependencies;
• docker-compose stop stops services in dependency order.

Note that depends_on does not actually wait for the containerised services to be ready, just for their encapsulating containers to have started.

Specify configuration related to the deployment and running of services. This only takes effect when deploying to a swarm with docker stack deploy, and is ignored by docker-compose up and docker-compose run.

Sub-options include endpoint_mode (being vip for virtual IP or dnsrr for DNS round-robin), labels, mode (either global for exactly one container per swarm node, or replicated for a specified number of containers), placement and resources (to specify constrains on allowed nodes), replicas and max_replicas_per_node, restart_policy

List of device mappings. Uses the same format as the --device docker client create option.

Custom DNS servers and search domains. Can be a single value or a list.

Override the default entrypoint; this maye also be a list, in a manner similar to Dockerfile

Add environment variables from a file; can be a single value or a list. If you have specified a Compose file with docker-compose -f FILE, paths in env_file are relative to the directory that file is in.

Add environment variables; can be either an array or a dictionary. Any boolean values (true, false, yes, no) need to be enclosed in quotes to ensure they are not converted to True or False by the YML parser.

Environment variables with only a key are resolved to their values on the machine Compose is running on, which can be helpful for secret or host-specific values.

Expose ports without publishing them to the host machine — they’ll only be accessible to linked services. Only the internal port can be specified.

Link to containers started outside this docker-compose.yml or even outside of Compose, especially for containers that provide shared or common services. external_links follow semantics similar to the legacy option links when specifying both the container name and the link alias.

Add hostname mappings; use the same values as the docker client --add-host parameter.

Configure a check that's run to determine whether or not containers for this service are 'healthy'.

Specify the image to start the container from; can either be a repository/tag or a partial image ID.

Run an init inside the container that forwards signals and reaps processes; set this option to true to enable this feature for the service.

Add metadata to containers using Docker labels; you can use either an array or a dictionary. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software.

Link to containers in another service; either specify both the service name and a link alias (“SERVICE:ALIAS”), or just the service name.

Logging configuration for the service.

Network mode; use the same values as the docker client --network parameter, plus the special form service:[service name]. This option is ignored when deploying a stack in swarm mode.

Networks to join, referencing entries under the top-level networks key.

Expose ports. Note: port mapping is incompatible with network_mode: host, and docker-compose run ignores ports unless you include --service-ports.

profiles defines a list of named profiles for the service to be enabled under. When not set, the service is always enabled; for the services that make up your core application you should omit profiles so they will always be started.

no is the default restart policy, and it does not restart a container under any circumstance. When always is specified, the container always restarts. The on-failure policy restarts a container if the exit code indicates an on-failure error. unless-stopped always restarts a container, except when the container is stopped (manually or otherwise).

The restart option is ignored when deploying a stack in swarm mode.

Grant access to secrets on a per-service basis using the per-service secrets configuration.

The short syntax variant only specifies the secret name; this grants the container access to the secret and mounts it at /run/secrets/<secret_name> within the container.

The long syntax provides more granularity in how the secret is created within the service's task containers, and its options include source and target, and uid, gid and mode.

Override the default labelling scheme for each container.

Kernel parameters to set in the container; may be either an array or a dictionary.

Override the default ulimits for a container; may be specified either a single limit, or as soft/hard limits.

Mount host paths or named volumes, specified as sub-options to a service.

The short syntax uses the generic [SOURCE:]TARGET[:MODE] format, where SOURCE can be either a host path or volume name. TARGET is the container path where the volume is mounted. Standard modes are ro for read-only and rw for read-write (default).

The long form syntax allows the configuration of additional fields that can't be expressed in the short form. Options include type, source and target, read_only, bind, volume and tmpfs

The following commands are commonly used when working with Compose. Most are self-explanatory and have direct Docker equivalents, but it's worth being aware of them. Note that the following flags can be specified before the docker-compose commands to override the default expected compose filename and project paths:

• -f, --file FILE : specify an alternate compose file (default: docker-compose.yml)
• -p, --project-name NAME : specify an alternate project name (default: directory name)

A normal workflow begins with calling docker-compose up -d to start the application. The docker-compose logs and ps commands can be used to verify the status of the application and help debugging.

After changes to the code, call docker-compose build followed by docker-compose up -d. This will build the new image and replace the running container. Note that Compose will preserve any old volumes from the original containers, which means that databases and caches persist over containers (this can be confusing, so be careful).

If you don't need a new image but have modified the Compose YAML, calling up -d will cause Compose to replace the container with one with the new settings. If you want to force Compose to stop and recreate all the containers, use the --force-recreate flag.

When you're finished with the application, calling docker-compose stop will halt the application. The same containers will be restarted if docker-compose start or up is called, assuming no code has changed. Use docker-compose rm to get rid of them completely.

• The Official Docker homepage
• Docker Saigon (2016). Docker Internals: A Deep Dive Into Docker For Engineers Interested In The Gritty Details.
• Adrian Mouat (2016). Using Docker: Developing and Deploying Software with Containers. O'Reilly Media. ISBN: 9781491915769.
• Sean P. Kane and Karl Matthias (2018). Docker: Up & Running: Shipping Reliable Containers in Production, 2ed. O'Reilly Media. ISBN: 9781492036739.

• Docker-in-Docker
• github.com/collabnix/dockerlabs
  • dockerlabs.collabnix.com/beginners/saving-images-as-tar
• phusion.github.io/baseimage-docker
  • github.com/phusion/baseimage-docker
• mergeboard.com/blog/2-qemu-microvm-docker