Dockerizing your Application

General Description

The Docker Webfarm is a general purpose web service for multiple clients. Like the other Java and Ruby webfarms, Applications that are deployed will be fronted by the F5 load balancers and they can be connected to the NetApp NFS Volumes for persistent storage. Applications are deployed using Jenkins and DockerFile. Each application deployment is based on the stable branch of the Docker Community Edition. Clustering and orchestration capabilities for Docker Engine is provided by the implementation of Docker Swarm. 

Access

As this is a shared environment developers are not given access to any of the Docker nodes. Containers are not persistent and hence the shell access to any of them is also blocked. Instead to review the logs, developers are expected to log into GrayLog. Configurations can be passed through the application source code or can be added during the application startup process. Secret files and Persistent Volumes will be passed during the application/container startup process. [More on this below]

System Diagram

High level design of the stages involved in the container startup process. 

Dockerizing an Application

Different types of application (i.e tomcat, ruby) will have unique workflows involving how parameters are ingested during the container image build and deploy steps.  Depending on the type of application being built, you may need to provide additional information.

Docker File

This is the driving force for building any image. It contains all the instructions required to import the source code and start the application. A complete information can be found at Docker File.  The Docker File should reside on a Git repository in which the Jenkins user (s_ci) has access to pull from. 

Base Image

It is recommended to use an official image from the Docker Hub that is being updated regularly. Use the alpine version of the image to reduce the size & vulnerabilities drastically. You need to add the Deco params (described below) and the code in the Docker File before adding the actual application code - This is needed to successfully process any secret files. 

Images are stored in the container registry Harbor.  Access to projects/images within Harbor is applied based on Grouper group membership.  Harbor scans images that are uploaded upon initial push and reports are emailed weekly to the ISO team on vulnerabilities found on the container image. There is also functionality to restrict an image from being deployed if it contains customizable level of vulnerability.

Deco Params

DECO, short for Docker Environment Control is a home grown application which is being used to process any secrets before starting the application. The purpose of the Deco/Secrets file is to contain the necessary parameters that needs to be injected into the application config files. This takes a JSON file as input as shown below in the sample, where each level after 'filters' is a different file that needs to be injected with the variables provided. This should contain a Key: Value pairs for passwords, database url and any other such secure variables. More information about deco can be found here. 

{
    "filters": {
        "/apps/tomcat/8.5.34/conf/server.xml": {
            "httpPort": "8080",
            "httpProtocol": "org.apache.coyote.http11.Http11NioProtocol"
        },
        "path/to/file2": {
            "replace": "withThis"
        }
    }
}

For the files that needs injection, the place at which the replacement should happen must adhere to the Deco Format. Variable must be replaced with the syntax {{ .Key }}. An example is provided below.

<Connector port="{{ .httpPort }}" protocol="{{ .httpProtocol }}"
               connectionTimeout="20000"
               redirectPort="8443" />

If you are using a public Docker image, the following deco arguments and the tool needs to be added in to the Docker file before adding the application code.

#Deco Args
ARG DECO_VERSION=0.3.1
ARG DECO_OS=linux
ARG DECO_ARCH=amd64

#Add Deco
ADD https://github.com/YaleUniversity/deco/releases/download/v${DECO_VERSION}/deco-v${DECO_VERSION}-${DECO_OS}-${DECO_ARCH} /usr/local/bin/deco
RUN chmod 555 /usr/local/bin/deco && deco version

Application Source Code

This step adds the application source code to the image. 

For a Tomcat/Java based application, the war file can be added using the 'ADD' object and pointing it to the location in the artifactory as shown here. 

ADD https://<Artifactory location for the war file> $CATALINA_HOME/webapps/<fileName>.war

For a Ruby application since the source code is not packaged into a gem file, Jenkins user will (During the image build process) clone the source code into an 'src' directory and Docker File should ADD the code from here. An example is as shown here

ADD src/ ${WORKDIR}

Entrypoint Script

This, when defined in a Dockerfile will be used to execute the wrapper script that starts the application. This will run any additional steps required by the application and also execute Deco, which will process the secret file and update any configuration files. 

#!/bin/sh

# variables
secret_file=/run/secrets/deco.json

# process deco file
deco validate $secret_file || exit 1
deco run $secret_file

#Start the App
catalina.sh run

DockerFile Tips

Apart from making use of regular Docker File Objects, you can use many other options that can be found here. 

• HEALTHCHECK object tells docker how to test a container to check that it is still working. This is recommended to use to detect any cases where a container is stuck in an infinite loop and cannot take any new connections.  This function also has the ability to restart a container that fails the HEALTHCHECK automatically.
• LABEL object can be used as a descriptor for the application. (To store details about the Maintainer, Owner and others)
• While declaring environment variables using ENV/ARG objects, multiple variables can be declared in a single line or by using the next line char '\'. This will help reduce the overall size of an Image. 
• EXPOSE can be used to open any ports on the container. 

Sample Docker File

Here is a sample Docker file making use of a public tomcat image

FROM tomcat:8-alpine
LABEL Email=Yale.ITS@Yale.edu

#Deco Args
ARG DECO_VERSION=0.3.1
ARG DECO_OS=linux
ARG DECO_ARCH=amd64

#Add Deco to the Image (Must)
ADD https://github.com/YaleUniversity/deco/releases/download/v${DECO_VERSION}/deco-v${DECO_VERSION}-${DECO_OS}-${DECO_ARCH} /usr/local/bin/deco
RUN chmod 555 /usr/local/bin/deco && deco version

#Add App
ADD https://tomcat.apache.org/tomcat-8.5-doc/appdev/sample/sample.war /usr/local/tomcat/webapps

COPY entrypoint.sh /
RUN chmod 700 /entrypoint.sh

EXPOSE 8080
ENTRYPOINT ["/entrypoint.sh"]

#!/bin/bash

# variables
secret_file=/run/secrets/deco.json

# process deco file
deco validate $secret_file || exit 1
deco run $secret_file

#Start the App
catalina.sh run

Changes to be made in the Source Code

To Dockerize your application there are only two places that needs to be changed

• For files that need to be injected with the parameters, the place at which this needs to be happened should adhere to the above described Deco Format. 
• Since no application container is persistent, storing any log files in them will be of no use. Instead pointing them to stdout will help docker daemon to pick the logs and push it to the Graylog. Recommended method is to replace the destination file with '/proc/1/fd/1'. Redirecting the output to stdout of PID 1 is recommended as PID 1 is the process launched by the Docker daemon and its stdout will be what docker picks up

Handover to Sys Admins

Once the above files are successfully tested on your local machine, provide the following details to start the process

• Git repository which Dockerfile resides - this needs to be readable by the s_ci Jenkins user.
  
  • for Ruby apps, if the application source is in a different repository than the Dockerfile, provide that as well

• Secrets file (Deco formatted) - Must be sent over via Secure File Transfer. If the secrets are different between the environment, provide one for each. 
• Preferred Frontend URL.
• Expected CPU and RAM consumption. 
• Building Image
  • Image Name = Application Name
  • Tag = <DEV/TST/PRD>
  • Base Image that you are using.
  • Version Number
• COA - Charging Account. 
• DR Tier. 
• Owner Department. 
• Owner Department Contact. 
• Support Department. 
• Support Department Contact. 
• Environment. 

Build/Deploy

After SysAdmins construct the workflow for the application, developers will have access to build their own images and deploy to development and test environments.

It is recommended to match the container image VERSION_NUM field with the application version number.

Build

Initial image will be built by SysAdmin and available to view in Harbor. Images are stored under ‘Projects’ organized by group name.  When an image is built it is tagged twice, once for ‘latest’ and tagged with the “VERSION_NUM” field and are appended with the build id that corresponds to the Jenkins Build job run number. This can be seen in the console output when the build job is run (this is used during the Deploy process "IMAGE_TAG").  Note: Because of this, builds will not overwrite one another (with exception of ‘latest’).

The APP_REPO and APP_BRANCH fields allow control of the Dockerfile; be sure to allow the Jenkins user s_ci access to read.

Match the BASE_IMAGE parameter with what is provided within the Dockerfile, this is necessary for cleanup during the build process.

Deploy

Access to deploy dev and test container images is allowed. Supply the Image Tag (can be verified through the project in Harbor or from the build job) that needs to be deployed.  Note: if the container image is marked as vulnerable within Harbor, the image will not be able to be pulled.

Do I need a rebuild or redeploy?

Information for application SDR:

Please reach out to a Linux SysAdmin to get information that you might need for SDR on your application. [Note for admins: This can be found here. Please extract a .Pdf copy and send it to the application owners.]