TECH.insight

In-container testing for AEM with Docker

Monday 10 November 2014

Docker and AEM are a perfect pairing when it comes to quick and effective integration testing.

The key to Docker is that it does one thing and it does it well: it runs a single process and then finishes. You can continue where you left off or throw your work away and start again.

When it comes to integration or in-container testing, Docker really excels – and even more so when you add AEM to the mix. If you’ve ever tried to use the AEM quickstart application to run in-container tests as part of a Maven build, you’ll know how painfully slow and inconsistent it can be.

What if you could have an AEM instance in a clean state, ready and waiting to run at a moment’s notice, and have it ready in under 30 seconds – wouldn’t that be great?

Getting started

First thing to do is to build a container with CQ installed. Using a basic Dockerfile, we can script the minimum steps to set up a container. As an added bonus, the container includes the JaCoCo agent, which would enable us to get code coverage statistics.

FROM dockerfile/java:oracle-java7

RUN mkdir /opt/aem/

WORKDIR /opt/aem/

ADD ~/Documents/demo/6.0/cq-quickstart-6.0.jar /opt/aem/cq-quickstart-6.0.jar
ADD ~/Documents/demo/6.0/license.properties /opt/aem/license.properties

RUN java -jar cq-quickstart-6.0.jar -unpack -v

RUN mkdir /opt/aem/jacoco-0.7.1.201405082137

ADD ~/Documents/demo/jacoco-0.7.1.201405082137.zip \
    /opt/aem/jacoco-0.7.1.201405082137/jacoco-0.7.1.201405082137.zip

RUN unzip jacoco-0.7.1.201405082137/jacoco-0.7.1.201405082137.zip \
          lib/* \
          -d jacoco-0.7.1.201405082137/

ENV CQ_FOREGROUND y
ENV CQ_VERBOSE    y
ENV CQ_NOBROWSER  y
ENV CQ_RUNMODE    "dev,author,nosamplecontent"
ENV CQ_JVM_OPTS   "-server -Xmx1524M -Xms512M -XX:MaxPermSize=512M \
                   -javaagent:/opt/aem/jacoco-0.7.1.201405082137/lib/org.jacoco.agent-0.7.1.201405082137.jar=output=tcpserver,port=6300"


CMD crx-quickstart/bin/quickstart

The key point in this is that we are starting with a container that already has Java 7 installed for us, so we don’t need to do it. This is the beauty of Docker – you can build upon other containers.

To build the container, we issue a build command:

 $ docker build --tag="aem/author:6.0" .` will give us a standard container with CQ.

I’ll spare you the output of the command, but at the end you will have a basic container with the CQ install unpacked into the directory /opt/aem.

Now, as anybody who has started AEM for the first time will know, the initial start-up can take a while. While adding nosamplecontent to the run list can speed up the process, this first time we run the container will still take a few ‘go and get a coffee’ minutes.

Eventually, you will see the trusty response ‘Started’. At this point, you have AEM up and running inside a container.

This is where Docker comes into play. What we do now is to save the state of the container to have a starting point.

First, we need to get the ID of the container.

$ docker ps

CONTAINER ID    IMAGE          COMMAND    CREATED      STATUS          PORTS
85b7f041a69c    5a49c9efaac6   bash       3 days ago   Up 5 minutes    0.0.0.0:4502->4502/tcp

We can then use the ID to save the state of the container as a new image. In this case, we are just retagging the original image we started with.

$ docker commit 85b7f041a69c aem/author:6.0
$ docker images

REPOSITORY      TAG    IMAGE ID       CREATED       VIRTUAL SIZE
aem/author      6.0    7eeb28d10fff   3 days ago    2.974 GB

Ready to go

Now that we have a starting point – a clean instance of CQ – we can launch in a known state every time.

The key here is that we have to tell Docker which ports to expose when we start. In this case, we map port 4502 on the host to 4502 on the guest.

$ docker run -i -d -p 4502:4502 -p 6300:6300 aem/author:6.0 964f884fc0af0c1dd7a1ba696afad5767081a27727a93a0c37c39294a151f4ae

The other callout here is the use of the flag -d. This runs the container in the background so that you can still continue with the same shell. What you get back is the ID of the container to run further commands with.

Hardcoding the port-mapping is fine for a situation in which we know only one instance will be required, but in a continuous integration (CI) environment this is not so practical. Supplying just the guest ports, we can let Docker decide on which ports to use. Also, to make it easier to identify the running container, use the name parameter – which you can then use later to control the container.

$ docker run -i -d -p 4502 -p 6300 --name "aem_author_demo_itest_1410622055" aem/author:6.0
bdd421837e4cc0bc948a21e17508ecf84966373fd02d67ad26b473c6dd275db0

Here, we would point our browser at http://localhost:49177 and see AEM 6.0 running in all its glory.

$ docker port aem_author_demo_itest_1410622055 4502
0.0.0.0:49177

And to consume the JaCoCo coverage statistics, we would connect to port 49178.

$ docker port aem_author_demo_itest_1410622055 6300
0.0.0.0:49178

The ability to ‘ask’ Docker for the ports means that we can then pass the port to our build, so that it knows where to connect to when running the integration tests.

Using the out-of-the-box Maven set-up that AEM Blueprints gives you, we can pass the port to the integration tests. As an added bonus, we can pull down the JaCoCo coverage report after the integration tests have run.

$ mvn -P integrationTests \
      -Dcq.port=49177 \
      -Djacoco.address=localhost \
      -Djacoco.port=49178 \
      verify jacoco:dump

Once we are done, we tear it all down, grab the logs and throw it all away.

$ docker stop aem_author_demo_itest_1410622055
$ docker copy aem_author_demo_itest_1410622055:/opt/aem/crx-quickstart/logs itest-logs
$ docker rm -v aem_author_demo_itest_1410622055

No having to reset a central environment, no waiting for the quickstart jar to run through the initial start-up – just straight down to the business of testing.

What happens when a Service Pack is released?

This is where the use of Docker can really excel. As seen with some AEM Service Packs, you cannot always automate the upgrade simply by including the package to update the install directory.

Instead of building a new instance, we would spin up a new instance as if we were running AEM to do some testing. Then, instead, we would apply the Service Pack update. Once finished, we would save the container state with a new version tag.

$ docker run -i -d -p 4502 -p 6300 --name aem_demo_author_60 aem/author:6.0
#....do manual upgrade....
$ docker commit aem_demo_author_60 aem/author:6.0_SP1
$ docker images
REPOSITORY      TAG       IMAGE ID       CREATED         VIRTUAL SIZE
aem/author      6.0       7eeb28d10fff   3 days ago      2.974 GB
aem/author      6.0_SP1   1933e6f48fe5   2 minutes ago   3.127 GB

If it’s possible to automate the upgrade, you can create a Dockerfile extending from the base AEM container.

FROM aem/author:6.0

RUN mkdir -p /opt/aem/crx-quickstart/install
ADD ~/Documents/demo/6.0/AEM-Service-Pack-SP1.zip /opt/aem/crx-quickstart/install/

To build, we would use the build command to generate the new container. Run it once, so that the update is applied, and then save the state to:

$ docker build -t aem/author:6.0_SP1 .
$ docker run -i -d -p 4502 -name aem_demo_autor_60_SP1 aem/author:6.0_SP1
# verify update
# docker commit aem_demo_autor_60_SP1 aem/author:6.0_SP1
$ docker images
REPOSITORY      TAG       IMAGE ID       CREATED         VIRTUAL SIZE
aem/author      6.0       7eeb28d10fff   3 days ago      2.974 GB
aem/author      6.0_SP1   1933e6f48fe5   30 seconds ago   3.127 GB

With either method, you now end up with an original and a patched version of CQ. Using this method, you could start building application-specific containers as well.

Benefits of using Docker

So why would you use Docker rather than just running the quickstart program?

First, you no longer have to include the quickstart jar and licence file in your Maven repository. The space saved on the CI build server is probably worth it alone.

Second, the upgrade process becomes easier, especially with Service Packs. The process makes it even easier to test your AEM project against different versions of AEM – again without having to include each quickstart in the source repository.

The final benefit is in the reduction in the time it takes to run the full suite of in-container tests. From projects we have carried out, we have seen improvements of up to 200%. This brings it into the realm of being able to run as part of the commit phase of a project instead of a post-deployment task when it is already too late to fail the CI deployment.

Lessons from the road

So far, on projects we have adopted Docker to run integration tests with, we have been able to cut by more than half the time it takes to build, deploy and test an application. This means that it starts to be quick enough to introduce into the commit phase of a project. Even before an application has made it into a CI environment, we can be confident that it has passed the automated tests. This compares to the typical approach of running the tests against a CI environment post-deployment.

The biggest lesson learned from setting up the framework was: always, always clean up after yourself.

Creating lots of containers and images will eventually lead you to run out of space. Most often when you run a container you will be starting a new instance, and these will slowly eat up resources. The same goes with building base images – if you are using boot2docker, you will quickly run out of space in your VM.

When it comes to using Docker for integration testing, setting the DAEMON up on a central server means you can share the resources between a CI server and Docker, instead of loading up the poor CI server. This spreads the load between the servers of (potentially) having to run multiple AEM instances at the same time.

While Docker files are good for throwing simple containers together, for more complex and consistent set-ups, consider using Packer with Chef to provision the containers.

Docker is not a replacement for other virtualisation tools. It is complementary to the existing toolset and will make testing AEM projects even easier.

About The Author

Peter Abbott is a Technical Architect based in Wellington, New Zealand. He's a Cycling and Gelato fanatic.

@piemanpete