Tuesday, 7 April 2015

Running Keycloak cluster with Docker

This is the first of two articles that will describe how to run Keycloak in clustered mode - first with Docker, and then with Kubernetes running on OpenShift 3.


The preferred way to run a Keycloak server - an authentication, and authorization server with support for single sign-on - is to run it as an isolated application in its own process. What you specifically don’t want is run any kind of applications in the same JVM instance, and it’s also not the best idea to run any other publicly facing applications on the same server.

The reason is of course security, but also stability. You don’t want your Keycloak process to suffer security vulnerabilities, or ‘Out of memory’ errors because of another application deployed in the same JVM. It is one thing to lose one application, another - more serious thing - to lose login capability for many applications and services, or even have your Keycloak private keys compromised.

Even an isolated instance, though, can occasionally experience a failure. Therefore, the proper way is to have a cluster of instances with load balancing router or reverse proxy in the front, that detects a failed instance, and diverts traffic away from it. One way to set that up would be to have one production instance at a time, and another failover instance idling until it’s needed. Another, even better way is to use all the running instances as production instances - this way having a horizontal scaling, whereby bringing up more instances linearly increases the number of requests your cluster is capable of handling.

It is this horizontal scaling capability that is the goal of Kubernetes project - the open source solution for provisioning of Docker containers.

In this first article I’ll show how to set up two Keycloak instances in clustered mode, each running in its own Docker container, and using a PostgreSQL database running in another Docker container.

In the next article we’ll enhance that set up by configuring these Docker instances as Kubernetes Pods using OpenShift 3. That will give us a scalable runtime environment where we can remove and add server instances virtually unnoticeable to clients.

Buckle up, and let’s get started.

Installing Docker


The first thing we’ll need is Docker. Docker is a containerization technology - as opposed to virtualization, and is Linux specific. Multiple processes can run each in its own isolated chrooted environment with its own filesystem image, its own IP address within the same bridged network, while they’re all using the host’s Linux kernel - the same kernel process.

Since we can natively use Docker only on Linux, what we do when we’re on Windows or OS X is use a solution that runs a simple, small, headless (no desktop) Linux distribution on VirtualBox - it’s called boot2docker.

If you already use VirtualBox, and have an already created virtual Linux instance you can also use that one. If it includes a desktop it can even simplify things as you can reach Docker containers IP addresses directly from browser. You may want to add another network adapter to your virtual instance - by default there is 'Adapter 1' using NAT, and you should configure 'Adapter 2' to be of type Host-only Adapter. That will simplify connecting from you Windows / OS X host to your Linux guest.

You can find Docker installation instructions for your platform on docker.io site.


Starting Docker daemon


Once you have Docker installed make sure that your Docker daemon process is running. In your Linux shell you can execute:

ps aux | grep docker

You should see a line similar to:

root     31237  2.8  1.2 1203628 25188 ?       Ssl  Mar29  30:57 /usr/bin/docker -d --selinux-enabled -H unix://var/run/docker.sock -H tcp://0.0.0.0:2375 --insecure-registry 172.0.0.0/8

If you don’t see that, then your Docker daemon is not yet running, and it’s time to start it - you may have to prepend ‘sudo ’ if you’re not root:

service docker start


Using Docker client


We can now use Docker client to issue commands to the daemon. We first have to make sure that our shell environment has some environment variables set to allow Docker client to communicate with the daemon.

One way to provide proper environment is to execute Docker client through sudo or as a root user (su -) on the Docker host system.

Another is to specify an environment variable:

export DOCKER_HOST=tcp://192.168.56.101:2375

Where the IP address is one of the public interfaces on the Docker host system - one that can also be reached from you client terminal (which can be on another host). You can use ifconfig or ip addr to list the available interfaces and their IPs. Note, that docker configures virtual networks that are not directly reachable from another host, here we are not interested in those.

We can now list currently running Docker containers:

 docker ps

If this is the first time you’re using Docker, or if you have just started up the daemon, then no Docker container is running yet.


Starting Postgres as Docker container


We’re now going to set up a PostgreSQL database.

Docker uses a central repository of Docker images - each image representing a filesystem with startup configuration for application, and is thus a mechanism to package an application.

We’ll use the latest official PostgreSQL image to start PostgreSQL as a new container instance. You can learn more about it on DockerHub.

docker run --name postgres -e POSTGRES_DATABASE=keycloak -e POSTGRES_USER=keycloak -e POSTGRES_PASSWORD=password -e POSTGRES_ROOT_PASSWORD=password -d postgres

The basic form of this command is: docker run -d postgres

That command instructs Docker daemon to download the latest official postgres image from DockerHub and start it up as new Docker container. The -d switch instructs docker client to return immediately, while any processes executed in container keep running in a background.

Additionally we specified several environment variables to be passed to the container which are used to configure a new database, and a new user for accessing the database. Note that we used 'password' - you should really change it to something else!

By using --name postgres we assigned a name to the new container. We’ll use this name whenever we need to refer to this running container in subsequent invocations of docker client.

Note: if this is not the first time you’re working through these steps you may already have a container named 'postgres'. In that case, you won’t be able to create another one with the same name. You have two options - choose a different name for this one, or destroy the existing one using: docker rm postgres  


We can attach to the container output using:

docker logs -f postgres

We use -f to keep following the output - analogous to how tail -f works.

You should see the output finish with something like:

PostgreSQL stand-alone backend 9.4.1
backend> statement: CREATE DATABASE "keycloak" ;

backend>

PostgreSQL stand-alone backend 9.4.1
backend> statement: CREATE USER "keycloak" WITH SUPERUSER PASSWORD 'password' ;

Use CTRL-C to exit the client.

We can now check that the database accepts connections, since it will be accessed via TCP from other docker instances.

We can start a new shell process within the same Docker container:

 docker exec -ti postgres bash

With this command we don’t start a new container - that would create a whole new copy of the chrooted file system environment with a new IP address assigned. Rather, we execute another process within the existing container.

By using -ti we tell docker that we want to allocate a new pseudo tty, and that we want this terminal’s input to be attached to container. That will allow us to interactively use the container’s bash.

We can find out what the container’s IP address is:

ip addr

We should see two interfaces:
  • lo with address 127.0.0.1
  • eth0 with address 172.17.0.x


eth0 will have an IP address within 172.17.x.x network.

This IP address is visible from all other docker containers running on the same host.

Let’s make sure that we are in fact attached to the same container running PostgreSQL server. Let’s use postgres client to connect as user keycloak to the local db:

# psql -U keycloak
psql (9.4.1)
Type "help" for help.


keycloak=# \l
                                List of databases
  Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
keycloak  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
          |          |          |            |            | postgres=CTc/postgres
template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
          |          |          |            |            | postgres=CTc/postgres
(4 rows)




We’re in fact inside the correct container, and have confirmed that the database is correctly configured with user keycloak.

Exit postgres client with \q. And then exit the shell with exit.



Another way to find out the container’s address is using docker’s inspect command:

 docker inspect -f '{{ .NetworkSettings.IPAddress }}' postgres

That should return the same IP address as we saw assigned to eth0 inside ‘postgres’ container.


Testing remote connectivity


We can test that remote connectivity works by starting a new docker container based on the same postgres image so that we have access to psql tool:

  docker run --rm -ti --link postgres:postgres postgres bash

We use run, therefore the last postgres argument is not a reference to existing running container, but an id of a Docker image to use for a new container. An extra bash argument instructs docker to skip executing the default startup script (the one that starts up a local postgres server), and to execute the command that we specified - bash.

The --rm argument instructs docker to completely clean up the container instance once the command exits - i.e. once we type exit in the bash.

We have also specified --link postgres:postgres which instructs Docker to add the IP address of existing ‘postgres’ container to ‘/etc/hosts’ file mapped to host name postgres. We can thus use postgres as a hostname, instead of having to look for its IP address.


Run the following:

# psql -U keycloak -h postgres
Password for user keycloak:
psql (9.4.1)
Type "help" for help.


keycloak=#

We have successfully connected to PostgreSQL server on another host.

It is now time to set up Keycloak.


Starting new Keycloak cluster as Docker container


We’ll use a prepared Docker image from DockerHub to run two Keycloak containers, each connecting to the PostgreSQL container we just started. In addition, the two Keycloak containers will establish a cluster for a distributed cache so that any state in between requests is instantly available to both instances. That way any one instance can be stopped, and users redirected to the other, without any loss of runtime data.

Issue the following command to start the first Keycloak container - make sure that environment variables are the same as those passed to postgres container previously:

 docker run -p 8080:8080 --name keycloak --link postgres:postgres -e POSTGRES_DATABASE=keycloak -e POSTGRES_USER=keycloak -e POSTGRES_PASSWORD=password -d jboss/keycloak-ha-postgres

Docker will download jboss/keycloak-ha-postgres image from DockerHub, and then create a new container instance from it, allocating a new IP address in the process. We used -p to map the port 8080 of the Docker host to port 8080 of the new container so that we don’t need to know container’s IP in order to connect to it. We can simply connect to the host’s port.

Monitor Keycloak as it’s coming up:

 docker logs -f keycloak


Let’s now start another container, and let’s name it keycloak2 - this one will get another IP address:

 docker run -p 8081:8080 --name keycloak2 --link postgres:postgres -e POSTGRES_DATABASE=keycloak -e POSTGRES_USER=keycloak -e POSTGRES_PASSWORD=password -d jboss/keycloak-ha-postgres

Wait for it to start completely:

 docker logs -f keycloak2


Pay attention to the following section towards the end of the log:

20:07:16,843 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------
20:07:16,844 INFO  [stdout] (MSC service thread 1-1) GMS: address=f25f922ce14d/keycloak, cluster=keycloak, physical address=172.17.0.10:55200
20:07:16,846 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------
20:07:17,044 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view: [b5356f1050cc/keycloak|1] (2) [b5356f1050cc/keycloak, f25f922ce14d/keycloak]
20:07:17,049 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Cache local address is f25f922ce14d/keycloak, physical addresses are [172.17.0.10:55200]
20:07:17,083 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Infinium' 6.0.2.Final

We can see that JGroups cluster was formed over two nodes - in bold. We can also find this container’s IP address in the log - it’s 172.17.0.10 in this case.


Each Keycloak instance can now be accessed from Docker host (where Docker daemon is running) via port 8080 of its container’s IP address. Since we mapped ports 8080, and 8081 of Docker host to Keycloak containers, we can connect directly to these ports on Docker host.

As an alternative we could forego mapping container ports to Docker host’s ports, and instead set up routing / forwarding using Docker host’s iptables - to let the traffic through the firewall, and set up routes on client hosts connecting to those instances to direct any trafic bound for 172.17.0.0/16 through Docker host.


Customizing the Keycloak image


The jboss/keycloak-ha-postgres image we have used was built from official JBoss Docker project on GitHub.

In keycloak-ha-postgres subdirectory there is a Docker file used to build the image.

From this directory you can perform your own build using:

 docker build --tag myrepo/keycloak-ha-postgres .

Where you can replace myrepo/keycloak-ha-postgres with some other image id.

See README.md file for more information.


Conclusion


We have shown how to start multiple Docker containers running a cluster of Keycloak servers, and connecting to another Docker container running a PostgreSQL database.

In the process we have demonstrated Docker client usage, and techniques for checking if the different servers running inside these containers have started up properly, and can connect to one another.

In the next article we’ll show how to install OpenShift 3, and run these Docker images as Kubernetes services, and virtual servers (pods).

15 comments:

  1. You really only have high availability for the Keycloak server in this, but not the database. What about configuring more than one Docker container for PostgreSQL and using multi-master replication? My memory may be faulty, on PostgreSQL's replication capabilities, but you definitely need replication to have a complete HA solution.

    ReplyDelete
    Replies
    1. Absolutely, the title of the article is clustering Keycloak though and we expect users to bring their own database and know how to cluster (and backup) their selected database.

      Delete
  2. I agree that it would be pretty neat to prepare a ready-for-use solution for DB high availability. Good idea for another blog :)

    ReplyDelete
  3. Marko,

    The above process seems broken for me. Have you got a troubleshooting guide?

    I get a good 'postgres' container, but 'keycloak' comes up with errors and gives a 404 on port 8080. I've tried bringing up 'keycloak2' but it was even worse.

    I've tried the image "jboss/keycloak-ha-postgres" and built my own from the Dockerfile in 'keycloak/server-ha-postgres' with the exact same results.

    ...abbreviated output...

    [root@nb-willson-7 server-ha-postgres]# docker run --name keycloak --link postgres:postgres -e POSTGRES_DATABASE=keycloak -e POSTGRES_USER=keycloak -e POSTGRES_PASSWORD=monkey -d jboss/keycloak-ha-postgres
    ...
    [root@nb-willson-7 server-ha-postgres]# docker logs -f keycloak
    ...
    JBoss Bootstrap Environment

    JBOSS_HOME: /opt/jboss/keycloak

    JAVA: /usr/lib/jvm/java/bin/java

    JAVA_OPTS: -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
    ...
    00:47:01,301 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 67) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)
    ...
    Caused by: java.lang.RuntimeException: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
    at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)
    ...
    Caused by: java.lang.NullPointerException
    at org.keycloak.models.sessions.infinispan.initializer.OfflineUserSessionLoader.init(OfflineUserSessionLoader.java:25)
    ...
    00:47:01,308 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "keycloak-server.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.undertow.deployment.default-server.default-host./auth" => "org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
    Caused by: java.lang.RuntimeException: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
    ...
    WFLYCTL0186: Services which failed to start: service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)

    00:47:01,572 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
    00:47:01,573 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
    00:47:01,573 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: Keycloak 1.6.1.Final (WildFly Core 1.0.1.Final) started (with errors) in 9681ms - Started 343 of 594 services (2 services failed or missing dependencies, 339 services are lazy, passive or on-demand)

    ReplyDelete
    Replies
    1. I'm seeing this same problem; so far haven't figured it out.

      Delete
    2. I got this to work by adding the following to keycloak-server.json:

      "userSessionPersister": {
      "provider" : "jpa"
      },

      However, I don't know if that changes the characteristics of the cluster.

      Delete
    3. The server-ha-postgres docker image is a bit outdated. Keycloak now comes bundled with HA config, so all that's required is to setup the db. This means the docker image doesn't have to do any configuration in keycloak-server.json or adding cache config to standalone-ha.xml. It should only add postgres and make sure it runs standalone-ha.xml.

      Delete
  4. I have followed the instructions on this page, but the outcome is always:

    Received new cluster view for channel ejb: [9d21bc8778f0|0] (1) [9d21bc8778f0]
    Received new cluster view for channel hibernate: [9d21bc8778f0|0] (1) [9d21bc8778f0]
    Received new cluster view for channel keycloak: [9d21bc8778f0|0] (1) [9d21bc8778f0]
    Received new cluster view for channel web: [9d21bc8778f0|0] (1) [9d21bc8778f0]
    Received new cluster view for channel server: [9d21bc8778f0|0] (1) [9d21bc8778f0]

    Only takes one node :(

    I used version 2.5.1.Final

    Any idea?

    ReplyDelete
    Replies
    1. This image hasn't been updated for a while. The problem is most likely that you'll need to set the bind address for the clustering. Take a look at https://keycloak.gitbooks.io/server-installation-and-configuration/content/topics/clustering/multicast.html

      Delete
    2. As Stian already said you need to set clustering ip address, here is a docker file you can create your image https://github.com/jmowla/keycloak/blob/master/server-ha-postgres/Dockerfile
      This Dockerfile sets the container's ip as clustering bind address.

      Delete
  5. If multiple Keycloak instances are running using the same Postgres database, what exactly is the reason and benefit to run the instances as a Wildfly cluster?
    I presume they will share some distributed cache, but running a cluster will also complicate things, like network and configuration setup.

    Wouldn't it be an okay solution for instances to share state only in database and not have them clustered?

    ReplyDelete
    Replies
    1. Keycloak heavily relies on caches. This is to prevent having to read everything from the db for each request and it also uses caches to store non-persisted data like sessions.

      Delete
  6. This is a nice guide for setting up a cluster running on one host.

    I would think this gets much more complicated when the cluster needs to be deployed, using Docker, to multiple hosts - as it often would in redundant production environments. You would probable need to do more Docker port mappings to make it possible for JGroup to discover and communicate with other nodes.

    Any experience doing such a setup?

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. I have setup the keyclaock HA with mysql but when i create the realm and user its not reflect to the other one until i restart the other keycloak services.

    Could you help me to fix this

    ReplyDelete

Please only add comments directly associated with the post. For general questions use the Keycloak user mailing list.