Tuesday, 22 September 2015

Having fun with REST API documentation

Recently I’ve been looking at ways to replace our current REST API documentation build with one that works with Java 8. Our requirements are - generate static html from JAX-RS annotations and JavaDoc. Simple, right? It shouldn’t be much more than finding a maven plugin and using it, right? Wrong.

Here’s how it went …

For a long time we’ve been using JAX Doclet. It generates JavaDoc html, except that it describes REST endpoints rather than classes, supported verbs rather than class methods, parameter bindings rather than method arguments, and status codes rather than method return types. It’s simple to integrate, and produces decent static html. It has one caveat though - it only works with Java 7. Java 8 brought changes in internal java doclet classes which caused many third party doclets to now maintain separate branches for Java 7, and Java 8. JAX Doclet has been feature complete and dormant for some time now, and there is no Java 8 compatibility on the horizon.

Here we are at a decision point - fork JAX Doclet and tailor it for Java 8, or find a replacement?

Programmers are a lazy bunch, or rather - we like to economize in order to reach our goal faster. Find a replacement then ...

Go to GitHub, and search for ‘jaxrs doclet’. There’s exactly one project with some traction on the topic. This is probably going to work.

The project we find is Swagger JAXRS Doclet, and it’s been quite some time since it’s seen any activity, but there’s a link to a fork that’s still very active, so we end up using teamcarma/swagger-jaxrs-doclet.

Good thing is it uses JAX-RS annotations, and JavaDoc descriptions to generate docs. Bad thing is the only output it generates are Swagger 1.2 compatible .json files, and an option to package them together with swagger-ui to get a rich single page web application to explore the API.

Sure, why not, surely you can double-click on index.html and run swagger-ui locally in your browser. It turns out that’s not the case.

Swagger-ui relies on XHR requests, and unbeknownst to me browsers have recently become immensely suspicious of XHR to the point of basically disabling it for local disk access.

That’s a problem for me, as I need static html. There might be magic switches to get browsers to perform XHR against local filesystem, but who wants to deal with that for the sake of browsing documentation ...

What now?

Well, find another maven plugin that takes REST API description in swagger .json format, and creates static HTML from it, of course.

Back on GitHub, searching for ‘swagger maven’, we get many results.

First hit is a very popular project, but immediately it doesn’t sound quite right for what we need - swagger-maven-plugin. It’s really a replacement for swagger-jaxrs-doclet, except it doesn’t understand JAX RS annotations, or JavaDoc comments, but requires classes to be annotated with Swagger annotations. That’s a deal breaker for me, as I don’t want to pollute our code with non-standard third-party annotations, and we don’t even particularly care about Swagger at this point - it’s just a tool to get to the static html documentation.

But this tool does something neat - it is able to generate not only swagger json files, but static text files of different formats as well - html, asciidoc, markdown … Sigh, if only it could work with JAX RS annotations, and JavaDoc … Would it be difficult to add such functionality? Could we generate Swagger annotations where they are missing, based on JAX RS annotations, and JavaDoc? Do we explore the idea, or look for existing working solution?

Let’s look further …

The next interesting project we find is swagger2markup-maven-plugin. This one looks exactly as what we need. It takes Swagger .json files, and converts them to asciidoc or markdown - unfortunately not also directly to html. For that we would then need another plugin - asciidoctor-maven-plugin. Not ideal, as we would end up with a bloated pom.xml, and a slow doc generation process, but as long as it does the job ...

Looking further still, we find that there are no more better candidates for what we need. So let’s work with swagger2markup-maven-plugin, and see how far it takes us.

It looks quite easy to use. We just add a few snippets to our pom.xml.

First, let’s make sure all of these extra plugins work with Java 8, since that is the whole reason we’re doing this:

    <!-- maven-compiler-plugin -->

Then we add swagger-jaxrs-doclet to javadoc-maven-plugin:



              <additionalparam> -skipUiFiles -apiVersion 1 -includeResourcePrefixes org.keycloak.services.resources.admin,org.keycloak.protocol.oidc -docBasePath /apidocs -apiBasePath http://localhost:8080/auth</additionalparam>




This should trigger the doclet during JavaDoc generation, and produce swagger .json files in target/apidocs-rest/swagger directory without also including swagger-ui.

Now that we got the first part done, we move to the next step - configuring swagger2markup-maven-plugin to convert the created swagger .json files to asciidoc document.

This second maven plugin uses its own maven repository:





That should produce three .adoc files in target/apidocs-rest/asciidoc directory - overview.adoc, paths.adoc, and definitions.adoc. The idea of Swagger2Markup is that these are three distinct document segments that can be combined with additional separately prepared .adocs into a complete and final document. Swagger2Markup itself however will only take it as far as preparing these three document segments for you.

There is one problem though. If you have multiple swagger .json files for input - as a result of multiple REST endpoints, then it looks like the created files will reflect documentation for only the last .json file processed. As if every next .json file processed will generate the three .doc files of the same name, overwriting the previous ones. Lucky for me I’m only dealing with one endpoint, so no need to figure out how to make it work with multiple files.

Next, we have to generate a composite asciidoc, and then perform translation into html. For that we’ll use asciidoctor-maven-plugin.

For starters, we need an asciidoc file that will tie together the three input .adoc files into one.

Let’s call it index.adoc, and let’s keep it very simple:
We just combine all the three documents into one without any additional content.

We place this file into src/docs/asciidoc directory of our maven project.

Now we configure asciidoctor-maven-plugin:

                <!-- List of attributes: 

That will produce a single index.html file in target/apidocs-rest/html directory that is what we were after. Mission accomplished!

Well ... sort of.

There are some problems with this document.

For one, it has a big ‘null’ for its title.

Tracing back the issue, we can see it in overview.adoc already, and is the result of title info missing in service.json file generated by our very first plugin.

Turns out that’s easy to fix.

We create a file apiinfo.json:
  "title": "Keycloak Admin REST API",
  "description": "This is a REST API reference for Keycloak Admin"

We place it in src/docs/swagger directory.

Then we add another parameter to <additionalParam> line in the first plugin:
-apiInfoFile ${project.basedir}/src/docs/swagger/apiinfo.json
That should fix the title.

Next problem is that the document is not very readable. We could address that by tweaking the .css.

But that's a styling issue.

More problematic is that the document contains some dead links for generic types like Request, Object, Map which have no model definition in definitions.adoc. We only notice that now when we view the document as html, but the problem originates in swagger .json file already.

Also, some of our operations apparently don’t have the JavaDoc descriptions, and for those the .json file doesn’t contain any description, rather the output falls back to auto-generated string composed of HTTP method and uri. This wasn’t an issue with original JAX Doclet since output was JavaDoc html, and content organisation followed endpoint / uri / method hierarchy, analogous to package / class / method hierarchy - therefore descriptions were optional.

With Swagger2Markup the hierarhy is flat. For a single endpoint all you get is a list of method-uri items where each one’s description serves as its title.

There is one way to fix this - adding the missing JavaDoc descriptions to our code. But then a more readable form would be one without using descriptions as titles - but rather using “{method} {uri}” as a title and putting description in content underneath. Is there a way to do that with Swagger2Markup? The underlying library is certainly capable of it. But can we reach it through maven plugin? Doesn’t look like we can.

If we could do that, we also then wouldn’t have to fill out all the JavaDoc descriptions for our methods - many are self-explanatory, and don’t require a description.

At the end of the day the result we got is not perfect, we’ll have to do more work. Looking over at swagger-maven-plugin (the one that is based on Swagger annotations), it has a very flexible templating support which allows to define how content is generated, by using something called moustache templates. We could skip the whole swagger.json-to-asciidoc-to-html chain of three plugins, and do it all in one - much more elegant. But for our use-case we would first have to enhance that library to support auto-generating the default Swagger annotation values based not only on JAX RS annotations, but also on JavaDoc tags. That’s not a trivial undertaking, especially since here we’re dealing with a standalone maven plugin, not a doclet.

Another option would be to enhance Swagger2Markup library to give us more control over how the documentation content is organised, and make that available through its maven plugin.

Yet another option is to decide that the solution we got thus far is good enough, and not worth investing more time in.

We could also change our mind about swagger annotations. Many projects have gone the Swagger way. In the end we might be better off by simply putting swagger annotations in our code, and use swagger-maven-plugin.

There’s so much other important, and interesting things to do on Keycloak. Let's use this solution for now, and move on to other things. We'll revisit REST API docs generation some other time.

Monday, 21 September 2015

Google Summer of Code 2015

Google Summer of Code (GSoC) inspires young developers to begin participating in open source development and gives them a sound exposure to real-world software development (for example, distributed development and version control, software licensing issues, and mailing list etiquette). Luckily, I was a part of GSoC 2015 for JBoss community where I contributed bits and bytes to JBoss Security i.e., SSO SaaS  Keycloak. I was being mentored by Keycloak co-lead Stian Thorgersen.
Contributing to JBoss Community has been a great experience for a newbie like me starting right from GSoC 2014 (Picketlink @asaldhan @pedroigor) to GSoC 2015 (Keycloak @stianst @mposolda). JBoss Community offers awesome projects ranging from Testing to Tooling, Application development to Security, Run time to Web frameworks, Asynchronous messaging to Integration and the Ceylon programming language. Although GSoC is over, but, I will always be up contributing bits and bytes to JBoss community as and when time will permit.
It was my second GSoC summer with JBoss community. I contributed during GSoC 2014 as well. GSoC entire coding period was an extremely elegant and eternal experience for me. I learnt about some of the best practices followed while working and collaborating on a very large code base. Building a beautiful API design, writing high quality code, synchronizing code with a version control system, raising issues on JIRA were some of the skills honed during GSoC in addition to establishing connections with some of the most awesome developers across the globe.
Contributing to Keycloak provided me a deep insight into Single Sign On security. Keycloak is an authentication server that provide users with the ability to centrally login, logout, register and manage their user accounts. The Keycloak administration UI manages roles and role mappings of any application secured by Keycloak. The core concept in Keycloak is a Realm. A realm secures and manages security metadata for a set of users and registered clients. Users can be created within a specific realm within the Administration console.
I had from little to no idea about OAuth, OpenID, Mutual SSL authentication before I started contributing to Keycloak. Hacking on SSO SaaS Keycloak is a great way to achieve proficiency over single sign on security and be a part of the awesome community. Keycloak UI is built upon Angular JS with Patternfly and can be customized. With each Keycloak contribution, I learnt something about security. With regards to Keycloak contributions, I achieved skills in ssl configuration over Keycloak, mutual ssl authentication for user’s and client’s over Keycloak, effective management of admin actions for auditing and monitoring, maintenance of sound password policies etc.
I managed to setup mutual SSL client authentication over WildFly. Similarly, I also had success in securing a war app and assigning application authorization access to the users via certificate roles. Setting and achieving the same functionality with Keycloak is still a work in progress. Keycloak will feature a Certificate Authority which would be responsible for issuing certificates and keys to realms, users and clients. Keycloak will have certificate authentication where in mutual SSL will be enabled i.e., Keycloak would not only check for a valid client certificate but will also verify the identity of certificate owner. In addition, writing test cases for features or fixing JIRA bugs helped in realizing open source software development cycle.
If you love security, Keycloak will surely lead you to the rescue path of learning hell lot of security in and around single sign on. Keycloak truly defines the next generation authentication server for cloud, HTML5 and mobile.

Friday, 11 September 2015

Keycloak 1.5.0.Final released

We've just released Keycloak 1.5.0.Final!

A few highlights from the release includes:

  • Custom auth flows - in 1.4.0 we introduced a new authentication and required actions SPIs, but we didn't make it possible to define your own flows. In this release you can now use the SPIs to create your own authenticators and also define your own authentication flows through the admin console
  • Custom auth flows for clients - we also made it possible to create your own authenticators and flows to authenticate clients.
  • Client signed token authentication - we introduced support to authenticate users with public/private keys and signed JWT tokens. This complements the introduction of service accounts from 1.4.0
  • Event/counter OTP for Google Authenticator and FreeOTP - through the admin console you can now configure if you want to use time based OTP or event based OTP
  • Make Infinispan cache default and remove in-mem cache - we removed the custom in-mem caches we had in previous releases and now only support the Infinispan based caches
  • Make Infinispan user session default and remove in-mem, jpa and mongo user session providers - we removed the JPA, Mongo and in-mem user session providers and will now focus on just the Infinispan based provider
  • Deal with time inconsistency in keycloak.js - the time on end-users computers can often be out of sync so we now deal with this in the javascript adapter by estimating the time skew between Keycloak and the users computer
  • Remove address from registration and account management - we've simplified the registration screen and account management and removed the address fields. If you need to add these or others you can now refer to the new address example theme

As usual we've been real busy and there's loads more, for full details check the release notes. To download the release go to keycloak.org.

Monday, 7 September 2015

Windows 10 app with integrated Keycloak OAuth 2.0

Team Aerogear provides many useful libraries not only for Android, iOS and Cordova. They developed OAuth2.0 Nuget package for Windows Phone platform enabling user to easily authorized through Google, Facebook and even through Keycloak. On their github you can find Windows Phone 8.1 example application using this library to upload a captured photo on web application which displays these uploaded photos in slideshow. What I did was conversion of this example app to Windows 10 platform and analysis of cooperation with the latest version of Keycloak 1.4.0. After reading of several tutorials from various sources I decided to create one summarizing step by step tutorial which describes how to deploy this app on Windows 10 mobile emulator.

General requirements

Pre-requisites and settings

  1. Download Keycloak Demo Bundle (version 1.4.0 Final)
  2. Setup Keycloak to use SSL (original resource)
    1. Create Self Signed Certificate by command:
    2. keytool -genkey -alias localhost -keyalg RSA -keystore keycloak.jks -validity 10950
      (answer for the What is your first and last name? should be DNS name (IP adress) of PC with running server). PC and mobile should be connected to the same Wi-Fi network.
    3. Move generated keycloak.jks to keycloak-demo-1.4.0.Final/keycloak/standalone/configuration
    4. To security-realms in keycloak-demo-1.4.0.Final/keycloak/standalone/configuration/standalone.xml add:
      <security-realm name="UndertowRealm">
                  <keystore path="keycloak.jks" relative-to="jboss.server.config.dir" 
       keystore-password="SELECTED_PASSWORD" />
    5. And to <server name="default-server"> element add:
<https-listener name="https" socket-binding="https" security-realm="UndertowRealm" />

Starting the server

  1. Start the server keycloak-demo-1.4.0.Final/keycloak/bin/standalone.bat -b
  2. Go to http://localhost:8080/auth/admin/index.html
  3. Login using admin / admin
  4. Click on Add realm and import this configuration file

Starting and setting the web application

  1. Download this project and get only the Shoot directory
  2. Add "auth-server-url": "/auth" to Shoot/src/main/webapp/WEB-INF/keycloak.json
  3. Build (mvn clean install)
  4. Open Shoot/target/shoot.war (with WinRAR for example) and add keycloak.cer there
  5. Go to http://localhost:9990/console/App.htmland navigate to Deployments and add shoot.war
    (if we deploy the .war by mvn wildfly:deploy, we would lost added certificate)

Installing certificate to Windows 10 emulator

  1. Open browser(Edge) and download the certificate by navigating to http://<IP>:8080/shoot/keycloak.cer where IP is the name of the certificate chosen earlier.
  2. Open and install the certificate

Preparing example app

Because this app was built for Windows Phone 8.1 platform, we need to convert the project to be executable on Windows 10, in other words the project has to target the Universal Windows Platform (UWP). There exists some scripts or nuget plugins which can do the conversion automatically, but finally there is more mess then before and you have to manually edit lot of stuffs in our case, so the most easiest way is to create new solution in VS15 for UWP and copy the source code into it.
  1. Download the source code of example app.
  2. Create new solution in VS15 and import the source code into it.
  3. Download the aerogear-windows-oauth2 project which is actually the source code of library available as Nuget package and import it to our solution
  4. Next you have to edit the MainPage.xaml.cs by configuring the IP adress to the same adress as the server is running on (it is assuring that it is going to run also on real device) and then add the line adding the scope to the configuration.
  5. Then in aerogear-windows-oauth2 project you should edit the class AccountManager.cs as below. That is because the manifest files of Windows Phone 8.1 and Windows 10 are different and the redirect protocol (org.aerogear.Shoot) is not that easy to parse from Windows 10 file.
  6. Build and run the solution on one of the Windows 10 emulators. The result is shown on screenshots below.

Application flow

  • You capture a photo (on emulator only this "tv noise"). Click on Keycloak icon, then you are redirected to login Keycloak page where you enter user / password (according to settings in realm). Finally we get feedback about succesfully uploaded image to our web application.
  • If we check the web application then, the image is visible there.


Library aerogear-windows-oauth2 was recently updated for Windows 10 as well so there is no need to manually edit the code in AccountManager and do other changes. It is good to say that this tutorial was focused mainly on Windows 10 mobile application, if we want to deal with desktop Windows 10 application, there has to be done some other additional changes such as adding the certificate directly to the "Trusted Publishers" and change of piece of code which does the authentication part.

Monday, 31 August 2015

Service accounts support in Keycloak

From 1.4 we added support for Service accounts to Keycloak. This feature allows to authenticate the client application with Keycloak server and retrieve the access token dedicated to this application. No interaction with users needed. This is great for administrative tasks executed on behalf of a service instead of individual user. You also don't even need browser available, the communication with Keycloak server is done in background.

To support service accounts, you just need to enable switch "Service accounts enabled" for particular client in admin console and then map roles will, which will be available in access token granted to the client. How to send client authentication request and retrieve service account is described in documentation . There is also service account example, which is part of the main demo example application from keycloak example distribution.

Right now, we support authenticating clients just through Client Credentials Grant from OAuth 2.0 spec, but in the future we plan to add more ways how to authenticate clients. This will include public/private key pair (similarly like Google is doing), 2-way SSL with client certificates and maybe more. We also plan to add suport for offline tokens, which might be quite useful for the service accounts, as I suppose one of the main usecases for them is triggering various offline administrative tasks periodically etc.

Thursday, 20 August 2015

Improved LDAP integration with Keycloak

LDAP integration in Keycloak is done through User Federation SPI. It works in a way that users are synced from LDAP into Keycloak database. This is needed because Keycloak needs to store various metadata about users and some of them may not be supported by the LDAP schema. The configuration of LDAP Federation provider in admin console is flexible enough, so you can choose whether your LDAP server is read-only or writable and whether new users created in Keycloak will be synced back to LDAP or just into Keycloak database etc.

Before 1.3.1 release we pulled just basic data of users from LDAP like first name, surname, email, username and password. From 1.3.1 we improved this and added support for LDAP mappers. This allows you even more flexibility in configuring which LDAP data are synced to Keycloak and back (for writable LDAP). Right now we have:

  • User Attribute mapper - This allows to specify which LDAP attributes are mapped to which User attributes in Keycloak. As you can see there are various things you can configure. For example you can specify if attribute is read-only. In this case will be data synced from LDAP to Keycloak DB, but not back to LDAP in case that user is changed in Keycloak.
  • Full name mapper - Allows some additional flexibity in mapping full name. Many LDAP deployments have full name in single LDAP attribute like "cn: John Doe". This mapper will ensure that this single attribute is properly mapped to first name and lastname in Keycloak DB
  • Role mapper - Allows to map role mappings from LDAP into Keycloak. You can specify LDAP DN where are roles configured and you can specify whether the roles are mapped into realm roles in Keycloak or to client roles of specified client. You can also specify the mode like whether role mappings are read-only or writable to LDAP etc.
Mapper SPI provides few callback methods, which are executed at the point when new user is registered in Keycloak, imported from LDAP or searched in Keycloak. You can implement the SPI and add your own mapper if default implementations are not sufficient.

Mappers were added in older 1.3.1 release, but in 1.4 we added more improvements and fixed lot of bugs based on the feedback from community. We also added simple LDAP example, which is part of Keycloak example distribution . It's really easy to have the example up and running, as it contains embedded ApacheDS based LDAP server, so only thing you need is JDK and maven. More documentation about mappers is in Keycloak documentation but note that admin console contains tooltips on various options and hence is quite self-documenting.

It's still possible that you can see the issues - various organizations have various data in their LDAP servers and it's quite a challenge to handle everything correctly in Keycloak. In this case, feel free to write questions on our keycloak-user mailing list or create JIRA.

Wednesday, 29 July 2015

Keycloak 1.4.0.Final released

Keycloak 1.4.0.Final is available for download. As usual we've been busy bringing new features as well as a number of improvements and bug fixes (in fact we resolved 119 issues for this release!). The highlights of this release includes:
  • Authentication and Required Actions SPI - We've create SPIs for authentication flows and required actions and made existing flows pluggable. Currently the flows are hard-coded, but can be configured to some extent. In the future we'll make it possible to define your own authentication flows, authenticators as well as required actions.
  • Service account - We've added support for clients to obtain tokens on behalf of themselves. These tokens can be obtained by using the Client Credentials Grant flow. In the future we'll add support for additional mechanisms to authenticate clients.
  • User impersonation - Admins can now impersonate users. This allows admins to login to applications as a specific user without knowing the users credentials.
  • Re-captcha for registration - Re-captcha can be enabled for the registration page.
  • Upgrade to WildFly 9.0.1.Final - Keycloak is now built on top of WildFly 9.0.1.Final.
  • More details on server info page - There's now a bit more details about the Keycloak server on the server info page in the admin console.
  • Improvements to admin console We've had the UXP experts look at our admin console and we've already done some improvements. Expect more fine tuning of the admin console in the future.
For all changes in this release check out JIRA, to download the release go to the Keycloak website.