Wednesday, 21 March 2018

Clustering Vert.x with Infinispan

Welcome to the third in a multi-part series of blog posts about creating Eclipse Vert.x applications with Infinispan. In the previous blog posts we have seen how to create REST and PUSH APIs using the Infinispan Server. The purpose of this tutorial is to showcase how to create clustered Vert.x applications using Infinispan in embedded mode.

Why Infinispan ?

Infinispan can be used for several use cases. Among them we find that it can be used as the underlying framework to cluster your applications. Infinispan uses peer-to-peer communication between nodes, so the architecture is not based on master/slave mode and there is no single point of failure. Infinispan supports replication and resilience across data centers, is fast and reliable. All the features that make this datagrid a great product, make it a great cluster manager. If you need to create clustered applications or microservices, this can be achieved with Vert.x using the Vert.x-Infinispan cluster manager.

Creating a clustered application

The code of this tutorial is available here.

Dummy Application

Let’s start with a dummy clustered system with 3 verticles.

WebService Status Producer

Produces randomly [0,1,2] values every 1000 milliseconds and sends them to the event bus "ids" address.

Reboot Consumer

Consumes messages from the event bus "ids" address, and launches a "reboot" that lasts for 3000 milliseconds whenever the value is 0. If a reboot is already happening, we don’t need to relaunch any new reboot. When a reboot starts or ends, a message is sent to the event bus to the "reboot" address.

Notice that:
  • We use a simple boolean to check if there is a reboot going on. This is safe because every verticle is executed from a single event loop thread, so there won’t be multiple threads executing the code at the same time.
  • An ID is generated to identify the Verticle. The message sent to the event bus is a JsonObject


Consumes monitoring messages from the event bus "reboot" address and logs them.

Clustering the dummy Application

To create a cluster of these applications, we just need to do 2 things:

  1. Add the cluster manager maven dependency.  
  2. Run and deploy each verticle in cluster mode. Each Verticle class has a main method that deploys each verticle separetly. Example for the Monitoring verticle.

Running the application, we can monitor the logs
Each clustered application contains - or embeds - an Infinispan instance. Under the hood, the 3 Infinispan instances will form a cluster.

What if I need to scale

Imagine you need to scale the Reboot Consumer application. We can run it multiple times, let’s say 2 more times. The two new instances will join the cluster. In this case, we have “ID-93EB” ”ID-45B8” and “ID-247A”, so now we have a cluster of 5. It's very simple but if we have a look to the monitoring console, we will notice reboots are now happening in parallel.

3 Reboot Consumers

As I mentioned before, this example is a dummy application. But in real life you could need to trigger a process from a verticle that runs multiple times and need to be sure this process is happening just once at a time. How can we fix this ? We can use Vert.x Shared Data structures API.

Shared data API to rescue

In this particular case, we are going to use a clustered lock. Using the lock, we can now synchronise the reboots among the 3 nodes.

Using Shared Data API, one reboot at a time

Vert.x clustered lock in this example is using an emulated version of the new Clustered Lock API of Infinispan introduced in 9.2 which has been freshly released. I will come back to share about this API in particular in further blog posts. You can read about it on the documentation or run the infinispan-simple-tutorial.

One node at a time

When clustering applications with Vert.x, there is something you need to take care of. It is important to understand that each node contains an instance of the datagrid. This means that scaling up and down needs to be done one at a time. Infinispan, as other datagrids, reshuffles the data when a new node joins or leaves a cluster. This process is done following a distributed hashing algorithm, so not every data is moved around, just the data that is supposed to live in the new node, or the data owned by a leaving node. If we just kill a bunch of nodes without taking care of the cluster, consequences can be harming! This is something quite obvious when dealing with databases : we just don’t kill a bunch of database instances without taking care of every instance at a time. Even when Infinispan data is only in memory we need to take care about it in the same way. Openshift, which is built on top of Kubernetes, helps dealing properly and safely with these scale up and down operations.


As you have seen, creating clustered applications with Vert.x and Infinispan is very straightforward. The clustered event bus is very powerful. In this example we have seen how to use a clustered lock, but other shared data structures built on top of Counters are available.

About the Vert.x Infinispan Cluster Manager status

At the time of this writing, Infinspan 9.2.0.Final has been released. From vert.x-infinispan cluster manager point of view, before Vert.x 3.6 (which is not out yet) the cluster manager is using Infinispan and it’s using an emulation layer for locks and counters. In this tutorial we are using Vert.x 3.5.1 version.

This tutorial will be updated with the version using Infinispan 9.2 as soon as the next vert.x-infinispan will be released, which will happen in a few months. Meanwhile, stay tuned!

Tuesday, 13 March 2018

Final release for Hotrod clients C++ and C# 8.2.0 are out!

We're pleased to announce the availability of the 8.2.0.Final release of the C++ and C# Hotrod clients.
Here is what happened in the 8.2.0 episode:

  • SASL: PLAIN, MD5, EXTERNAL, GSSAPI (linux only)
  • Continuous Queries
  • getAll operation
  • simplified remote exec API

  • Continuous Queries
  • GetAll operation
  • simplified remote exec API

You can find more info and even the binaries at the usual places [1][2][3][4]

In the backstage people are already working on the 8.3.0 episode, you can partecipate expressing your opinion or adding your ideas here [5][6].

Thank you for reading.

The Infinispan Team

[1] Project Issues
[2] C++ Source
[3] C# Source
[4] Download
[5] C++ Features List for 8.3.0
[6] C# Features List for 8.3.0

Wednesday, 7 March 2018

REST with HTTP/2

HTTP has become one of the most successful and heavily used network protocols around the world. Version 1.0 was created in 1996 and received a minor update 3 years later. But it took more than a decade to create HTTP/2 (which was approved in 2015). Why did it take so long? Well, I wouldn’t tell you all the truth if I didn’t mention an experimental protocol, called SPDY. SPDY was primarily focused on improving performance. The initial results were very promising and inside Google’s lab, the developers measured 55% speed improvement. This work and experience was converted into HTTP/2 proposal back in 2012. A few years later, we can all use HTTP/2 (sometimes called h2) along with its older brother - HTTP/1.1.

Main differences between HTTP/1.1 and HTTP/2

HTTP/1.1 is a text-based protocol. Sometimes this is very convenient, since you can use low level tools, such as Telnet, for hacking. But it doesn’t work very well for transporting large, binary payloads. HTTP/2 solves this problem by using a completely redesigned architecture. Each HTTP message (a request or a response) consists of one or more frames. A frame is the smallest portion of data travelling through a TCP connection. A set of messages is aggregated into a, so called stream.

HTTP/2 allows to lower the number of physical connections between the server and the client by multiplexing logical connections into one TCP connection. Streams allow the server to recognize, which frame belongs to which conversation.

How to connect using HTTP/2?

There are two ways for starting an HTTP/2 conversation.

The first one, and the most commonly used one, is TLS/ALPN. During TLS handshake the server and the client negotiate protocol for further communication. Unfortunately JDK below 9 doesn’t support it by default (there are a couple of workarounds but please refer to your favorite HTTP client’s manual to find some suggestions).

The second one, much less popular, is so called plain text upgrade. During HTTP/1.1 conversation, the client issues an HTTP/1.1 Upgrade header and proposes new conversation protocol. If the server agrees, they start using it. If not, they stick with HTTP/1.1.

The good news is that Infinispan supports both those upgrade paths. Thanks to the ALPN Hack Engine (the credit goes to Stuart Douglas from the Wildfly Team), we support TLS/ALPN without any bootstrap classpath modification.

Configuring Infinispan server for HTTP/2

Infinispan’s REST server already supports plain text upgrades out of the box. TLS/ALPN however, requires additional configuration since the server needs to use a Keystore. In order to make it even more convenient, we support generating keystores automatically when needed. Here’s an example showing how to configure a security realm:

The next step is to bind the security realm to a REST endpoint:

You may also use one of our configuration examples. The easiest way to get it working is to use our Docker image:

Let’s explain a couple of things from the command above:
  • -e "APP_USER=test" - This is a user name we will be used for REST authentication.
  • -e "APP_PASS=test" - Corresponding password.
  • ../../docs/examples/configs/standalone-rest-ssl.xml - Here is a ready-to-go configuration with REST and TLS/ALPN support
Unfortunately, HTTP/2 functionality has been broken in 9.2.0.Final. But we promise to fix it as soon as we can :) Please use 9.1.5.Final in the meantime.

Testing using CURL

Curl is one of my favorite tools. It’s very simple, powerful, and… it supports HTTP/2. Assuming that you already started Infinispan server using `docker run` command, you can put something into the cache:

Once, it’s there, let’s try to get it back:

Let’s analyze CURL switches one by one:
  • -k - Ignores certificate validation. All automatically generated certificates and self-signed and not trusted by default.
  • -v - Debug logging.
  • -u test:test - Username and password for authentication.
  • -d test - This is the payload when invoking HTTP POST.
  • -H “Accept: text/plain” - This tells the server what type of data we’d like to get in return.

Conclusions and links

I hope you enjoyed this small tutorial about HTTP/2. I highly encourage you to have a look at the links below to learn some more things about this topic. You may also measure the performance of your app when using HTTP/1.1 and HTTP/2. You will be surprised!

Tuesday, 6 March 2018

Accessing Infinispan inside Docker for Mac

Connecting to Infinispan instances that run inside Docker for Mac using the Java Hot Rod client can be tricky. In this blog post we'll be analyzing what makes this environment tricky and how to get around the issue.

The tricky thing about Docker for Mac is that internal container IP addresses are not accessible externally. This is a known issue and it can be hard to workaround it. In container orchestrators such as Openshift, you can use Routes to allow external access to the containers. However, if running vanilla Docker for Mac, the simplest option is to map ports over to the local machine.

Why is this important? When someone connects using the Hot Rod protocol, the server returns the current topology to the client. When Infinispan runs inside of Docker, this topology by default contains internal IP addresses. Since those are not accessible externally in Docker for Mac, the client won't be able to connect.

To workaround the issue, Infinispan server Hot Rod endpoint can be configured with external host/port combination, but doing this would require modifying the server's configuration. A simpler method to get around the issue is to configure the client's intelligence to be Basic. By doing this the server won't send topology updates nor will the client be able to locate where keys are located using hashing. This has a negative performance impact since all requests to Infinispan single server or server cluster would need to go over the same IP+port. However, for demo or sample applications on Mac environments, this is reasonable thing to do.

So, how do we do all of this?

First, start Infinispan server and map Hot Rod's default port 11222 to the local 11222 port:

docker run -it -p 11222:11222 jboss/infinispan-server:9.2.0.Final

Open your IDE and create a project with this dependencies:

Finally, create a class that connects to Infinispan and does a simple put/get sequence:


Monday, 5 March 2018

A SWIG based framework to build Hotrod client prototype in your preferred language

If your are working on a non Java/C++/C#/JS application and you need to interact with Infinispan via Hotrod you may be interested in the idea behind the HotSwig[1] project.

Hotswig proposes a framework to build Hotrod client prototypes quickly and for a generic SWIG[2] supported language.
As people familiar with C++ and C# Infinispan native clients know, SWIG plays a role in both the projects:

  • is used to build the base of the C# client wrapping the C++ core with a C# layer;
  • is used in the C++ project to run (part of) the Java test suite against the client, in this way: a Java wrapper is built via SWIG to make the C++ client looks like its Java big brother so it can be tested with the Java test suite.

The main goal was to produce for a specific language an almost complete client reusing the C++ core features and the following workflow has been setup to do that:

  • the whole C++ interface is processed by SWIG. The resulting wrapper exposes almost all the C++ functions;
  • a user friendly adaptation layer is build on top of the SWIG result.

This approach doesn't work for the HotSwig goal, mainly because the effort need by the second step is usually not-negligible and prevents the rapid development of prototype in a generic language.

In the HotSwig approach, this limitation is removed moving the adaptation layer from the target language to the C++ side and then letting SWIG generate a ready to use client prototype. So the HotSwig workflow is the following:

  • build an adaptation facade around the C++ core to make it SWIG friendly (do the adaptation work once for all on the C++ side);
  • explicitly define what we want in the produced SWIG wrapper (keep things simple excluding everything by default);
  • run SWIG to produce the client.

At the moment HotSwig is just a proof of concept, but you can try to run it and produce a ready to work Infinispan client for the language you need. Examples are already provided for python, ruby and Octave, but HotSwig should work with all the SWIG supported languages. If you get it to run in your preferred programming language, please share your experience with us.

I've listed here[3] some tasks for the roadmap, with the idea to test the flexibility of the framework trying to extend it in different directions. Maybe the idea is good and it can grow up from a PoC to something that can really help devs. You can add you ideas of course.

So if you need to do math against your Infinispan data set why don't you try the Octave client? Or maybe you want to do analytics with R, or presentation with PHP. Or you just like parenthesis and you want to use Lisp. Or you're working for the Klingon empire and you must use ylDoghQo'[4]... well ok just joking now...

Thanks for reading!

The Infinispan Team


Wednesday, 28 February 2018

Infinispan 9.2.0.Final

Infinispan 9.2.0.Final "Gaina" is out !

Our three-month time-boxing for a minor release plan got a little bit skewed this time in order to accommodate for some additional overhauls. This also means that, for a minor release, this is much meatier than usual.

Core improvements

  • Conflict resolution
    Automatic conflict resolution after a partition merge is now supported for all partition handling strategies and is enabled by default. Furthermore, it is now possible to deploy custom EntryMergePolicy implementations to the server
  • Reactive streams-based distributed Iteration improvements
    Distributed iterator now uses less threads and allows for efficient parallel retrieval providing for improved throughput
  • Biased reads for scattered caches
    Originator can read the ‘backup’ copy locally until the data gets overwritten again. Together with improved read performance this migrates data to nodes that use it. 
  • Off-heap sizing
    Off-heap requires less overhead per entry and provides for more accurate sizing allowing you to maximize your memory used
  • Exception based evictionA new "eviction" that instead of removing old entries prevents new entries being inserted (supported by all memory storage and eviction types)

API improvements

  • Multimap caches
    Available for both Embedded and for Hot Rod, these maps which can store multiple values for the same key
  • Clustered Counters
    Clustered counters are now available for Hot Rod and in non-clustered deployments.
  • Clustered Locks
    Available in embedded mode. They allow concurrent synchronization between nodes in the same cluster
  • Wildcard configurations
    Implicitly use a predefined configuration for all caches whose name matches a wildcard. This is particularly useful when using Infinispan through an API which doesn't allow for additional configuration properties (such as JCache).
  • Cluster-wide cache admin with optional persistence
    The CacheManager API has been enhanced with methods to create/destroy caches across a cluster, in both Embedded and Hot Rod scenarios (REST will come in 9.3). Optionally, configurations can be made persistent across restarts.
  • Cache Stream
    Overloaded collect() method to take Supplier so that collect() in clustered environments is more user-friendly.

Data Interoperability

Transcoding is a powerful new feature which allows for transparent conversion between a number of formats across different endpoints. For example, it is now possible to write ProtoBuf-encoded data through the Hot Rod endpoint and retrieve that same data as a JSON document through the REST endpoint and vice versa. Additionally, such data is also indexable and queryable.

Indexing and Query

  • POJO queries over Hot Rod
    It is now possible to directly use Hibernate Search-annotated objects through JBoss Marshalling/Java serialization without the need for ProtoBuf over Hot Rod.
  • Broadcast queries
    Clustered queries have been unified with non-clustered queries under a single API, making their use transparent.


Infinispan Server

  • Rebased on WildFly 11
    The server baseline has been updated to WildFly 11
  • Async Hot Rod server
    The Hot Rod server now uses async ops, sparing CPU cycles from context-switching and reducing the latency.
  • Queries over REST
    The REST endpoint now supports running Ickle queries. This is fully integrated with the above-mentioned JSON support, so your results will be returned to you as JSON documents.
  • Netty Hot Rod Client
    The Hot Rod Java client network layer has been completely rewritten to use Netty, bringing true asynchronous calls and some performance benefits.


Management, monitoring and logging

  •     Console support for counters
  •     Improved remote protocol access logging
  •     Jolokia is now integrated as a part of the server.


Infinispan on OpenShift

We have been doing a lot of work in making Infinispan a first-class citizen of OpenShift. Check out the OpenShift templates for more details.


  • JCache 1.1
    This release is now aligned with JCache 1.1.
  • Hibernate second-level cache provider
    Traditionally shipped by our friends on the Hibernate ORM team, this component has now changed ownership over to us. This release includes a provider for both Hibernate 5.1 and 5.2.
  • Azure cloud discoveryCourtesy of JGroups' extras, we now support discovery in Azure.


The codename

In the grand-old tradition of giving major and minor Infinispan releases a beer-themed codename, 9.2 is no exception.

"Gaina", which means "chicken" in the milanese dialect, also happens to be one of the great beers of the Birrificio Lambrate in Milan.


Onwards to 9.3

We have already started working on our next release, 9.3 which should be with you at the end of May. This will continue the work to make Infinispan fully asynchronous inside out, reducing resource usage and increasing performance. We are also working on a new modular API which will improve usability, increase interoperability between embedded and remote scenarios and take advantage of reactive designs. Transactions should finally make their appearance in Hot Rod and security will be greatly enhanced, by taking advantages of the great work done by our friends over on the Elytron team. We have much more planned, so please consult our roadmap for details.


Download, learn and play

You will find downloads, documentation, tutorials, quickstarts and demos over on our website.

Please let us know on our forum, on IRC, on our issue tracker if you have any issues with this release, if there is any feature you would like to see in the future, or just to chat.

Wednesday, 21 February 2018

Infinispan 9.2.0.CR3

This should have been the announcement for Final, but we discovered a number of performance regressions as well as a few important bugs that needed fixing. We also slipped in a few features and improvements. So, without further ado, here's what is new and noteworthy in Infinispan 9.2.0.CR3:
  • Various component upgrades
    • Netty 4.1.21
    • Hibernate Search 5.9.0.Final
    • Protostream to 4.2.0.CR1
  • Features/Enhancements
    • Azure discovery
    • Use async ops in the Hot Rod server
    • Simplified client configuration when security is enabled
  • Lots of documentation updates
    • REST server changes
    • Data Encoding
    • Server tasks
  • And many bugfixes

Get your artifacts from maven, the distributions from our download page, the fixed issues from our issue tracker and read the updated documentation. Come and talk to us on IRC (#infinispan on Freenode) or ask questions on the forum.