Thursday, 29 March 2018

Infinispan Spark connector 0.7 released!

A new version of connector that integrates Infinispan and Apache Spark has just been released!

This release brings compatibility with Infinispan 9.2.x and Spark 2.3.0.

Also included a new feature that allows to create and delete caches on demand, passing custom configurations when required. For more details, please consult the documentation.

To quickly try the connector, make sure to check the Twitter demo, and for any issues or suggestions, please report them on our JIRA.

Cheers!


Tuesday, 27 March 2018

Infinispan 9.2.1.Final

Infinispan users,

we have just released 9.2.1.Final which includes 65 fixes. Highlights of this release include:
  • Many fixes/improvements to the REST endpoint
    • Configurable CORS settings
    • HTTP/2 now works
    • Accept-Encoding and Content-Encoding handling
  • It is now possible to retrieve the list of cache names over Hot Rod
  • Substantial performance improvements when iterating over the file store
  • Lots of bug fixes, test fixes and documentation improvements
As usual you can find all the bits on our website. If you find any issues, don't hesitate to report them on our issue tracker.

Enjoy

The Infinispan Team

Wednesday, 21 March 2018

Clustering Vert.x with Infinispan

Welcome to the third in a multi-part series of blog posts about creating Eclipse Vert.x applications with Infinispan. In the previous blog posts we have seen how to create REST and PUSH APIs using the Infinispan Server. The purpose of this tutorial is to showcase how to create clustered Vert.x applications using Infinispan in embedded mode.


Why Infinispan ?


Infinispan can be used for several use cases. Among them we find that it can be used as the underlying framework to cluster your applications. Infinispan uses peer-to-peer communication between nodes, so the architecture is not based on master/slave mode and there is no single point of failure. Infinispan supports replication and resilience across data centers, is fast and reliable. All the features that make this datagrid a great product, make it a great cluster manager. If you need to create clustered applications or microservices, this can be achieved with Vert.x using the Vert.x-Infinispan cluster manager.


Creating a clustered application


The code of this tutorial is available here.

Dummy Application


Let’s start with a dummy clustered system with 3 verticles.


WebService Status Producer

Produces randomly [0,1,2] values every 1000 milliseconds and sends them to the event bus "ids" address.



Reboot Consumer

Consumes messages from the event bus "ids" address, and launches a "reboot" that lasts for 3000 milliseconds whenever the value is 0. If a reboot is already happening, we don’t need to relaunch any new reboot. When a reboot starts or ends, a message is sent to the event bus to the "reboot" address.

Notice that:
  • We use a simple boolean to check if there is a reboot going on. This is safe because every verticle is executed from a single event loop thread, so there won’t be multiple threads executing the code at the same time.
  • An ID is generated to identify the Verticle. The message sent to the event bus is a JsonObject



Monitoring

Consumes monitoring messages from the event bus "reboot" address and logs them.



Clustering the dummy Application


To create a cluster of these applications, we just need to do 2 things:

  1. Add the cluster manager maven dependency.  
  2. Run and deploy each verticle in cluster mode. Each Verticle class has a main method that deploys each verticle separetly. Example for the Monitoring verticle.

Running the application, we can monitor the logs
Each clustered application contains - or embeds - an Infinispan instance. Under the hood, the 3 Infinispan instances will form a cluster.


What if I need to scale


Imagine you need to scale the Reboot Consumer application. We can run it multiple times, let’s say 2 more times. The two new instances will join the cluster. In this case, we have “ID-93EB” ”ID-45B8” and “ID-247A”, so now we have a cluster of 5. It's very simple but if we have a look to the monitoring console, we will notice reboots are now happening in parallel.

3 Reboot Consumers


As I mentioned before, this example is a dummy application. But in real life you could need to trigger a process from a verticle that runs multiple times and need to be sure this process is happening just once at a time. How can we fix this ? We can use Vert.x Shared Data structures API.

Shared data API to rescue


In this particular case, we are going to use a clustered lock. Using the lock, we can now synchronise the reboots among the 3 nodes.


Using Shared Data API, one reboot at a time

Vert.x clustered lock in this example is using an emulated version of the new Clustered Lock API of Infinispan introduced in 9.2 which has been freshly released. I will come back to share about this API in particular in further blog posts. You can read about it on the documentation or run the infinispan-simple-tutorial.

One node at a time


When clustering applications with Vert.x, there is something you need to take care of. It is important to understand that each node contains an instance of the datagrid. This means that scaling up and down needs to be done one at a time. Infinispan, as other datagrids, reshuffles the data when a new node joins or leaves a cluster. This process is done following a distributed hashing algorithm, so not every data is moved around, just the data that is supposed to live in the new node, or the data owned by a leaving node. If we just kill a bunch of nodes without taking care of the cluster, consequences can be harming! This is something quite obvious when dealing with databases : we just don’t kill a bunch of database instances without taking care of every instance at a time. Even when Infinispan data is only in memory we need to take care about it in the same way. Openshift, which is built on top of Kubernetes, helps dealing properly and safely with these scale up and down operations.


Conclusions


As you have seen, creating clustered applications with Vert.x and Infinispan is very straightforward. The clustered event bus is very powerful. In this example we have seen how to use a clustered lock, but other shared data structures built on top of Counters are available.


About the Vert.x Infinispan Cluster Manager status


At the time of this writing, Infinspan 9.2.0.Final has been released. From vert.x-infinispan cluster manager point of view, before Vert.x 3.6 (which is not out yet) the cluster manager is using Infinispan 9.1.6.final and it’s using an emulation layer for locks and counters. In this tutorial we are using Vert.x 3.5.1 version.

This tutorial will be updated with the version using Infinispan 9.2 as soon as the next vert.x-infinispan will be released, which will happen in a few months. Meanwhile, stay tuned!

Tuesday, 13 March 2018

Final release for Hotrod clients C++ and C# 8.2.0 are out!

We're pleased to announce the availability of the 8.2.0.Final release of the C++ and C# Hotrod clients.
Here is what happened in the 8.2.0 episode:

C++
  • SASL: PLAIN, MD5, EXTERNAL, GSSAPI (linux only)
  • Continuous Queries
  • getAll operation
  • simplified remote exec API

C#
  • SASL: PLAIN, MD5, EXTERNAL
  • Continuous Queries
  • GetAll operation
  • simplified remote exec API

You can find more info and even the binaries at the usual places [1][2][3][4]

In the backstage people are already working on the 8.3.0 episode, you can partecipate expressing your opinion or adding your ideas here [5][6].

Thank you for reading.

The Infinispan Team

[1] Project Issues
[2] C++ Source
[3] C# Source
[4] Download
[5] C++ Features List for 8.3.0
[6] C# Features List for 8.3.0

Wednesday, 7 March 2018

REST with HTTP/2

HTTP has become one of the most successful and heavily used network protocols around the world. Version 1.0 was created in 1996 and received a minor update 3 years later. But it took more than a decade to create HTTP/2 (which was approved in 2015). Why did it take so long? Well, I wouldn’t tell you all the truth if I didn’t mention an experimental protocol, called SPDY. SPDY was primarily focused on improving performance. The initial results were very promising and inside Google’s lab, the developers measured 55% speed improvement. This work and experience was converted into HTTP/2 proposal back in 2012. A few years later, we can all use HTTP/2 (sometimes called h2) along with its older brother - HTTP/1.1.

Main differences between HTTP/1.1 and HTTP/2


HTTP/1.1 is a text-based protocol. Sometimes this is very convenient, since you can use low level tools, such as Telnet, for hacking. But it doesn’t work very well for transporting large, binary payloads. HTTP/2 solves this problem by using a completely redesigned architecture. Each HTTP message (a request or a response) consists of one or more frames. A frame is the smallest portion of data travelling through a TCP connection. A set of messages is aggregated into a, so called stream.



HTTP/2 allows to lower the number of physical connections between the server and the client by multiplexing logical connections into one TCP connection. Streams allow the server to recognize, which frame belongs to which conversation.

How to connect using HTTP/2?

There are two ways for starting an HTTP/2 conversation.

The first one, and the most commonly used one, is TLS/ALPN. During TLS handshake the server and the client negotiate protocol for further communication. Unfortunately JDK below 9 doesn’t support it by default (there are a couple of workarounds but please refer to your favorite HTTP client’s manual to find some suggestions).

The second one, much less popular, is so called plain text upgrade. During HTTP/1.1 conversation, the client issues an HTTP/1.1 Upgrade header and proposes new conversation protocol. If the server agrees, they start using it. If not, they stick with HTTP/1.1.

The good news is that Infinispan supports both those upgrade paths. Thanks to the ALPN Hack Engine (the credit goes to Stuart Douglas from the Wildfly Team), we support TLS/ALPN without any bootstrap classpath modification.

Configuring Infinispan server for HTTP/2

Infinispan’s REST server already supports plain text upgrades out of the box. TLS/ALPN however, requires additional configuration since the server needs to use a Keystore. In order to make it even more convenient, we support generating keystores automatically when needed. Here’s an example showing how to configure a security realm:

The next step is to bind the security realm to a REST endpoint:

You may also use one of our configuration examples. The easiest way to get it working is to use our Docker image:

Let’s explain a couple of things from the command above:
  • -e "APP_USER=test" - This is a user name we will be used for REST authentication.
  • -e "APP_PASS=test" - Corresponding password.
  • ../../docs/examples/configs/standalone-rest-ssl.xml - Here is a ready-to-go configuration with REST and TLS/ALPN support
Unfortunately, HTTP/2 functionality has been broken in 9.2.0.Final. But we promise to fix it as soon as we can :) Please use 9.1.5.Final in the meantime.

Testing using CURL

Curl is one of my favorite tools. It’s very simple, powerful, and… it supports HTTP/2. Assuming that you already started Infinispan server using `docker run` command, you can put something into the cache:

Once, it’s there, let’s try to get it back:

Let’s analyze CURL switches one by one:
  • -k - Ignores certificate validation. All automatically generated certificates and self-signed and not trusted by default.
  • -v - Debug logging.
  • -u test:test - Username and password for authentication.
  • -d test - This is the payload when invoking HTTP POST.
  • -H “Accept: text/plain” - This tells the server what type of data we’d like to get in return.

Conclusions and links

I hope you enjoyed this small tutorial about HTTP/2. I highly encourage you to have a look at the links below to learn some more things about this topic. You may also measure the performance of your app when using HTTP/1.1 and HTTP/2. You will be surprised!

Tuesday, 6 March 2018

Accessing Infinispan inside Docker for Mac

Connecting to Infinispan instances that run inside Docker for Mac using the Java Hot Rod client can be tricky. In this blog post we'll be analyzing what makes this environment tricky and how to get around the issue.

The tricky thing about Docker for Mac is that internal container IP addresses are not accessible externally. This is a known issue and it can be hard to workaround it. In container orchestrators such as Openshift, you can use Routes to allow external access to the containers. However, if running vanilla Docker for Mac, the simplest option is to map ports over to the local machine.

Why is this important? When someone connects using the Hot Rod protocol, the server returns the current topology to the client. When Infinispan runs inside of Docker, this topology by default contains internal IP addresses. Since those are not accessible externally in Docker for Mac, the client won't be able to connect.

To workaround the issue, Infinispan server Hot Rod endpoint can be configured with external host/port combination, but doing this would require modifying the server's configuration. A simpler method to get around the issue is to configure the client's intelligence to be Basic. By doing this the server won't send topology updates nor will the client be able to locate where keys are located using hashing. This has a negative performance impact since all requests to Infinispan single server or server cluster would need to go over the same IP+port. However, for demo or sample applications on Mac environments, this is reasonable thing to do.

So, how do we do all of this?

First, start Infinispan server and map Hot Rod's default port 11222 to the local 11222 port:

docker run -it -p 11222:11222 jboss/infinispan-server:9.2.0.Final

Open your IDE and create a project with this dependencies:



Finally, create a class that connects to Infinispan and does a simple put/get sequence:



Cheers,
Galder

Monday, 5 March 2018

A SWIG based framework to build Hotrod client prototype in your preferred language

If your are working on a non Java/C++/C#/JS application and you need to interact with Infinispan via Hotrod you may be interested in the idea behind the HotSwig[1] project.

Hotswig proposes a framework to build Hotrod client prototypes quickly and for a generic SWIG[2] supported language.
As people familiar with C++ and C# Infinispan native clients know, SWIG plays a role in both the projects:

  • is used to build the base of the C# client wrapping the C++ core with a C# layer;
  • is used in the C++ project to run (part of) the Java test suite against the client, in this way: a Java wrapper is built via SWIG to make the C++ client looks like its Java big brother so it can be tested with the Java test suite.

The main goal was to produce for a specific language an almost complete client reusing the C++ core features and the following workflow has been setup to do that:

  • the whole C++ interface is processed by SWIG. The resulting wrapper exposes almost all the C++ functions;
  • a user friendly adaptation layer is build on top of the SWIG result.

This approach doesn't work for the HotSwig goal, mainly because the effort need by the second step is usually not-negligible and prevents the rapid development of prototype in a generic language.

In the HotSwig approach, this limitation is removed moving the adaptation layer from the target language to the C++ side and then letting SWIG generate a ready to use client prototype. So the HotSwig workflow is the following:

  • build an adaptation facade around the C++ core to make it SWIG friendly (do the adaptation work once for all on the C++ side);
  • explicitly define what we want in the produced SWIG wrapper (keep things simple excluding everything by default);
  • run SWIG to produce the client.

At the moment HotSwig is just a proof of concept, but you can try to run it and produce a ready to work Infinispan client for the language you need. Examples are already provided for python, ruby and Octave, but HotSwig should work with all the SWIG supported languages. If you get it to run in your preferred programming language, please share your experience with us.

I've listed here[3] some tasks for the roadmap, with the idea to test the flexibility of the framework trying to extend it in different directions. Maybe the idea is good and it can grow up from a PoC to something that can really help devs. You can add you ideas of course.

So if you need to do math against your Infinispan data set why don't you try the Octave client? Or maybe you want to do analytics with R, or presentation with PHP. Or you just like parenthesis and you want to use Lisp. Or you're working for the Klingon empire and you must use ylDoghQo'[4]... well ok just joking now...

Thanks for reading!

Cheers
The Infinispan Team


[1] https://github.com/rigazilla/hotswig
[2] http://www.swig.org/
[3] https://github.com/rigazilla/hotswig/issues
[4] https://www.kli.org/about-klingon/klingon-phrases