The following describe how expiration works in certain circumstances with Infinispan 7 or older.
Max IdleMax Idle, I would say is the black sheep of clustered expiration. It works great with a single node, but if you have a cluster where a key is accessed on different nodes the recent access time is not in sync. When a clustered cache is used with max idle this can make some nodes contain data and some others to not, it can be a bit confusing.
Single node expirationExpiration has only ever took place on a per node basis. That is that the entry is only removed from a node when it has that key accessed or the reaper thread finds it. This means that the different nodes can have different amount of entries (although expired entries - don't show up).
Expiration Event?When an entry expires it should raise an expiration event, correct? Infinispan didn't have such an event, instead an invalidation event was raised. This could be for obvious reasons a bit confusing. The worst part is that the event is raised on each node at possibly different times since the entries aren't removed at approximately the same time.
The following are new enhancements added with Infinispan 8 to allow for better handling of entries expiring.
Cluster wide expirationWhen an entry in a replicated or distributed cache expires it will now expire that entry across the entire cluster at once.
When an entry expires on one node (either by access or reaper thread) that node will asynchronously send a remove expired command. This command runs just like a remove except it has some conditional values such as checking for the lifespan and value to make sure they match before actually removing the entry. This is to prevent a concurrent write from being overwritten. This then ensures that the entry is removed from all nodes at approximately the same time.
Unfortunately cluster wide expiration is not as safe when an entry expires from a store and it wasn't in memory. In very rare circumstances, it can overwrite a concurrent update. This case is very rare because entries only expire from a store when the reaper thread runs, you would then have to have a concurrent put at the precise moment the reaper thread is expiring that entry. This is because the API for cache store expiration only exposes the key and doesn't include the value or metadata. This is planned to be enhanced hopefully somewhat soon though!
Yes, Expiration Events!Infinispan also has added a new event, CacheEntryExpired. This is fired whenever an entry expires or is removed due to a cluster wide expiration. Note in the latter case, the event is raised across the entire cluster at the same time, which also includes only receiving a cluster expiration event.
Also there was a bug here where the event was only raised for store expirations, in memory expirations never even raised an event, doh!