...
During normal CAS processing, the addTicket() and deleteTicket() methods lock the registry for just long enough to add an item to the end of the one of the two incremental collections. Cushy uses locks only for very simple updates and copies so it cannot deadlock and performance should not be affected. This is the only part of Cushy that runs under the normal CAS HTTP request processing.
Quartz maintains a pool of threads independent of the threads used by JBoss or Tomcat to handle HTTP requests. Periodically a timer event is triggered, Quartz assigns a thread from the pool to handle it, the thread calls the timerDriven() method of the primary CushyTicketRegistry object, and for the purpose of this example, let us assume that it is time for a new full checkpoint.
Java provides a complex built in class called ConcurrentHashMap that handles the coordination of requests to the cache of tickets. Using that build in service, the primary CushyTicketRegistry object generates a separate point-in-time snapshot allows a collection of Tickets to be shared by request threads. The JASIG DefaultTicketRegistry uses this service, and Cushy adopts the same design. One method exposed by this built in class provides a new list of references to all the Tickets in the cache. This does not copy the tickets, it only creates a list of pointers to the tickets that existed at that time. Subsequent adds or deletes to the real cache do not affect the snapshot. This separate collection can then be serialized to a disk file in a single Java writeObject() call where all the work is done automatically by Java. After that, we just have to close the file.
However, before returning to Quartz, we call the Notify logic that sends an HTTP GET request to the /cluster/notify URL on each other CAS node. This step occurs under the Quartz thread, and it has to wait until each node returns from the GET request.
On the other node, a standard HTTP request arrives which is processed by the container (JBoss or Tomcat) just like any other request. Spring routes the /cluster/notify suffix to the CacheNotifyController class which is added by the CushyTicketRegistry component. CacheNotifyController calls the primary CushyTicketRegistry object on that node which in turn selects the appropriate secondary object corresponding to the node that sent the Notify request.
Now there are two ways to handle the next step. The simplest logic, but not necessarily the best performance, is for the secondary object to issue the /cluster/getCheckpoint request back to pick up a copy of the just generated checkpoint file, and then restore the data from that file back into the cache memory, before returning from the call. That means, however, that the GET request doesn't end until the data has been completely processed, and remember that the Quartz thread on the node that took the checkpoint is waiting for the response from this node before it can call Notify on the next node. If the network is particularly slow or the file is particularly large or there are a lot of CAS nodes, then sequentially processing each node, and waiting for each node to fetch and load the data before going on to the next node, may generate an unreasonable delay.
If this is an issue, there is an option in the CushyTicketRegistry class. Set "useThread" to true and each secondary CushyTicketRegistry object contains its own thread (the NotifyProcessThread). Then the /cluster/notify GET returns immediately, and the rest of the processing (to fetch the file over the network and load the data into memory) runs under the NotifyProcessThread AFTER the Notify processing appears to have completed from the point of view of the other nodes.
Between full checkpoints, the Quartz timer thread generates the incremental file on disk and then fetches incremental files from the other nodes of the cluster. There is no Notify, and incremental files are so small that the time it takes to write them to disk or read them over the network is not enough to optimize. So everything happens under the Quartz timer thread and the thread can be expected to end well before the next timer tickTicket objects at some point in time. Cushy uses this service to obtain its own private list of all the Tickets that it can checkpoint without affecting any other thread doing normal CAS business.
The collection returned by ConcurrentHashMap is not serializable, so Cushy has to copy Tickets from it to a more standard colleciton, and it uses this opportunity to exclude expired tickets. Then it uses a single Java writeObject statement to write the List and a copy of all the Ticket objects to a checkpoint file on disk. Internally Java does all the hard work of figuring out what objects point to other objects so it can write only one copy of each unique object. When it returns, Cushy just has to close the file.
Between checkpoints the same logic applies, only instead of writing the complete set of Tickets, Cushy only serializes the addedTickets and the deletedTicket Ids to the disk file.
After writing a full checkpoint, Cushy generates a new dummyServiceTicket ID string and issues a Notify (calls the /cluster/notify URL of CAS on all the other nodes of the cluster) passing the dummyServiceTicket string so the other nodes can use it as a password to access the checkpoint and incremental files over the Web.
On the other nodes, the Notify request arrives through HTTP like any other CAS request (like a ST validate request). Spring routes the /cluster/notify suffix to the small Cushy CacheNotifyController Java class. We want all the other nodes to get a new copy of the new full checkpoint file as soon as possible there are two strategies to accomplish this.
Cushy does not expect a meaningful return from the /cluster/notify HTTP request. The purpose is just to trigger action on the other node, and the response is empty. Therefore, one simple strategy is to set an short Read Timeout on the HTTP request. The other node receives the Notify and begins to read the checkpoint file. Meanwhile, the node doing the Notify times out having not yet received a response, and so it goes on to Notify the next node in the cluster. Eventually when the checkpoint file has been fetched and restored to memory the Notify logic returns to the CacheNotifyController bean which then tries to generate an empty reply but discovers that the client node is no longer waiting for a reply. Things may end with a few sloppy exceptions, but the code expects and ignores them.
The other approach has the Notify request on the receiving node wake up a thread in the Secondary CusyTicketRegistry object coresponding to the node that sent the Notify. That thread can fetch the checkpoint file and restore the tickets to memory. Meanwhile, the CacheNotifyController returns immediately and sends the null response back to the notifying node. Nothing times out and no exceptions are generated, but now you have to use threading, which is a bit more heavy duty technology than Web applications prefer to use.
There is no notify for an incremental file. The nodes do not synchronize incrementals (too much overhead). So when the timerDriven() method is called between checkpoints, it writes an incremental file for the current node and then checks each Secondary object and attempts to read an incremental file from each other node in the cluster.
There is a chase condition between one node taking a full checkpoint when another node is trying to read an incremental. A new checkpoint deletes the previous incremental file. As each of the other nodes receives a Notify from this node they realize that there is a new checkpoint and no incremental, so a flag gets set and the next timer cycle through no incremental is read. However, after the checkpoint is generate and before the Notify is sent there is a opportunity for the other node to wake up, ask for the incremental file to be sent, and to get back an HTTP status of FILE_NOT_FOUND.
Security
The collection of tickets contains sensitive data. With access to the TGT ID values, a remote user could impersonate anyone currently logged in to CAS. So when checkpoint and incremental files are transferred between nodes of the cluster, we need to be sure the data is encrypted and goes only to the intended CAS servers.
...