...
The various cache (in memory object replication) solutions should also work. Unfortunately, they have massively complex configuration parameters with multicast network addresses and timeouts.They also tend to be better at detecting a node that is dead and does not respond than they are at dealing with nodes that are sick and accept a message but then never really get to processing it and responding. They operate entirely in memory, so at least one node has to remain up while the others reboot in order to maintain the content of the cache. While node failure is well defined, the status of objects is ambiguous if the network is divided into two segments by a linkage failure, the two segments operate independently for a while, and then connection is reestablished.
Since Cushy is specifically designed to handle the CAS Ticket problem, you will not understand it without a more detailed discussion of CAS Ticket connections and relationships. There are some specific CAS design problems that cannot be solved at the TicketRegistry layer. Cushy doesn't fix them, but neither do any of the cache solutions. This document will identify them and suggest how to fix them elsewhere in the CAS code.
Cushy is a cute word that roughly stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale".
The name explains what it does. Java has a built in operation called writeObject that writes a binary version of Java objects to disk. If you use it on a complex object, like a list of all the tickets in the Registry, then it creates a disk file with all the tickets in the list. Later on you use readObject to turn the disk file back into a copy of the original list, and all the objects in the list are automatically connected back and forth with references identical to the original objects. Java calls this mechanism "Serialization". Using just one statement and letting Java do all the work and handle all the complexity makes this easy.
The other mechanisms (JPA or the cache technologies) operate on single tickets. They write individual tickets to the database or replicate them across the network. Obviously this is vastly more efficient than periodically copying all the tickets to disk. Except that at Yale (a typical medium sized university), the entire Registry of less than 20,000 tickets can be written to a disk file in 1 second and it produces a file about 3 megabytes in size. That is a trivial use of modern multicore server hardware, and copying 3 megabytes of data over the network every 5 minutes, or even every minute, is a trivial use of network bandwidth. So Cushy is less efficient, but in a way that is predictable and insignificant, in exchange for code that is simple and easy to completely understand.
...
Cache solutions go memory to memory. Adding an intermediate disk file wasn't an obvious step, but once you think of it it has some added benefits. If you reboot the CAS server, the local disk file allows CAS to immediately restore the tickets and therefore its state from before the reboot. Serializing the tickets to disk will work no matter how badly the network or other nodes are damaged, and it is the only step that involves the existing CAS code. Although the second step, transferring the file from one server to another, is accomplished with new code that runs in the CAS Web application, it does not touch a single existing CAS object or class. So whatever unexpected problems the network might create, they affect only the independent file transfer logic leaving normal CAS function untouched. And while the cache solutions require complex logic to reconcile cache on different machines after communication between nodes is restored, Cushy retransmits the entire set of tickets every (configurable number) few minutes after which everyone is guaranteed to be back in synchronization.
...
Back in the 1960's a "checkpoint" was a copy of the important information from a program written on disk so if the computer crashed the program could start back at almost the point it left off. If a CAS server saves its tickets to a disk file, reboots, and then reads the tickets from the file back into memory it is back to the same state it had before rebooting. If you transfer the file to another computer and bring CAS up on that machine, it you have moved the CAS server from one machine to another. Java's writeObject and readObject guarantee the state and data are completely saved and restored.
If you have no cluster and just a single CAS server, then replacing the DefaultTicketRegistry class with a CushyTicketRegistry class creates a CAS Server that you can shut down and restart without losing any previous logons.
JPA and the cache technologies try to maintain the image of a single big common bucket of shared tickets. This seems like it is fairly simple, but it creates synchronization problems between servers while serving no useful CAS purpose. Cushy maintains a separate TicketRegistry for each CAS server, but server that holds all the tickets created by that server. It then replicates a copy of the each TicketRegistry of each server to all the other servers in the cluster. No server has to look at the Registry of another server until the other server fails and the Front End starts routing its requests to the other nodes.
You could configure a Cushy cluster to only make full checkpoints of files containing all the tickets. The cost of a checkpoint is small, but it is large enough that you might be reluctant to schedule them frequently enough to provide the best protection. So between full checkpoints, Cushy creates and transmits a sequence of "incremental" change files that each have all the changes since the last full checkpoint. In the Spring XML configuration file you set the time between incrementals and the time between checkpoints. The choice is up to you, but a reasonable suggestion is to exchange incrementals every 5-15 seconds and checkpoints every 3-15 minutes. That is not even a recommendation, but we will use it as a rough estimate in the rest of this discussion.
Each incremental has a small number of new Login (TGT) tickets and maybe a few unclaimed service tickets. However, because we do not know whether any previous incremental was or was not processed, it is necessary to transmit the list of every ticket that was deleted since the last full checkpoint, and that will contain the ID of lots of Service Tickets that were created, validated, and deleted within a few milliseconds. That list is going to grow, and its size is limited by the fact that we can start over again after each full checkpoint.
A Service Ticket is created and then is immediately validated and deleted. Trying to replicate Service Tickets to the other nodes before the validation request comes in is an enormous problem that screws up the configuration and timing parameters for all the other Ticket Registry solutions. Cushy doesn't try to do replication at this speed. Instead, it has CAS configuration elements that ensure that each Ticket ID contains an identifier of the node that created it, and it depends on a front end smart enough to route any of the ticket validation requests to the node that created the ticket and already has it in memory. Then replication only is needed for crash recoverrecovery.
Note: If the front end is not fully programmable it is a small programming exercise to be considered in Cushy 2.0 to forward the validation request from any CAS node to the node that owns the ticket and then pass back the results of the validation to the app.
...