...
The TicketRegistry is the component of CAS that stores the ticket objects. There are at least 5 different versions of TicketRegistry that you can choose (Default, JPA, JBoss, Ehcache, Memcached) and Cushy simply adds one additional choice. While traditional applications are assembled at build time, CAS uses the Spring Framework to essentially create logical sockets for components that are plugged in at application startup time. This is driven by XML configuration files. The ticketRegistry.xml file configures whichever registry option you choose. For a simple single standalone CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in memory Java table keyed by the ticket id string.
Cushy is a cute name that roughly stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale".
The Standalone Server
Suppose you simply change the class name from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry code, so while CAS runs everything works exactly the same. The change occurs if you shut CAS down, particularly because you have to restart it. Since the DefaultTicketRegistry only has an in memory table, all the tickets are lost when the application restarts. Cushy detects the shutdown and saves all the ticket objects to a file on disk, using a single Java writeObject statement on the entire collection. Unless that file is deleted while CAS is down, then when CAS restarts Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown. Users do not have to login again, and no user even notices that CAS restarted unless they tried to access CAS while it was down.
...
The big difference here is that Cushy is designed 100% to satisfy the needs of CAS, and so we can discuss and fix those specific problems. The larger off the shelf generic libraries provide no place to fix problems specific to CAS and up to this point nobody seems to have noticed or fixed the problems.
Summary of CAS Clustering
(For those unfamiliar with the CAS system)
CAS is a Single SignOn solution. Internally, it creates a set of objects called Tickets. There is a ticket for every logged on user, and short term Service Tickets that exist while a user is being authenticated to an application. The Business Layer of CAS creates tickets by, for example, validating your userid and password in a back end system like Active Directory. The tickets are stored in a plug in component called a Ticket Registry. The tickets are the only data CAS maintains about its users or previous activity.
For a single CAS server, the Ticket Registry is just a in memory table of tickets indexed by the ticket ID string. When more than one CAS server is combined to form a cluster, then an administrator chooses one of several optional Ticket Registry solutions that allow the CAS servers to share the tickets.
...
Why another Clustering Mechanism?
One solution is to share all the ticket objects and their associated components in database tables using JPA. JPA is the standard Java mechanism for mapping objects to tables. It is an enormously powerful tool for ordinary Web applications. It is a possible solution, but CAS doesn't have a database problem:
- CAS tickets all timeout after a number of hours. They have no need for long term persistence.
- There are no meaningful SQL operations in CAS. Nobody will generate reports based on tickets.
- CAS has no transactional structure or need for a conventional commit operation.
Most importantly, having created a cluster for availability, JPA now makes the database a single point of failure. Configuring a database for 24x7x366 availability and guaranteeing that it comes up before CAS places a significant and unnecessary burden on most CAS installations.
The alternative is to use one of several "cache" libraries (Ehcache, JBoss Cache, Memcached) where CAS puts the tickets into what appears to be a common container of Java objects and, under the covers, the cache technology ensures that new tickets are copied to all the other nodes.
JPA makes CAS dependent on a database. It doesn't really use the database for any queries or reports. You can use any database, but the database is a single point of failure. At Yale CAS is nearly the first thing that has to come up during disaster recovery, but if it uses JPA then you have to bring up the database (or have a special standalone CAS configuration for disaster recovery only). If you already have a 24x7x365 database managed by professionals who can guarantee availability, this is a good solution. If not, then this is an insurmountable prerequisite for bringing up an application like CAS.
The various "cache" (in memory object replication) solutions should also work. Unfortunately, some have massively complex configuration parameters with multicast network addresses and timeout values to determine node failure.They also tend to be better at detecting a node that is dead than they are at dealing with nodes that are sick and accept a message but then never really get to processing it and responding. They operate entirely in memory, so at least one node has to remain up while the others reboot in order to maintain the content of the cache. While node failure is well defined, the status of objects is ambiguous if the network is divided into two segments by a linkage failure, the two segments operate independently for a while, and then connection is reestablished.
Cushy is a cute name that roughly stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale".
The name explains what it does. Java has a built in operation named writeObject that writes a binary ("serialized") version of Java objects to disk. You can use it on a single object, but if you pass it a list or table of objects then it copies everything in the list and captures all the relationships between the objects. Later on you can use readObject from the same program, from a different JVM, or from a different computer and restore to memory an exact copy of the original list or table and all the objects it contains. This is a very complex process, but Java handles all the complexity inside the writeObject statement.
Comparison of Cushy and previous cluster technologies:
...
. They create the impression of a large pool of ordinary Java objects shared by all the CAS servers. Any change made to objects in the pool are automatically and transparently replicated to all the other servers. These systems also solve very large problems and they can have very complicated configurations with exotic network parameters.
A common problem with both JPA and the generic "cache" solutions is that they integrate into CAS "inline". JPA is driven by annotations that are added to the Java source of the Ticket classes, but under the covers it dynamically generates code that it transparently "weaves" into the classes. The cache systems intercept TicketRegistry operations such as addTicket to make sure that copies of the tickets are moved to some network communications queue. In either case we have observed that when things get bad, when the network is sick or something generates an unexpected error, the problem can "back up" from the replication mechanism back into the TicketRegistry and then into CAS itself.
In Cushy, the only connection between the CAS mainline function (the part of CAS that responds to Web requests and services users and applications) is that references to objects are occasionally copied from one in memory collection to another. Separately, on a timer driven basis collections of objects are periodically written to disk with a single Java writeObject statement. Separately, on a network request driven basis, copies of those files are then send to other CAS nodes. If there is a network problem, the HTTP GET fails with an I/O error, this operation is aborted completely, then the servers try again 10 or 15 seconds later. Each step places an absolute boundary between itself and the other steps. None of them can interfere with CAS mainline services. There are no queues or operations to clog and back up into the running code.
Comparison of Cushy and previous cluster technologies:
- Existing cluster technologies maintain the image of a single pool of shared tickets. Cushy exploits modern programmable Front End network devices (such as the BIG-IP F5) to distribute initial CAS logons across different members of the cluster, but then to route subsequent CAS requests to the node that handled the specific user logon unless that node crashes. Each Cushy node maintains its own set of tickets.
- Existing cluster technologies try to replicate individual tickets (although the nature of Java's writeObject drags along copies of additional associated tickets). Cushy replicates a batch of tickets at regular time intervals (say every 10 seconds) and less frequently it replicates a copy of the entire collection of tickets.
- Existing cluster technologies use complex logic and databases or complex network configuration. Cushy uses HTTP that everyone understands, although you can replace this with shared files or your own trivial programs. As a result you can know how things work and how they will respond to any type of network or system failure.
- Existing cluster technologies require a cluster. Cushy does something useful on a single machine, and its clustering capability is simply an extension of that simple design.
- Existing cluster technologies are general purpose off the shelf libraries designed to handle any application. Cushy was written to handle CAS tickets. There are unresolved problems when CAS tickets are replicated using generic replication. In its initial distribution as a TicketRegistry component, Cushy cannot solve bugs in other CAS components, but because it exposes 100% of the logic as simple Java it provides the framework to resolve these problems when you start to use the new features of CAS 4.
- Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiency.
...
Thou Shall Not
Cushy is based on four basic design principles:
- CAS is very important, but it is small and cheap to run.
- Emphasize simplicity over efficiency as long as the cost to run remains trivial.
- Assume the network front end is programmable.
- Trying for perfection is the source of most total system failures. Allow one or two users to get a temporary error message when a CAS server fails., but make it impossible for any single failure to crash everything.
A Bit More Detail on CAS Tickets
...
In various node failure scenarios, at one of the "pointing to" breaks you can jump from the current node's TicketRegistry to a backup shadow TicketRegistry copy of tickets belonging to a failed node. For example, the ST could point to the PGT and TGT in the failed node's registry, or the ST could point to a local PGT that then points to a TGT in the failed node's registry. Create the possibilities and verify that they work, but also remember that these have to work in order for the design to handle failures properly.
How it works
...
Keep It Simple
Back in the 1960's a "checkpoint" was a copy of the important information from a program written on disk so if the computer crashed the program could start back at almost the point it left off. If a CAS server saves its tickets to a checkpoint disk file, reboots, and then restores the tickets from the file back into memory it is back to the same state it had before rebooting. If you transfer the file to another computer and bring CAS up on that machine, you have moved the CAS server from one machine to another. Java's writeObject and readObject guarantee the state and data are completely saved and restored.
If you have no cluster and just a single CAS server, then replacing the DefaultTicketRegistry class with a CushyTicketRegistry class creates a CAS Server that you can shut down and restart without losing any previous logons.
JPA and the cache technologies try to maintain the image of a single big common bucket of shared tickets.This was necessary when the network Front End device simply accepted HTTP requests and assigned them to CAS servers is a round robin manner. Today network Front End devices are programmable and they can make decisions based on specific CAS logic. This allows each CAS server to own its own private slice of the problem.
When a new user is redirected to CAS, then the Front End can randomly choose a server. However, after the user logs in and is assigned a Cookie, the Front End should always route subsequent requests to the server that issued the cookie. That means that Service Tickets and Proxy Tickets are issued by the CAS server you logged into. The Front End can also be programmed to recognize validation requests (/validate, /serviceValidate, etc.) and route those requests to the server that issued the ticket identified by the ticket= parameter. Configuration for the BIG-IP F5 will be provided. If you do not have a smart Front End device, then use a different Ticket Registry technology.
With an intelligent Front End, there is no need for a Ticket Registry that simulates a big shared pool of tickets. Each node has its own registry with its own logged in users and the tickets they create. No other node needs to access these tickets, unless the node that owns them fails. Then any other node, or all the other nodes, handle requests until the failed node is restarted.
You could configure a Cushy cluster to only make full checkpoints files containing all the tickets. The cost of a checkpoint is small, but it is large enough that you might be reluctant to schedule them frequently enough to provide the best protection. So between full checkpoints, Cushy creates and transmits a sequence of "incremental" change files that each have all the changes since the last full checkpoint. In the Spring XML configuration file you set the time between incrementals and the time between checkpoints. The choice is up to you, but a reasonable suggestion is to exchange incrementals every 5-15 seconds and checkpoints every 3-15 minutes.
Each incremental has a small number of new Login (TGT) tickets and maybe a few unclaimed service tickets. However, because we do not know whether any previous incremental was or was not processed, it is necessary to transmit the list of every ticket that was deleted since the last full checkpoint, and that will contain the ID of lots of Service Tickets that were created, validated, and deleted within a few milliseconds. That list is going to grow, and its size is limited by the fact that we can start over again after each full checkpoint.
Note: Replicating Service Tickets between nodes is almost never useful. The "p:excludeSTFromFiles" parameter in the Spring configuration XML causes Cushy to ignore Service Tickets when writing files, which keeps the deleted tickets list small and limits the growth of incrementals, if you prefer a very long time between full checkpointsSo the basic idea of Cushy is about 50 years old.
Writing the checkpoint file is fairly trivial, so would typically configure the time between checkpoints in minutes.
Between checkpoints you can write the changed objects to a separate file. You could write a sequence of files with changes and require that they be applied in order, but that it more complicated. The amount of data between full checkpoints is small, so it makes sense to maintain a single file with all the accumulated changes. Whenever you see a new file you can apply it to get a current set of tickets without worrying about whether previous versions of the file have been applied or if any version has been missed.
A user requests a Service Ticket. CAS generates the ST and writes its ID string back to the browser. The browser then forwards the ID string on to the application, which opens a separate HTTPS session to CAS to validate the ID and return the netid and possible attributes. This can be handled by replicating the Service Ticket across the network in milliseconds, or even worse by requiring that it has been replicated before you return the ID string to the browser. It is much simpler to program the network Front End device (by writing an iRule for the F5) to recognize the /cas/validate, /cas/serviceValidate, and similar path elements and route all such requests to the node whose id is appended to the ticket= parameter. Then you can replicate in seconds instead of milliseconds and that make everything much easier.
Ticket Names
As with everything else, CAS has a Spring bean configuration file (uniqueIdGenerators.xml) to configure how ticket ids are generated. The default class generates tickets in the following format:
...