Content Comparison

• Recover tickets after reboot without JPA, or a separate server, or a cluster (works on a standalone server)
• Recover tickets after a crash, except for the last few seconds of activity that did not get to disk.
• No dependency on any external library. Pure Java using only the standard Java SE runtime.
• All source in one class. A Java programmer can read it and understand it.
• Can also be used to cluster CAS servers, if the Front End is programmed to route requests with an understanding of CAS protocol.
• Cannot crash CAS ever, no matter what is wrong with the network or other servers.
• A completely different and simpler approach to the TicketRegistry. Easier to work with and extend.
• Probably less efficient than the other TicketRegistry solutions, but it has a constant predictable overhead you can measure and "price out".

...

CAS stores its tickets in a administrator plug-in selectable component called a TicketRegistry. CAS provides one implementation of the TicketRegistry for single-server configurations, and at least four alternatives that can be used to share tickets among several CAS servers operating in a network cluster. This document describes a new implementation called CushyTicketRegistry that is simple, provides added function for the standalone server, and yet also operates in clustered configurations.

Four years ago Yale implemented a "High Availability" CAS cluster using JBoss Cache to replicate tickets. After that, the only CAS crashes were caused by failures of the ticket replication mechanism. Red Hat failed to diagnose or fix the problem. We considered replacing JBoss Cache, but there is a more fundamental problem here. It should not be possible for any failure of the data replication mechanism to crash all of the CAS servers at once. One of the other mechanisms might fail less frequently, but we were reluctant to switch from one "big magic black box" library to another.

It will always be necessary to copy data to a backup location in order to recover from a crash. However, if you assume that the ticket validation request from the application will be randomly distributed to any CAS server in a cluster, then the Service Ticket has to be copied to all the other servers really, really fast.

Modern network front end devices have become very smart. They have to be smart, because this is a competitive market and you can't afford to sell a box that is less capable than the competition at accelerating applications, or fending off DOS attacks, or intelligently routing requests. These boxes already know about JSESSIONID and will send a user back to the particular Web Server in the cluster that he previously connected to.

While CAS performs a Single Sign On function, the logic is actually designed to create, read, update, and delete tickets. The ticket is the center of each CAS operation. In different requests there are only three places to find the key ticket that defines this operation:

1. In the ticket= parameter at the end of the URL for validation requests.
2. In the pgt= parameter for a proxy request.
3. In the CASTGC Cookie for browser requests.

Programming the front end to know that "/validate", "/serviceValidate", and two other strings in the URL path means that this is case 1, and "/proxy" means it is case 2, and everything else is case 3 is pretty simple.

Of course, finding the ticket is not helpful unless you use the feature that has always been part of CAS to put the nodename or any other identifier for the CAS server at the end of every ticket generated by that server. If you start to use that option, now an intelligent front end can send every request to the node that created the ticket and therefore is best able to process the request.

Adding some intelligent routing to your network front end reduces the burden on replication for all the ticket registry solutions. It means that you can relax the timing parameters, and make some replication asynchronous instead of synchronous, and still the validation request will never arrive at a server before the Service Ticket being validated arrives. However, while this may be a good thing to do for all the other Ticket Registry solutions, it is required by Cushy. Cushy doesn't have a very, very fast replication mode.

"Cushy" stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale". This summarizes what it is and how it works.

There are two big ideas.

1. Modern Front End devices can be programmed to understand CAS protocol and make smarter decisions about how to route requests.
2. If you clone the Collection that points to all the ticket objects, you can make a copy of the entire current state of the CAS server to a disk file with a single writeObject statement. That can be restored to memory with a single readObject statement. All the complexity is handled automatically by Java!

After that the design of Cushy almost writes itself. It is clear that this is an entirely different approach to the TicketRegistry. It is so simple it HAS to work. Whether it would be efficient enough to actually use in production was the question that could not be answered until the code was written and tested. It was clear that it would be useful to small CAS deployments, particularly single servers, even if it turned out to be unworkable for large institutions. It was not a lot of code, and after the first few days there was enough working to run some test cases and benchmark the results. It turned out to be much better than expected. Still to understand the code, begin with the small size and work up.

The Standalone Server

For a single CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in-memory Java table keyed by the ticket id string. Suppose you change the name of the Java class in the Spring ticketRegistry.xml file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry source code, so everything works the same as it did before, until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when CAS restarts and users all have to login again. Cushy detects the shutdown and using a single Java writeObject statement it saves all the ticket objects in the Registry to a file on disk (called the "checkpoint" file). When CAS restarts, Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown (although a few tickets may have expired). No user even notices that CAS restarted unless they tried to access CAS during the restart.

The number of tickets CAS holds grows during the day and shrinks over night. At Yale there are fewer than 20,000 ticket objects in CAS memory, and Cushy can write all those tickets to disk in less than a second generating a file around 3 megabytes. Other numbers of tickets scale proportionately (you can run a JUnit test and generate your own numbers). This is such a small amount of overhead that Cushy can be proactive.

So to take the next logical step, start with the previous ticketRegistry.xml configuration and duplicate the XML elements that currently call a function in the RegistryCleaner every few minutes. In the new copy of the elements, call the "timerDriven" function in the (Cushy)ticketRegistry bean every few minutes. Now Cushy will not wait for shutdown but will back up the ticket objects regularly just in case the CAS machine crashes without shutting down normally. When CAS restarts after a crash, it can load a fairly current copy of the ticket objects which will satisfy the 99.9% of the users who did not login in the last minutes before the crash.

The next step should be obvious. Can we turn "last few minutes" into "last few seconds". You could create a full checkpoint of all the tickets every few seconds, but now the overhead becomes significant. So go back to ticketRegistry.xml and set the parameters to call the "timerDriven" function every 10 seconds, but set the "checkpointInterval" parameter on the CushyTicketRegistry object to only create a new checkpoint file every 300 seconds. Now Cushy creates the checkpoint file, and then the next 29 times it is called by the timer it generates an "incremental" file containing only the changes since the checkpoint was written. Incremental files are cumulative, so there is only one file, not 29 separate files. If CAS crashes and restarts, Cushy reads the last checkpoint, then applies the changes in the last incremental, and now it has all the tickets up to the last 10 seconds before the crash. That satisfies 99.99% of the users and it is probably a good place to quit.

What about disaster recovery? The checkpoint and incremental files are ordinary sequential binary files on disk. When Cushy writes a new file it creates a temporary name and then swaps it for the old file, so other programs authorized to access the directory can safely open or copy the files while CAS is running. Feel free to write a shell script or Pearl or Python program to use SFTP or any other program or protocol to back up the data offsite or to the cloud.

Some people use JPATicketRegistry and store a copy of the tickets in a database to accomplish the same single server restart capability that Cushy provides. If you are happy with that solution, stick with it. Cushy doesn't require the database, it doesn't require JPA, and it may be easier to work with.

Before you configure a cluster, remember that today a server is typically a virtual machine that is not bound to any particular physical hardware. Ten years ago moving a service to a backup machine involved manual work that took time. Today there is VM infrastructure and automated monitoring and control tools. A failed server can be migrated and restarted automatically or with a few commands. If you can get the CAS server restarted fast enough that almost nobody notices, then you have solved the problem that clustering was originally designed to solve without adding a second running node.

The Cushy Cluster

Cushy can be configured node by node, but Yale Production Services did not want to manage individual machine specific files. So Cushy adds a cluster configuration class.   Actually, you configure every CAS cluster you have in your enterprise (desktop sandbox, development, test, stress test, production, ...). When CAS starts up the configuration class figures out which of the configured clusters includes the current server machine. It generates parameters that configure the CushyTicketRegistry. It also generates and feeds a server specific "ticket ID suffix" string to the Spring XML that configures ticket ID generation.

Thanks to the smart Front End, a Cushy cluster is really a collection of standalone CAS servers that back each other up in the event one goes down. To each node the "cluster" is simply a list of other CAS servers that it is supposed to fill in for if one of them fails. When Cushy starts up with a cluster definition, then in addition to its normal initialization, it also creates a "secondary" TicketRegistry object for every other machine in the cluster.Another choice of cache might be more reliable, but it would suffer from the same fundamental structural problem.

All of the previous CAS cluster solutions create a common pool of tickets shared by all of the cluster members. They are designed and configured so that the Front End can distribute requests in a round-robin approach and any server can handle any request. However, once the Service Ticket is returned by one server, the request to validate the ST comes back in milliseconds. So JPA must write the ST to the database, and Ehcache must synchronously replicate the ST to all the other servers, before the ST ID is passed back to the browser. This imposes a sever performance constraint that requires the CAS servers to be connected by very high speed networking.

However, reliability is best served by keeping different members of the cluster at physical distances so they will not all go down if there is a single data center or power failure. So the requirements of recovery are at odds with the standard configuration to handle ST validation.

Ten years ago, when CAS was being designed, the Front End that distributed requests to members of the cluster was typically an ordinary computer running simple software. Today networks have become vastly more sophisticated, and Front End devices are specialized machines with powerful software. They are designed to detect and fend off Denial of Service Attacks. They improve application performance by offloading SSL/TLS processing. They can do "deep packet inspection" to understand the traffic passing through and route requests to the most appropriate server (called "Layer 5-7 Routing" because requests are routed based on higher level protocols rather than just IP address or TCP session). Although this new hardware is widely deployed, CAS clustering has not changed and has no explicit option to take advantage of it.

Front End devices know many protocols and a few common server conventions. For everything else they expose a simple programming language. While CAS performs a Single Sign On function, the logic is actually designed to create, read, update, and delete tickets. The ticketid is the center of each CAS operation. In different requests there are only three places to find the ticketid that defines this operation:

1. In the ticket= parameter at the end of the URL for validation requests.
2. In the pgt= parameter for a proxy request.
3. In the CASTGC Cookie for browser requests.

Programming the Front End to know that "/validate", "/serviceValidate", and two other strings in the URL path means that this is case 1, and "/proxy" means it is case 2, and everything else is case 3 is pretty simple.

Of course, finding the ticket is not helpful unless you use a feature that has always been part of CAS configuration but previously was not particularly useful. Each server can put a specific identifier on the end of every ticketid it creates. This is the "suffix" of the ticket in the configuration parameters, but typically it has been left as the default string "-CAS". If the suffix is configured seriously, and if it is set to a value the Front End can use to identify the node, then combined with the previous three steps the Front End can be configured to route ticket requests preferentially to the node that created the ticket and therefore holds it in memory without depending first on cluster replication.

Of course, tickets still have to be replicated for recovery purposes, but that means that tickets can be replicated in seconds instead of milliseconds, and they can be queued and replicated periodically instead of synchronously (while the request waits). This makes the clustering mechanism much easier.

Of course, the Front End is owned by the Networking staff, and they are not always responsive to the needs of the CAS administrator. Although it is obviously more efficient to program the Front End, the CushyFrontEndFilter can be added to the Servlet configuration of the CAS server to do in Java the same thing the Front End should be doing, at least until your network administrators adopt a more enlightened point of view.

"Cushy" stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale". This summarizes what it is and how it works.

There are two big ideas.

1. Before the request has been turned over to Spring or any of the main CAS code, route or forward requests for operations on tickets generated by another server to the server that created the ticket and is guaranteed to have it in memory.
2. While it is more efficient to replicate an individual ticket, it is more efficient to send a block of tickets together than to send one at a time, and every few minutes it is not unreasonably expensive to replicate the entire collection of tickets. Java does this with a single writeObject statement (if you tell it to write out a Collection object).

Given these two ideas, the basic Cushy code could be written in a couple of days. Start with the DefaultTicketRegistry code that CAS uses to hold tickets in memory on a single CAS standalone server. Then add the writeObject statement (surrounded by the code to open and close the file) to create a checkpoint copy of all the tickets, and a corresponding readObject and surrounding code to restore the tickets to memory. The first thought was to do the writeObject to a network socket, because that was what all the other TicketRegistry implementations were doing, but then it became clear that it was simpler, and more generally useful, and a safer design, if the data was first written to a local disk file. Then the file could be transmitted over the network in a completely independent operation. This provided code that was useful for both standalone and clustered CAS servers, and it guaranteed that the network operations were completely separated from the main CAS function.

This was so simple and modular that it had to work. It was not clear that it would be good enough to be useful, but it was worth a few days of coding to get some preliminary results. The benchmarks turned out to be even better than had been expected, and that justified further work to complete the option.

CushyTicketRegistry and the Standalone Server

For a single CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in-memory Java table keyed by the ticket id string. Suppose you change the name of the Java class in the Spring ticketRegistry.xml file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry source code, so everything works the same as it did before, until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when CAS restarts and users all have to login again. Cushy detects the shutdown and using a single Java writeObject statement it saves all the ticket objects in the Registry to a file on disk (called the "checkpoint" file). When CAS restarts, Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown (although a few tickets may have expired). No user even notices that CAS restarted unless they tried to access CAS during the restart.

The number of tickets CAS holds grows during the day and shrinks over night. At Yale there are fewer than 20,000 ticket objects in CAS memory, and Cushy can write all those tickets to disk in less than a second generating a file around 3 megabytes. Other numbers of tickets scale proportionately (you can run a JUnit test and generate your own numbers). This is such a small amount of overhead that Cushy can be proactive.

So to take the next logical step, start with the previous ticketRegistry.xml configuration and duplicate the XML elements that currently call a function in the RegistryCleaner every few minutes. In the new copy of the elements, call the "timerDriven" function in the (Cushy)ticketRegistry bean every few minutes. Now Cushy will not wait for shutdown but will back up the ticket objects regularly just in case the CAS machine crashes without shutting down normally. When CAS restarts after a crash, it can load a fairly current copy of the ticket objects which will satisfy the 99.9% of the users who did not login in the last minutes before the crash.

The next step should be obvious. Can we turn "last few minutes" into "last few seconds". You could create a full checkpoint of all the tickets every few seconds, but now the overhead becomes significant. So go back to ticketRegistry.xml and set the parameters to call the "timerDriven" function every 10 seconds, but set the "checkpointInterval" parameter on the CushyTicketRegistry object to only create a new checkpoint file every 300 seconds. Now Cushy creates the checkpoint file, and then the next 29 times it is called by the timer it generates an "incremental" file containing only the changes since the checkpoint was written. Incremental files are cumulative, so there is only one file, not 29 separate files. If CAS crashes and restarts, Cushy reads the last checkpoint, then applies the changes in the last incremental, and now it has all the tickets up to the last 10 seconds before the crash. That satisfies 99.99% of the users and it is probably a good place to quit.

What about disaster recovery? The checkpoint and incremental files are ordinary sequential binary files on disk. When Cushy writes a new file it creates a temporary name and then swaps it for the old file, so other programs authorized to access the directory can safely open or copy the files while CAS is running. Feel free to write a shell script or Pearl or Python program to use SFTP or any other program or protocol to back up the data offsite or to the cloud.

Some people use JPATicketRegistry and store a copy of the tickets in a database to accomplish the same single server restart capability that Cushy provides. If you are happy with that solution, stick with it. Cushy doesn't require the database, it doesn't require JPA, and it may be easier to work with.

Before you configure a cluster, remember that today a server is typically a virtual machine that is not bound to any particular physical hardware. Ten years ago moving a service to a backup machine involved manual work that took time. Today there is VM infrastructure and automated monitoring and control tools. A failed server can be migrated and restarted automatically or with a few commands. If you can get the CAS server restarted fast enough that almost nobody notices, then you have solved the problem that clustering was originally designed to solve without adding a second running node.

CushyClusterConfiguration

If you use the JPATicketRegistry, then you configure CAS to know about the database in which tickets are stored. None of the nodes knows about the cluster as a whole. The "cluster" is simply one or more CAS servers configured with the same database.

If you use Ehcache or one of the other object replication "cache" technologies, then there is typically some automatic discovery mechanism based on multicast messages. That would be a good solution if you have only the one production CAS cluster, but it becomes harder to configure if you have separate Test and Development clusters (unless they are completely isolated by being run as non-routed virtual machines on a single host).

The alternative is to configure each node in each cluster separately, but that is hard to maintain. You want to test and validate a single CAS WAR artifact. Production Services wants to churn out identical server VMs with minimal differences. Where do you keep and introduce the node specific configuration? How do you make sure that changes are correctly implemented on every machine.

CushyClusterConfiguration provides an alternative approach to cluster configuration, and while it was originally designed for CushyTicketRegistry it also works for Ehcache. Instead of defining the point of view of each individual machine, the administrator defines all of the CAS servers in all of the clusters in the organization. Production, Functional Test, Load Test, Integration Test, down to the developers desktop or laptop "Sandbox" machines.

CushyClusterConfiguration is a Spring Bean that is specified in the CAS Spring XML. It only has a function during initialization. It reads in the complete set of clusters, uses DNS (or the hosts file) to obtain information about each CAS machine referenced in the configuration, it uses Java to determine the IP addresses assigned to the current machine, and then it tries to match one of the configured machines to the current computer. When it finds a match, then that configuration defines this CAS, and the other machines in the same cluster definition can be used to manually configure Ehcache or CushyTicketRegistry.

CushyClusterConfiguration exports the information it has gathered and the decisions it has made by defining a number of properties that can be referenced using the "Spring EL" language in the configuration of properties and constructor arguments for other Beans. This obviously includes the TicketRegistry, but the ticketSuffix property can also be used to define a node specific value at the end of the unique ticketids generated by beans configured by the uniqueIdGenerators.xml file.

There is a separate page to explain the design and syntax of CushyClusterConfiguration.

Front End or CushyFrontEndFilter

If the Front End can be programmed to understand CAS protocol, to locate the ticketid, to extract the node identifying suffix from the ticketid, and to route requests to the CAS server that generated the ticket, then CAS does not have to wait for each Service Ticket ID to be replicated around the cluster. This is much simpler and more efficient, and the Cushy design started by assuming that everyone would see that this is an obviously better idea.

Unfortunately, it became clear that people in authority frequently had a narrow view of what the Front End should do, and that was frequently limited to the set of things the vendor preprogrammed into the device. Furthermore, there was some reluctance to depend on the correct functioning of something new no matter how simple it might be.

So with another couple of day's worth of programming (much spent understanding the multithreaded support in the latest Apache HttpClient code), CushyFrontEndFilter was created. The idea here was to code in Java the exact same function that was better performed by an iRule in the BIG_IP F5 device, so that someone would be able to run all the Cushy programs even if he was not allowed to change his own F5.

CushyTicketRegistry and a CAS Cluster

The names of each checkpoint and incremental files are created from the unique node names each server in the cluster, so they can all coexist in the same disk directory. The simplest Cushy communication option is "SharedDisk". When this is chosen, Cushy expects that the other nodes are writing their full backup and incremental files to the same disk directory it is using. If Cushy receives a request that the Front End should have sent to another node, then Cushy assumes some node or network failure has occurred, loads the other node's tickets into memory from its last checkpoint and incremental file in the shared directory, and then processes the request on behalf of the other node.

...