...
After that the design of Cushy almost writes itself. It is clear that this is an entirely different approach to the TicketRegistry. It is so simple it has to work. It will definitely be useful to smaller CAS installations and may scale up. It is not a lot of code, and might be finished in two weeks (it took four). Cushy was written simply because it was possible, and it turned out better than expected.
The Standalone Server
For a single CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in memory Java table keyed by the ticket id string. Suppose you change the name of the Java class in the Spring ticketRegistry.xml file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry code, so everything works the same as before until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when the application restarts and users have to login again. Cushy detects the shutdown and using a single Java writeObject statement it saves all the ticket objects in the Registry to a file on disk (called the "checkpoint" file). When CAS restarts, Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown. No user even notices that CAS restarted unless they tried to access CAS during the restart.
...
Before you configure a cluster, remember that today a server is typically a virtual machine that is not bound to any particular physical hardware. Ten years ago moving a service to a backup machine involved manual work that took time. Today there is VM infrastructure and automated monitoring and control tools. A failed server can be migrated and restarted automatically or with a few commands. If you can get the CAS server restarted fast enough that almost nobody notices, then you have solved the problem that clustering was originally designed to solve without adding a second running node.
The Cushy Cluster
Cushy can be configured node by node, but Yale Production Services did not want to manage individual machine specific files. So Cushy adds a cluster configuration class. Actually, you configure every CAS cluster you have in your enterprise (desktop sandbox, development, test, stress test, production, ...). When CAS starts up the configuration class figures out which of the configured clusters includes this server machine. It generates parameters that configure the CushyTicketRegistry. It also generates and feeds a server specific "ticket ID suffix" string to the Spring XML that configures ticket ID generation.
...
Of course, these node to node communication calls are all configured into CAS at the same place (using Spring MVC) and in the same way that the /serviceValidate and /samlValidate processing is configured.
Are You Prepared?
Everything that can go wrong will go wrong. We plan for hardware and software failure, network failure, and disaster recovery. To do this we need to know how things will fail and how they will recover from each type of problem.
...
Cushy is simple enough that anyone can figure out how it works and, therefore, the consequences of any type of failure. So now you really can plan for each type of failure and design the system to do the best thing. Cushy was specifically written for the people who feel more comfortable knowing exactly how stuff works.
Why another Clustering Mechanism?
One solution is to share all the ticket objects and their associated components in database tables using JPA. JPA is the standard Java mechanism for mapping objects to tables. It is an enormously powerful tool for ordinary Web applications. It is a possible solution, but CAS doesn't have a database problem:
...
- The other CAS cluster mechanisms are designed so the CAS servers all share a common pool of tickets. The problem then is to get a copy of a new or modified ticket to all the other servers before they receive a request that requires it. A Cushy cluster is really a connected group of standalone CAS servers. The user may initially login to a randomly chosen server, but once she logs on the Front End routes all requests for that user to that particular server.
- The other CAS cluster mechanisms try to replicate individual tickets. Cushy replicates the ticket registry as a whole, either as a full backup file or as cumulative incremental changes.
- Other mechanisms are driven by individual addTicket or deleteTicket operations. Cushy notes these operations, but it goes to the network on a regular timer driven basis and if an operation fails it retries after the next timer interval.
- For other mechanisms you configure a highly available database or multicast addresses, timeouts, and recovery parameters. Cushy uses HTTP, and you already have that from the Web server CAS is already running in. The only configuration is the URLs of the machines in the cluster.
- A lot of CAS users configure JPA just so they can reboot CAS without losing the tickets. Cushy does that without the database or cluster.
- Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiency.
Basic Principles
- CAS is very important, but it is also small and cheap to run.
- Emphasize simplicity over efficiency as long as the cost remains trivial.
- The Front End gets the request first and it can be told what to do to keep the rest of the work simple. Let it do its job.
- Hardware failure doesn't have to be completely transparent. We can allow one or two users to get a bad message if everything works for the other 99.9% of the users. Trying to do better than this is the source of most 100% system failures.
Ticket Chains (and Test Cases)
A TGT represents a logged on user. It is called a Ticket Granting Ticket because it is used to create Service and Proxy tickets. It has no parent and stands alone.
...
This becomes an outline for various cluster node failure tests. Whenever one ticket points to a parent there is a model where the ticket pointed to was created on a node that failed and the new ticket has to be created on the backup server acting on behalf of that node. So you want to test the creation and validation of a Service Ticket on node B when the TGT was created on node A, or the creation of a PGT on node B when the TGT was created on node A, and so on.
Front End Programming
CAS Ticket IDs have four sections:
...
- After receiving a Service Ticket ID, an application opens its own HTTPS session to CAS, presents the ticket id in a "validate" request. If the id is valid CAS passes back the Netid, and in certain requests can pass back additional attributes. The suffix on the ticket= parameter identifies the CAS server that created the ticket and has it in memory without requiring any high speed replication.
- When a middleware server like a Portal has obtained a CAS Proxy Granting Ticket, it requests CAS to issue a Service Ticket by making a /proxy call. Since the middleware is not a browser, it does not have a Cookie to hold the PGT. So it passes it explicitly in the pgt= parameter.
- After a user logs in, CAS creates a Login TGT that points to the Netid and attributes and writes the ticket id of the TGT to the browser as a Cookie. The Cookie is scoped to the URL of the CAS application as seen from the browser point of view. At Yale this is "https://secure.its.yale.edu/cas" and so whenever the browser sees a subsequent URL that begins with this string, it appends the CASTGC Cookie with the TGT ID. CAS uses this to find the TGT object and knows that the user has already logged in. This rule sends a browser back to the CAS node the user is logged into.
- If the first three tests fail, this request is not associated with an existing logged in user. CAS has a bug/feature that it depends on Spring Web Flow and stores data during login in Web Flow storage which in turn depends on the HTTPSession object maintained by the Web Server (Tomcat, JBoss, ...). You can cluster JBoss or Tomcat servers to share HTTPSession objects over the network, but it is simpler if you program the Front End so that if the user responds in a reasonable amount of time, the login form with the userid and password is send back to the Web Server that wrote the form it to the browser in response to the browser's original HTTP GET. This is called a "sticky session" and the F5 does it automatically if you just check a box. You don't need to write code.
- Otherwise, if this is a brand new request to login to CAS or if the CAS Server selected by one of the previous steps has failed and is not responding to the Front End, then send the request to any available CAS server.
What Cushy Does at Failure
It is not necessary to explain how Cushy runs normally. It is based on DefaultTicketRegistry. It stores the tickets in a table in memory. If you have a cluster, each node in the cluster operates as if it was a standalone server and depends on the Front End to route requests to the node that can handle them.
...
Again, Cushy 2.0 may address this problem.
CAS Cluster
In this document a CAS "cluster" is just a bunch of CAS server instances that are configured to know about each other. The term "cluster" does not imply that the Web servers are clustered in the sense that they share Session objects (JBoss "clustering"). Nor does it depend on any other type of communication between machines. In fact, a CAS cluster could be created from a CAS running under Tomcat on Windows and one running under JBoss on Linux.
...
Internally, Cushy configuration takes a list of URLs and generates a cluster definition with three pieces of data for each cluster member: a nodename like "vmfoodevapp01" (the first element of the DNS name with dashes removed), the URL, and the ticket suffix that identifies that node (the F5 prefers the ticket suffix to the an MD5 hash of the IP address of the VM).
Configuration
In CAS the TicketRegisty is configured using the WEB-INF/spring-configuration/ticketRegistry.xml file.
...
Then add a second timer driven operation to the end of the file to call the "timerDriven" method of the CushyTicketRegistry object on a regular basis (say once every 10 seconds) to trigger writing the checkpoint and incremental files.
The Cluster
We prefer a single "cas.war" artifact that works everywhere. It has to work on standalone or clustered environments, in a desktop sandbox with or without virtual machines, but also in official DEV (development), TEST, and PROD (production) servers.
...
In any given cluster, the hostname of both machines is identical except for a suffix that is either the three characters "-01" or "-02". So by finding the current HOSTNAME it can say that if this machine has "-01" in its name, the other machine in the cluster is "-02", or the reverse.
Configuration By File
You can define the CushyClusterConfiguration bean with or without a "clusterDefinition" property. If you provide the property, it is a List of Lists of Strings:
...
In practice you will have a sandbox you created and some machine room VMs that were professionally configured and do not have strange or unexpected IP addresses, and you can configure all the hostnames in a configuration file and Cushy will select the right cluster and configure itself the way you expect.
Autoconfigure
At Yale the names of DEV, TEST, and PROD machines follow a predictable pattern, and CAS clusters have only two machines. So production services asked that CAS automatically configure itself based on those conventions. If you have similar conventions and any Java coding expertise you can modify the autoconfiguration logic at the end of CushyClusterConfiguration Java source.
...
At Yale, the two servers in any cluster have DNS names that ends in "-01" or "-02". Therefore, Cushy autoconfigure gets the HOSTNAME of the current machine, looks for a "-01" or "-02" in the name, and when it matches creates a cluster with the current machine and one additional machine with the same name but substituting "-01" for "-02" or the reverse.
Standalone
If no configured cluster matches the current machine IP addresses and the machine does not autoconfigure (because the HOSTNAME does not have "-01" or "-02"), then Cushy configures a single standalone server with no cluster.
Even without a cluster, Cushy still checkpoints the ticket cache to disk and restores the tickets across a reboot. So it provides a useful function in a single machine configuration that is otherwise only available with JPA and a database.
You Can Configure Manually
Although CushyClusterConfiguration makes most configuration problems simple and automatic, if it does the wrong thing and you don't want to change the code you can ignore it entirely. As will be shown in the next section, there are three properties, a string and two Properties tables) that are input to the CusyTicketRegistry bean. The whole purpose of CushyClusterConfiguration is to generate a value for these three parameters. If you don't like it, you can use Spring to generate static values for these parameters and you don't even have to use the clusterConfiguration bean.
Other Parameters
Typically in the ticketRegistry.xml Spring configuration file you configure CushyClusterConfiguration as a bean with id="clusterConfiguration" first, and then configure the usual id="ticketRegistry" using CusyTicketRegistry. The clusterConfiguration bean exports some properties that are used (through Spring EL) to configure the Registry bean.
...
- p:sharedDisk="true" - disables HTTP communication for JUnit Tests and when the work directory is on a shared disk.
- p:disableJITDeserialization="true" - disables an optimization that only reads tickets from a checkpoint or incremental file the first time the tickets are actually needed. The only reason for using this parameter is during testing so that the number of tickets read from the file appears in the log immediately after the file is generated.
- p:excludeSTFromFiles="true" - this is plausibly an option you should use. It prevents Service Tickets from being written to the checkpoint or incremental files. This makes incremental files smaller because it is then not necessary to keep the growing list of ST IDs for all the Service Tickets that were deleted probably before anyone ever really cared about them.
- p:useThread="true" - use a thread to read the checkpoint file from another CAS node. If not set, the file is read in line and this may slow down the processing of a new checkpoint across all the nodes.
How Often?
"Quartz" is the standard Java library for timer driven events. There are various ways to use Quartz, including annotations in modern containers, but JASIG CAS uses a Spring Bean interface to Quartz where parameters are specified in XML. All the standard JASIG TicketRegistry configurations have contained a Spring Bean configuration that drives the RegistryCleaner to run and delete expired tickets every so often. CushyTicketRegistry requires a second Quartz timer configured in the same file to call a method that replicates tickets. The interval configured in the Quartz part of the XML sets a base timer that determines the frequency of the incremental updates (typically every 5-15 seconds). A second parameter to the CushyTicketRegistry class sets a much longer period between full checkpoints of all the tickets in the registry (typically every 5-10 minutes).
...
Nodes notify each other of a full checkpoint. Incrementals occur so frequently that it would be inefficient to send messages around. A node picks up the other incrementals from the other nodes each time it generates its own incremental.
Special Situations
Cushy stores tickets in an in-memory table. It writes tickets to a disk file with a single writeObject Java statement. It transfers files from machine to machine using an HTTPS GET. So far, everything seems to be rather simple. Cushy started that way, but then it became clear that there were a small number of optimizations that really needed to be made even if they added a slight amount of complexity to the code.
Notify
Once every 5-15 minutes a node generates a new full checkpoint file. It also generates a new dummy ServiceTicketId that acts as the password that other nodes will present to request the files over HTTPS. It then does a "Notify" operation. It generates a HTTPS GET to the /cas/cluster/notify URL on every other CAS node in the cluster. This request is routed by Spring MVC to the CacheNotifyController class provided by the Cushy package. A node also does a Notify immediately after it reboots to inform the other nodes that it is back up and to provide them with the password needed to communicate until the next checkpoint.
...
In a SharedDisk situation (see below) there is no HTTP and therefore no /cluster/notify call. Instead, the timerDriven routine checks the Last Modified date on the other node's checkpoint file. When it changes, it performs a subset of the full processNotify operations to reset flags and mark the other server healthy.
Just In Time Deserialization
In the first cut, each checkpoint and incremental file was turned into objects immediately after it was read. Then it became clear that Cushy was using lots of processing time to create objects that just got discarded a few minutes later without anyone actually using them, and that just used memory and left junk for Garbage Collection to clean up. So although it adds just a bit of extra complexity, Cushy now waits for an request for a ticket in the file before opening the file and extracting the objects.
...
The only time Cushy will start to stress Garbage Collection is if the other node is actually up and able to talk to the other nodes in the cluster, but the Front End is not able to talk to it. Then the front end is sending its requests to the other nodes (so objects are being loaded into memory) and the node is still routinely generating new checkpoint and incremental files. At this point we just have to assume that Garbage Collection can handle 20,000 objects every couple of minutes. In normal Java processing terms, that is not a big load.
SharedDisk
The SharedDisk parameter is typically specified in the ticketRegistry.xml Spring configuration file. It turns off the Cushy HTTP processing. There will be no Notify message, and therefore no HTTP fetching of the checkpoint or incremental file. There is no exchange of dummy ServiceTicketId for communication security because there is no communication. It is used in real SharedDisk situations and in Unit Test cases.
Since there is no notify, the timerDriven code that generates checkpoint and incremental files has to check the last modified timestamp on the checkpoint file of any other node. If the timestamp changes, then that triggers the subset of Notify processing that does not involve HTTP or file transfers (like the resetting of flags indicating possible node health).
Cold Start Quiet Period
When CAS starts up and finds no previous checkpoint file in its work directory, there are no tickets to restore. This is a Cold Start, and it may be associated with a change of CAS code from one release to another with possible changes to the Ticket object definitions. A cold start has to happen at one time and it has to restart all the servers the cluster. You do not want one server running on old code while another server runs on the new code. To give the operators time to make the change, after a cold start CAS enters the Cold Start Quiet Period which lasts for 10 minutes (built into the source). During this period it does not send or respond to HTTP requests from other nodes. That way the nodes cannot exchange mismatched object files.
Healthy
When CAS receives an HTTP GET I/O error attempting to contact or read data from another node, it marks that node as "unhealthy" It then waits for a Notify from the node, and then tries to read the new checkpoint file.
...
Note that Healthy deals with a failure of this server to connect to a node while Just In Time Deserialization is triggered when the Front End cannot get to the node and sends us a request that belongs to the other node. If a node really goes down, both things happen at roughly the same time. Otherwise, it is possible for just one type of communication to fail while the other still works.
Usage Pattern
Users start logging into CAS at the start of the business day. The number of TGTs begins to grow.
...
Translated to Cushy, the cost of the full checkpoint and the size of the checkpoint file grow over time along with the number of active tickets, and then the file shrinks over night. During any period of intense login activity the incremental file may be unusually large. If you had a long time between checkpoints, then around the daily minimum (8 AM) you could get an incremental file bigger than the checkpoint.
CAS Ticket Objects Need to be Fixed
CAS has some bugs. They are very, very unlikely to occur, but they are there. Cushy can't fix them because they are in the Ticket classes themselves.
ConcurrentModificationException
First, the login TGT object has some collections. One collection gets a new entry every time a Service Ticket is created and it is used for Single Sign Off. In CAS 4, a new collection is used to handle multiple factors of authentication. If two requests arrive at the same time to generate two Service Tickets on the same TGT, then one ST is created and is queued up by existing TicketRegistry implementations to be replicated to other nodes. Meanwhile the second Service Ticket is being created and is adding a new entry to the Single Sign Off collection in the TGT.
...
private synchronized void writeObject(ObjectOutputStream s) throws IOException { s.defaultWriteObject();}
Private Copy of the Login TGT
JPA handles the entire collection of tickets properly.
...
Cushy avoids this problem because the periodic checkpoint file captures all the tickets with all their relationships. Limited examples of this problem can occur around node failure, but for all the other TicketRegistry solutions (except JPA) this happens all the time to all the tickets during normal processing.
JUnit Testing
It is unusual for JUnit test cases to get their own documentation. Testing a cluster on a single machine without a Web server is complicated enough that the strategies require some documentation.
...