Content Comparison

• Recover tickets after reboot without JPA, or a separate server, or a cluster (works on a standalone server)
• Recover tickets after a crash, except for the last few seconds of activity that did not get to disk.
• No dependency on any large external librarylibraries. Pure Java using only the standard Java SE runtime and some Apache commons stuff.
• All source in one class. A Java programmer can read it and understand it.
• Can also be used to cluster CAS servers
• Cannot crash CAS ever, no matter what is wrong with the network or other servers.
• A completely different and simpler approach to the TicketRegistry. Easier to work with and extend.
• Probably less efficient than the uses more CPU and network I/O than other TicketRegistry solutions, but it has a constant predictable overhead you can measure and "price out"verify is trivial.

CAS is a Single SignOn solution. Internally the function of CAS is to create, update, and delete a set of objects it calls "Tickets" (a word borrowed from Kerberos). A Logon Ticket (TGT) object is created to hold the Netid when a user logs on to CAS. A partially random string is generated to be the login ticket-id and is sent back to the browser as a Cookie and is also used as a "key" to locate the logon ticket object in a table. Similarly, CAS creates Service Tickets (ST) to identity a user to an application that uses CAS authentication.

...

Four years ago Yale implemented a "High Availability" CAS cluster using JBoss Cache to replicate tickets. After that, the only CAS crashes were caused by failures of JBoss Cache. Red Hat failed to diagnose or fix the problem. As we tried to diagnose the problem ourselves we discovered both bugs and design problems in the structure of Ticket objects and the use of the TicketRegistry solutions that contributed to the failure. We considered replacing JBoss Cache with Ehcache, but there is a more fundamental problem here. It should not be possible for any failure of the data replication mechanism to crash all of the CAS servers at once. Another choice of cache might be more reliable, but it would suffer from the same fundamental structural problem.

All of the previous CAS cluster solutions create a common pool of tickets shared by all of the cluster members. They are designed and configured so that the Front End can distribute requests in a round-robin approach and any server can handle any request. However, once the Service Ticket is returned by one server, the request to validate the ST comes back in milliseconds. So JPA must write the ST to the database, and Ehcache must synchronously replicate the ST to all the other servers, before the ST ID is passed back to the browser. Synchronous replication was the option that exposed CAS to crashing if the replication system had problems, and it imposed a sever performance constraint that requires all the CAS servers to be connected by very high speed networking.

Disaster recovery and very high availability suggests that at least one CAS server should be kept at a distance independent of the machine room, its power supply and support systems. So there is tension between performance considerations to keep servers close and recovery considerations to keep things distant.

Ten years ago, when CAS was being designed, the Front End that distributed requests to members of the cluster was typically an ordinary computer running simple software. Today networks have become vastly more sophisticated, and Front End devices are specialized machines with powerful software. They are designed to detect and fend off Denial of Service Attacks. They improve application performance by offloading SSL/TLS processing. They can do "deep packet inspection" to understand the traffic passing through and route requests to the most appropriate server (called "Layer 5-7 Routing" because requests are routed based on higher level protocols rather than just IP address or TCP session). Although this new hardware is widely deployed, CAS clustering has not changed and has no explicit option to take advantage of it.

Front End devices know many protocols and a few common server conventions. For everything else they expose a simple programming language. While CAS performs a Single Sign On function, the logic is actually designed to create, read, update, and delete tickets. The ticketid is the center of each CAS operation. In different requests there are only three places to find the ticketid that defines this operation:

1. In the ticket= parameter at the end of the URL for validation requests.
2. In the pgt= parameter for a proxy request.
3. In the CASTGC Cookie for browser requests.

Programming the Front End to know that "/validate", "/serviceValidate", and two other strings in the URL path means that this is case 1, and "/proxy" means it is case 2, and everything else is case 3 is pretty simple.

Of course, finding the ticket is not helpful unless you use a feature that has always been part of CAS configuration but previously was previouslty not particularly useful. Each server can put a specific identifier on the end of every ticketid it creates. This is the "suffix" of the ticket in the configuration parameters, but typically it has been left as the default string "-CAS". If the suffix is configured meaningfully, and if it is set to a value the Front End can use to identify the node, then combined with the previous three steps the Front End can be configured to route ticket requests preferentially to the node that created the ticket and therefore holds it in memory without depending first on cluster replication.

Of course, tickets still have to be replicated for recovery purposes, but that means that tickets can be replicated in seconds instead of milliseconds, and they can be queued and replicated periodically instead of synchronously (while the request waits). This makes the clustering mechanism much easier and more reliable.

Of course, the Front End is owned by the Networking staff, and they are not always responsive to the needs of the CAS administrator. Although it is obviously more efficient to program the Front End, the CushyFrontEndFilter can be added to the Servlet configuration of the CAS server to do in Java the same thing the Front End should be doing, at least until your network administrators adopt a more enlightened point of view.

"Cushy" stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale". This summarizes what it is and how it works.

For objects to be replicated from one node to another, programs use the Java writeObject statement to "Serialize" the object to a stream of bytes that can be transmitted over the network and then restored in the receiving JVM. Ehcache and the other ticket replication systems operate on individual tickets. However, writeObject can operate just as well on the entire contents of the TicketRegistry. This is very simple to code, it is guaranteed to work, but it might not be efficient enough to use. Still, once you have the idea the code starts to write itself.

Start with the DefaultTicketRegistry source that CAS uses to hold tickets in memory on a single CAS standalone server. Then add the writeObject statement (surrounded by the code to open and close the file) to create a checkpoint copy of all the tickets, and a corresponding readObject and surrounding code to restore the tickets to memory. The first thought was to do the writeObject to a network socket, because that was what all the other TicketRegistry implementations were doing. Then it became clear that it was simpler, and more generally useful, and a safer design, if the data was first written to a local disk file. The disk file could then optionally be transmitted over the network in a completely independent operation. Going first to disk created code that was useful for both standalone and clustered CAS servers, and it guaranteed that the network operations were completely separated from the Ticket objects and therefore the basic CAS function.

The first benchmarks turned out to be even better than had been expected, and that justified further work on the system.

CushyTicketRegistry and the Standalone Server

For a single CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in-memory Java table keyed by the ticket id string. Suppose you change the name of the Java class in the Spring ticketRegistry.xml file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry source code, so everything works the same as it did before, until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when CAS restarts and users all have to login again. Cushy detects the shutdown and using a single Java writeObject statement it saves all the ticket objects in the Registry to a file on disk (called the "checkpoint" file). When CAS restarts, Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown. No user even notices that CAS restarted unless they tried to access CAS during the restart.

The number of tickets CAS holds grows during the day and shrinks over night. At Yale there are fewer than 20,000 ticket objects in CAS memory, and Cushy can write all those tickets to disk in less than a second generating a file around 3 megabytes in size. Other numbers of tickets scale proportionately (you can run a JUnit test and generate your own numbers). This is such a small amount of overhead that Cushy can be proactive.

So to take the next logical step, start with the previous ticketRegistry.xml configuration and duplicate the XML elements that currently call a function in the RegistryCleaner every few minutes. In the new copy of the XML elements, call the "timerDriven" function in the (Cushy)ticketRegistry bean every few minutes. Now Cushy will not wait for shutdown but will back up the ticket objects regularly just in case the CAS machine crashes without shutting down normally. When CAS restarts after a crash, it can load a fairly current copy of the ticket objects which will satisfy the 99.9% of the users who did not login in the last minutes before the crash.

The next step should be obvious. Can we turn "last few minutes" into "last few seconds". You could create a full checkpoint of all the tickets every few seconds, but now the overhead becomes significant. So go back to ticketRegistry.xml and set the parameters to call the "timerDriven" function every 10 seconds, but set the "checkpointInterval" parameter on the CushyTicketRegistry object to only create a new checkpoint file every 300 seconds. Now Cushy creates the checkpoint file, and then the next 29 times it is called by the timer it generates an "incremental" file containing only the changes since the checkpoint was written. Incremental files are cumulative, so there is only one file, not 29 separate files. If CAS crashes and restarts, Cushy reads the last checkpoint, then applies the changes in the last incremental, and now it has all the tickets up to the last 10 seconds before the crash. That satisfies 99.99% of the users and it is probably a good place to quit.

What about disaster recovery? The checkpoint and incremental files are ordinary sequential binary files on disk. When Cushy writes a new file it creates a temporary name, fills the file with new data, closes it, and then swaps the new for the old file, so other programs authorized to access the directory can safely open or copy the files while CAS is running. Feel free to write a shell script or Pearl or Python program to use SFTP or any other program or protocol to back up the data offsite or to the cloud.

Some people use JPATicketRegistry and store a copy of the tickets in a database to accomplish the same single server restart capability that Cushy provides. If you are happy with that solution, stick with it. Cushy doesn't require the database, it doesn't require JPA, and it may be easier to work with.

Before you configure a cluster, remember that today a server is typically a virtual machine that is not bound to any particular physical hardware. Ten years ago moving a service to a backup machine involved manual work that took time. Today there is VM infrastructure and automated monitoring and control tools. A failed server can be migrated and restarted automatically or with a few commands. If you can get the CAS server restarted fast enough that almost nobody notices, then you have solved the problem that clustering was originally designed to solve without adding a second running node.

You may still want a cluster.

CushyClusterConfiguration

If you use the JPATicketRegistry, then you configure CAS to know about the database in which tickets are stored. None of the nodes knows about the cluster as a whole. The "cluster" is simply one or more CAS servers all configured to backup tickets into the same database.

If you use Ehcache or one of the other object replication "cache" technologies, then there is typically an option to use an automatic node discovery mechanism based on multicast messages. That would be a good solution if you have only the one production CAS cluster, but it becomes harder to configure if you have separate Test and Development clusters that have to have their own multicast configuration.

It seems to be more reliable to configure each node to know the name and URL of all the other machines in the same cluster. However, a node specific configuration file on each machine is difficult to maintain and install. You do not want to change the CAS WAR file when you distribute it to each machine, and Production Services wants to churn out identical server VMs with minimal differences.

CushyClusterConfiguration (CCC) provides an alternative approach to cluster configuration, and while it was originally designed for CushyTicketRegistry it also works for Ehcache. Instead of defining the point of view of each individual machine, the administrator defines all of the CAS servers in all of the clusters in the organization. Production, Functional Test, Load Test, Integration Test, down to the developers desktop or laptop "Sandbox" machines.

CCC is a Spring Bean that is specified in the CAS Spring XML. It only has a function during initialization. It reads in the complete set of clusters, uses DNS (or the hosts file) to obtain information about each CAS machine referenced in the configuration, it uses Java to determine the IP addresses assigned to the current machine, and then it tries to match one of the configured machines to the current computer. When it finds a match, then that configuration defines this CAS, and the other machines in the same cluster definition can be used to manually configure Ehcache or CushyTicketRegistry.

CCC exports the information it has gathered and the decisions it has made by defining a number of properties that can be referenced using the "Spring EL" language in the configuration of properties and constructor arguments for other Beans. This obviously includes the TicketRegistry, but the ticketSuffix property can also be used to define a node specific value at the end of the unique ticketids generated by beans configured by the uniqueIdGenerators.xml file.

There is a separate page to explain the design and syntax of CCC.

Front End or CushyFrontEndFilter

If the Front End can be programmed to understand CAS protocol, to locate the ticketid, to extract the node identifying suffix from the ticketid, and to route requests to the CAS server that generated the ticket, then CAS does not have to wait for each Service Ticket ID to be replicated around the cluster. This is much simpler and more efficient, and the Cushy design started by assuming that everyone would see that this is an obviously better idea.

Unfortunately, it became clear that people in authority frequently had a narrow view of what the Front End should do, and that was frequently limited to the set of things the vendor pre-programmed into the device. Furthermore, there was some reluctance to depend on the correct functioning of something new no matter how simple it might be.

So with another couple of day's more programming (much spent understanding the multithreaded SSL session pooling support in the latest Apache HttpClient code), CushyFrontEndFilter was created. The idea here was to code in Java the exact same function that was better performed by an iRule in the BIG_IP F5 device, so that someone would be able to run all the Cushy programs even if he was not allowed to change his own F5.

CushyTicketRegistry and a CAS Cluster

Picking back up where we left off from the Standalone Server discussion, the names of each checkpoint and incremental files are created from the unique node names each server in the cluster, so they can all coexist in the same disk directory. The simplest Cushy communication option is "SharedDisk". When this is chosen, Cushy expects that the other nodes are writing their full backup and incremental files to the same disk directory it is using. If Cushy receives a request that the Front End should have sent to another node, then Cushy assumes some node or network failure has occurred, loads the other node's tickets into memory from its last checkpoint and incremental file in the shared directory, and then processes the request on behalf of the other node.

Of course you are free to implement SharedDisk with an actual file server or NAS, but technically Cushy doesn't know or care how the files got to the hard drive. So if you don't like real shared disk technology, you can write a shell script somewhere to wake up every 10 seconds copy the files between machines using SFTP or whatever file transfer mechanism you like to use. You could also put the 3 megabyte files on the Enterprise Service Bus if you prefer architecture to simplicity.

SharedDisk is not the preferred Cushy communication mechanism. Cushy is, after all, part of CAS where the obvious example of communication between computers is the Service Ticket validation request. Issue an HTTPS GET to /cas/serviceValidate with a ServiceTicket and get back a bunch of XML that describes the user. So with Cushy, one node can issue a HTTPS GET to /cas/cluster/getCheckpoint on another node and it gets back the current checkpoint file for that CAS server.

Obviously you need security for this important data. CAS security is based on short term securely generated Login and Service Tickets. So every time CAS generates a new checkpoint file it also generates a new "dummyServiceTicketId" that controls access to that checkpoint file and all the incrementals generated until there is a new checkpoint. So the full request is "/cas/cluster/getCheckpoint?ticket=..."  where the dummyServiceTicketId is appended to the end.

How do the other nodes get the dummyServiceTicketId securely? Here we borrow a trick from the CAS Proxy Callback. Each CAS node is a Web server with an SSL Certificate to prove its identity. So when a node generates a new checkpoint file, and a new dummyServiceTicketId, it issues an HTTPS GET to all the other configured CAS nodes using URL
/cas/cluster/notify?nodename=callernodename&ticket=(dummyServiceTicketId).

Thanks to https: this request will not transmit the parameters unless the server first proves its identity with its SSL Certificate. Then the request is sent encrypted so the dummyServiceTicketId is protected. Although this is a GET, there is no response. It is essentially a "restful" Web Service request that sends data as parameters.

Notify does three things:

1. It tells the other node there is a new checkpoint ready to pick up immediately
2. It securely provides the other node with the dummyServiceTicketId needed to read files for the next few minutes.
3. It is a general declaration that the node is up and healthy. When a node starts up it sends its first /cluster/notify to all nodes with the &reboot=yes parameter to announce that it is live again.

Notify is only done every few minutes when there is a new checkpoint. Incrementals are generated all the time, but they are not announced. Each server is free to poll the other servers periodically to fetch the most recent incremental with the /cas/cluster/getIncremental request (add the dummyServiceTicketId to prove you are authorized to read the data).

Since these node to node communication calls are modeled on existing CAS Service Ticket validation and Proxy Callback requests, they are configured into CAS in the same place (in the Spring MVC configuration, details provided below).

Note: Yes, this sort of thing can be done with GSSAPI, but after looking into configuring Certificates or adding Kerberos, it made sense to keep it simple and stick with the solutions that CAS was already using to solve the same sort of problems in other contexts.

Are You Prepared?

Everything that can go wrong will go wrong. We plan for hardware and software failure, network failure, and disaster recovery. To do this we need to know how things will fail and how they will recover from each type of problem.

JPA is pretty straight forward. CAS depends on a database. To plan for CAS availability, you have to plan for database availability. At this point you have not actually solved any problem, but you have redefined it from a CAS issue to a database issue. Of course there is now an additional box involved, and you now have to look at network failures between the CAS servers and the database. However, now the CAS programmers can dump the entire thing on the DBAs and maybe they will figure it out. Unfortunately, you are probably not their most important customer when it comes to planning recovery.

The other CAS clustering techniques (JBoss Cache, Ehcache, Memcached) are typically regarded as magic off the shelf software products that take care of all your problems automatically and you don't have to worry about them. Again you haven't actually solved the problem, but now you really have transformed it into something you will never understand and so you just have to cross your fingers and hope those guys know what they are doing.

Even it you do not understand Java programming, CushyTicketRegistry performs a sequence of steps described here that you can understand. It writes a file on disk, and from that point on everything is file transfer. You can use the built-in Web support, or replace it with something else. From that point on every type of node failure or network failure produces predictable behavior. Since the file transfer is being retried periodically, every type of hardware recovery also produces predictable results. This is something you can understand and take into consideration when you plan out the scenarios.

Why another Clustering Mechanism?

You can use JPA, but CAS doesn't really have a database problem.

• CAS tickets all timeout after a number of hours. They have no need for long term persistence.
• There are no meaningful SQL operations in CAS. Nobody will generate reports based on tickets.
• CAS has no transactional structure or need for a conventional commit operation.

JPA also weaves its own generated code into the methods exposed by the objects it manages. This causes the application (CAS) to fail in unpredictable and unavoidable ways if the database goes down or if network access to the database is interrupted.

There are a number of non-database central object server technologies available. There are no existing CAS TicketRegistry implementations for any of them, and the central server remains a problem.

JBoss Cache has proven unreliable, and it is terribly complex to configure with multicast addresses and complex network timeout and other parameters.

Ehcache appears to be the most commonly used CAS replication technology. It is fairly simple to configure, and it uses RMI calls to transmit tickets, a built in Java technology that is about as simple as Cushy HTTP. It can store tickets on local disk. It is the obvious alternative to CushyTicketRegistry and deserves special consideration.

Ehcache Compared to CushyTicketRegistry

CushyClusterConfiguration will configure either EhcacheTicketRegistry or CushyTicketRegistry, so it is certainly no easier to configure one or the other.

Although the default configuration of Ehcache uses synchronous replication for Service Tickets, if you program the Front End (or add the CushyFrontEndFilter) to a CAS using Ehcache in the same way described for CushyTicketRegistry, then ST validation requests will go to the CAS server that created the ST, so you can use the same lazy asynchronous replication for Service Tickets that normally Ehcache is configured to use for Logon Tickets (TGTs).

So the main difference between the two is that every 10 seconds or so Ehcache replicates all the tickets that have changed in the last 10 seconds, while Cushy transmits a file with all of the ticket changes since the last full checkpoint. Then every few minutes it generates a full checkpoint. So Ehcache transmits a lot less data. However, the cost of transmitting the extra data is so low that this may not matter if Cushy provides extra function.

Ehcache is a closed system that operates inside the CAS servers and exposes no external features. Cushy generates checkpoint and incremental files that are regular files on disk that can be accessed using any standard commands, scripts, or utilities. This provides new disaster recovery options.

Ehcache is designed to be a "cache". That is, it is designed to be a high speed, in memory or local disk, copy of some data that has a persistent copy off on some server. That is why it has a lot of configuration for "LRU" and object eviction, because it assumes that lost objects are reloaded from persistent storage. You can use it as a replicated in memory table, but you have to understand if you read the documentation that that is not its original design. Cushy is specifically designed to be a CAS TicketRegistry. That is the only thing it does, and it is very carefully designed to do that job correctly.

Cushy models its design on two 40 year old concepts. A common strategy for backing disks up to tape was to do a full backup of all the files once a week, and then during the week to do an incremental backup of the files changed since the last backup. The term "checkpoint" derives from a disk file into which an application saved all its important data periodically so it could restore that data an pick up where it left off after a system crash. These strategies work because they are too simple to fail. More sophisticated algorithms may accomplish the same result with less processing and I/O, but the more complex the logic the more vulnerable you become if the software, or hardware, or network failure occurs in a way that the complex sophisticated software did not anticipate.

Ehcache is a large library of complex code designed to merge changes to shared data across multiple hosts. Cushy is a single source file of pure Java written to be easily understood.

Basic Principles

1. CAS is very important, but it is also small and cheap to run.
2. Emphasize simplicity over efficiency as long as the cost remains trivial.
3. The Front End gets the request first and it can be told what to do to keep the rest of the work simple. Let it do its job.
4. Hardware failure doesn't have to be completely transparent. We can allow one or two users to get a bad message if everything works for the other 99.9% of the users. Trying to do better than this is the source of most 100% system failures.

Ticket Chains (and Test Cases)

A TGT represents a logged on user. It is called a Ticket Granting Ticket because it is used to create Service and Proxy tickets. It has no parent and stands alone.

When a user requests it, CAS uses the TGT to create a Service Ticket. The ST points to the TGT that created it, so when the application validates the ST id string, CAS can follow the chain from the ST to the TGT to get the Netid and attributes to return to the application. Then the ST is discarded.

However, when a middleware application like a Portal supports CAS Proxy protocol, the CAS Business Logic layer trades an ST (pointing to a TGT) in and turns it into a second type of TGT (the Proxy Granting Ticket or PGT). The term "PGT" exists only in documents like this. Internally CAS just creates a second TGT that points to the login TGT.

If the Proxy application accesses a backend application, it calls the /proxy service passing the TGT ID and gets back a Service Ticket ID. That ST points to the PGT that points to the TGT from which CAS can find the Netid.

So when you are thinking about Ticket Registries, or when you are designing JUnit test cases, there are four basic arrangements to consider:

1. a TGT
2. a ST pointing to a TGT
3. a PGT pointing to a TGT
4. a ST pointing to a PGT pointing to a TGT

This becomes an outline for various cluster node failure tests. Whenever one ticket points to a parent there is a model where the ticket pointed to was created on a node that failed and the new ticket has to be created on the backup server acting on behalf of that node. So you want to test the creation and validation of a Service Ticket on node B when the TGT was created on node A, or the creation of a PGT on node B when the TGT was created on node A, and so on.

Front End Programming

Any cluster of Web Servers requires some sort of Front End device to screen and forward network traffic. Ten years ago this was a simple computer that normally assigned traffic to servers on a round robin basis. Today the primary function of many Front Ends is to protect the servers from Denial of Service attacks, attempts to brute force passwords, and other security problems. To do this, the device understands many common network protocols so it can do "deep packet" inspection. HTTP is probably the simplest of the protocols. A Front End will examine the URL, remove certain headers regarded as dangerous, and add headers of its own. It can select a specific server from the pool based on data in the request, although this is most commonly used to maintain "sessions" between a particular client and server.

Users at Yale know that CAS is "https://secure.its.yale.edu/cas". In reality, DNS resolves secure.its.yale.edu to 130.132.35.49 and that is a Virtual IP address (a VIP) on the BIG-IP F5. The VIP requires configuration, because the F5 has to hold the SSL Certificate for "secure.its.yale.edu" and manage the SSL protocol.

Yale decided to make it appear that other security applications appear to run on the secure.its.yale.edu machine, even though each application has its own pool of VMs. So the F5 has to examine the URL to determine if it begins with "/cas" and therefore goes to the pool of CAS VMs, or if it begins with /idp and therefore goes to the Shibboleth pool. The F5 has to inspect and generate HTTP Headers if the real client IP address is passed on to a Web Server for processing.

This means that if CAS is going to use X.509 User Certificates as a non-interactive form of authentication, then all the configuration that would in a standalone server be managed by the X509 optional component of CAS has to be configured in the F5. This is required by SSL protocol, it is not CAS specific. There has to be a special list of "Trusted" Certificate Authorities from which User Certificates will be accepted. The browser has to be told that certificates are required, permitted, or not allowed. The signature in the submitted Certificate has to be validated against the Trusted CA list. The Certificate has to be ASN.1 decoded, and then the DN and/or one or more subjectAltNames has to be extracted, and they have to be turned into HTTP headers that can be forwarded to the application. The F5 has most of this programming built in, although the last step of creating headers has to be manually coded. By comparison, routing requests based on CAS ticketids is simple.

Routing requests to particular servers based on the content of request line and the headers is part of what generic Front End devices (not just the F5) call "Layer 4-7 routing". The internet routes messages between computers using Layer 3 routing (IP) but Front End devices select the last hop to the specific VM based on data and and understanding of the higher level protocols. In the previous example the F5 obtained the User X.509 Certificate from the browser and decoded it. The full "DN" name in the certificate provides not only the users identity but also the organizational unit to which he is assigned and the geographical location of his home campus or office. An F5 running on one campus of a large statewide university might use this information to route a CAS login request to a server on the specific home campus where the user has his specific local account and data.

In an eCommerce application, an F5 may be programmed to route the user's first request to a server chosen at random, but then to send all subsequent requests back to the same server. This can be done based on the client IP address or, for Java servers, based on the value of a parameter named "JSESSIONID" commonly used  by Java servers to manage sessions. Of course, Java servers also have complex clustering technologies to exchange session information between requests, but it is better to avoid that problem all together.

First, however, we need to understand the format of CAS ticketids because that is where the routing information comes from:

type - num - random - suffix

where type is "TGT" or "ST", num is a ticket sequence number, random is a large random string like "dmKAsulC6kggRBLyKgVnLcGfyDhNc5DdGKT", and the suffix at the end is configured in the uniqueIdGenerators.xml file.

A typical XML configuration for a particular type of ticket (when you use Cushy) looks like this:

<bean id="ticketGrantingTicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator">
<constructor-arg index="0" type="int" value="50" />
<constructor-arg  index="1"  value="#{clusterConfiguration.getTicketSuffix()}" />
</bean>

The suffix value, which is the index="1" argument to the Java object constructor, is obtained using a Spring "EL" expression to be the TicketSuffix property of the bean named clusterConfiguration. This is the CushyClusterConfiguration object that scans the configured cluster definitions to determine which cluster the server is running in and what name and IP address it uses.  By directly feeding the output of clusterConfiguration into the input of the Ticket ID Generator, this approach makes configuration simple and ensures that all the machines come up configured properly. There is special logic in Cushy for an F5 which, for some reason, likes to identify hosts by the MD5 hash of the character representation of their IP address.

Every CAS request except the initial login comes with one or more tickets located in different places in the request. There is a sequence of tests and you stop at the first match:

1. If the Path part of the URL is a validate request (/cas/validate, /cas/serviceValidate, /cas/proxyValidate, or /cas/samlValidate) then look at the ticket= parameter in the query string part of the URL
2. Otherwise, if the Path part of the URL is a /cas/proxy request, then look at the pgt= parameter in the query string.
3. Otherwise, if the request has a CASTGC cookie, then look at the cookie value.
4. Otherwise, use the built in support if the request has a JSESSIONID.
5. Otherwise, or if the node selected by 1-4 is down, choose any CAS node from the pool.

That is the code, now here is the explanation:

1. After receiving a Service Ticket ID from the browser, an application opens its own HTTPS session to CAS, presents the ticket id in a "validate" request. If the id is valid CAS passes back the Netid, and in certain requests can pass back additional attributes. This request is best handled by the server that issued the Service Ticket.
2. When a middleware server like a Portal has obtained a CAS Proxy Granting Ticket, it requests CAS to issue a Service Ticket by opening its own HTTPS connection to CAS to make a /proxy call. Since the middleware is not a browser, it does not have a Cookie to hold the PGT. So it passes that ticketid explicitly in the pgt= parameter. This request is best handled by the server that created the Proxy Granting Ticket.
3. After a user logs in, CAS creates a Login TGT that points to the Netid and attributes and writes the ticket id of the TGT to the browser as a Cookie. The Cookie is sent back from the browser in any request to "https://secure.its.yale.edu/cas". After initial login, all requests with cookies are requests to issue a Service Ticket for a new application using the existing CAS login. This is best handled by the server that created the TGT.
4. If there is no existing ticket, then the user is logging into CAS. This may be the GET that returns the login form, or the POST that submits the Userid and Password. Vanilla CAS code works only if the POST goes back to the same server than handled the GET. This is the only part of CAS that actually has an HttpSession.
5. Otherwise, if there is no JSESSIONID then this is the initial GET for the login form. Assign it to any server.

Except for Case 4 during login, neither the browser, JBoss, CAS, or the F5 is maintaining a "session" as that term is commonly used, where requests from the same client always go to the same server and the server maintains an HttpSession object. Since the entire CAS function is based on creating and updating Ticket objects, each CAS request except the initial browser logon references a specific Ticket ID. By storing in the Ticket ID a field that easily identifies to the F5 the CAS server that created and owns the Ticket, CAS protocol now provides a relatively simple algorithm for routing requests to the best server. It is vastly simpler than other protocols that the F5 has built in because of their wide use.

The F5 understands HTTP requests and already has both expressions and logic to locate "the Path part of the URL", "the ticket= parameter in the query string", and "the CASTGC Cookie value". All that has to be coded is the comparison of these predefined items to test values, and an expression to extract the string that follows the third "-" character in a given ticket value.

CAS does not require the F5 to create any new table. The pool of servers associated with /cas is already part of the F5 configuration. The logic depends on the CAS protocol, which has been updated only three times since CAS was created, rather than the characteristics of any particular CAS release.

Although HTTP is a "stateless" protocol, an SSL connection is frequently optimized to be a longer term thing that keeps a session alive between requests. The SSL connects the browser or application to the Front End, and there is probably a separate SSL connection from the Front End to the CAS VM. A common option for Front Ends is to notice any long running SSL connection and use it to route requests to the same backend VM node. You must be sure that you do not select this option with Cushy and CAS. For Service Ticket validation requests to work, the routing decision has to be made separately for each request because different tickets have to be routed to different CAS VMs even though they came from the same application.

If you cannot convince your network administrators to do the programming in the Front End where it belongs, you can get the same result slightly less efficiently using the CushyFrontEndFilter.  Just add it as a Servlet Filter to the WEB-INF/web.xml file and it examines incoming requests and executes the same logic described above, just coded in Java instead of F5 iRule syntax.

What Cushy Does at Failure

It is not necessary to explain how Cushy runs normally. It is based on DefaultTicketRegistry. It stores the tickets in a table in memory. If you have a cluster, each node in the cluster operates as if it was a standalone server and depends on the Front End to route requests to the node that can handle them.

Separately from the CAS function, Cushy periodically writes some files to a directory on disk. They are ordinary files. They are protected with ordinary operating system security.

In a cluster, the files can be written to a shared disk, or they can be copied to a shared location or from node to node by an independent program that has access to the directories. Or, Cushy will replicate the files itself using HTTPS GET requests.

A failure is detected when a request is routed by the Front End to a node other than the node that created the ticket.

Because CAS is a relatively small application that can easily run on a single machine, a "cluster" can be configured in either of two ways:

• A Primary server gets all the requests until it fails. Then a Backup "warm spare" server gets requests. If the Primary comes back up relatively quickly, then Cushy will work best if Front End resumes routing all request to the Primary as soon as it becomes available again.
• Users are assigned to login to a CAS Server on a round-robin or load balanced basis. After a user logs in, the suffix on the login, proxy, or service tickets in the URL or headers of an HTTP request route the request to that server. 

Each CAS server in the cluster has a shadow object representing the TicketRegistry of each of the other nodes. In normal operation the CAS nodes exchange checkpoint and incremental files but they do not restore objects from those files to memory. This is called "Tickets On Request". The first time a request arrives for a ticket owned by another node, the getTicket request restores tickets into memory from the files for that node.

However, every new ticket Cushy creates belongs to the node that created it. During a node failure, the new Service Tickets or Proxy Granting Tickets created for users logged into the failed node are created by and belong to the backup node. They each get a ticket ID that has the suffix of the backup node. They live forever in the Ticket Registry of the backup node. They just happen to be associated with and point to a TGT in the shadow registry on the backup node associated with the failed login node.

So while the failed node is down, and even after it comes back up again, requests associated with tickets created by the backup node are routed to the backup node by the Front End. However, after the failed node comes back new requests for new tickets associated with the login TGT will go back to being processed by the original while that might improve reliability somewhat it would not solve the fundamental structural problems.

Having been burned by software so complicated that the configuration files were almost impossible to understand, Cushy was developed to accomplish the same thing in a way so simple it could not possibly fail.

The existing CAS TicketRegistry solutions must be configured to replicate tickets to the other nodes and to wait for this activity to complete, so that any node can validate a Service Ticket that was just generated a few milliseconds ago. Waiting for the replication to complete is what makes CAS vulnerable to a crash if the replication begins but never completes. Synchronous ticket replication is a standard service provided by JBoss Cache and Ehcache, but is it the right way to solve the Service Ticket validation problem? A few minutes spent crunching the math suggested there was a better way.

It is easier and more efficient to send the request to the node that already has the ticket and can process it rather than struggling to get the ticket to every other node in advance of the next request.

In the current TicketRegistry implementations, any request in a cluster to create a Service Ticket must replicate the service ticket to at least one other computer (the database server in JPA, one or more nodes using Ehcache or any other ticket replication mechanism) before the Service Ticket ID is returned to the browser. This ensures that the Service Ticket can be validated by any node to which the application's validation request is directed. After validation, there is a second network transaction to delete the ticket. So every ST involves two backend synchronous operations.

However, it has always been part of CAS that every ticketid has a suffix that, at least on paper, can contain the node name of the CAS server that created the ticket. Using this feature in practice requires some node configuration methodology. Once this is done, then any validate request (for example, any call to /cas/serviceValidate contains in the query string part of the URL a ticket= parameter, and the end of the value of that parameter designates the node that created the ticket. Today you can program most modern network front end devices to extract this information from the request and route the validate request to the node that created the ticket and is guaranteed to have it in memory. If you cannot program your front end device, or if you cannot convince your network administrators to do the work for you, then CushyFrontEndFilter accomplishes the same thing by scanning requests as they arrive at a CAS server and forwarding requests like validation to the server that created the ticket. If you have two servers and requests are randomly assigned to them, then 50% of the time the request goes to the right server and there is no network transaction, and 50% of the time the request has to be forwarded by the Filter to the other server, which then validates the ST and deletes it returning the response message. So with the Filter you expect, on average, one network transaction half the time instead of, with current JPA or Cache technology, two network transactions every time. When the number of nodes in the cluster is more than 2, the Filter works even better.

CushyFrontEndFilter works with Ehcache or CushyTicketRegistry. When added to Ehcache you can change the cache configuration so that the Service Ticket cache does not use synchronous replication, or even better you can turn off replication entirely for the Service Ticket cache because every 10 seconds a Service Ticket is either used and discarded or else times out, so it makes no sense to replicate them at all if the front end or filter routes requests properly.

However, once you come up with the idea of using front end routing to avoid the synchronous ticket replication (which was the source of crashes in JBoss Cache at Yale), then some new more radical changes to TicketRegistry become possible. In addition to the various validate request, you can route the /proxy request to the node that owns the Proxy Granting Ticket, and you can route new Service Ticket requests to the node that issued the Ticket Granting Ticket (based on the suffix of the CASTGC cookie). Now a basic principle of all the existing ticket registry designs is no longer necessary. CAS Ticket objects do not have to be stored in what appears to be a common shared pool. Tickets can be segregated into separate collections based on the identity of the node that created and "owns" the ticket.

"Cushy" stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale". This summarizes what it is and how it works.

For objects to be replicated from one node to another, libraries use the Java writeObject statement to "Serialize" the object to a stream of bytes that can be transmitted over the network and then restored in the receiving JVM. Ehcache and JBoss Cache use writeObject on individual tickets (although it turns out they also end up serializing copies of all the other objects the ticket points to, including the TGT when attempting to replicate a ST). However, writeObject can operate just as well on the entire TicketRegistry. Making a "checkpoint" copy of the entire collection of tickets to disk (at shutdown for example) and then restoring this collection (after a restart) is very simple to code. Since Java does all the work, it is guaranteed to behave correctly. It is a useful additional function. However, you can be more aggressive in the use of this approach, and that suggests the design of an entirely different type of TicketRegistry.

Start with the DefaultTicketRegistry source that CAS uses to hold tickets in memory on a single CAS standalone server. Then add the writeObject statement (surrounded by the code to open and close the file) to create a checkpoint copy of all the tickets, and a corresponding readObject and surrounding code to restore the tickets to memory. The first thought was to do the writeObject to a network socket, because that was what all the other TicketRegistry implementations were doing. Then it became clear that it was simpler, and more generally useful, and a safer design, if the data was first written to a local disk file. The disk file could then optionally be transmitted over the network in a completely independent operation. Going first to disk created code that was useful for both standalone and clustered CAS servers, and it guaranteed that the network operations were completely separated from the Ticket objects and therefore the basic CAS function.

The first benchmarks turned out to be even better than had been expected, and that justified further work on the system.

CushyTicketRegistry and the Standalone Server

For a single CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in-memory Java table keyed by the ticket id string. Suppose you change the name of the Java class in the Spring ticketRegistry.xml file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry source code, so everything works the same as it did before, until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when CAS restarts and users all have to login again. Cushy detects the shutdown and using a single Java writeObject statement it saves all the ticket objects in the Registry to a file on disk (called the "checkpoint" file). When CAS restarts, Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown. No user even notices that CAS restarted unless they tried to access CAS during the restart.

The number of tickets CAS holds grows during the day and shrinks over night. At Yale there are fewer than 20,000 ticket objects in CAS memory, and Cushy can write all those tickets to disk in less than a second generating a file around 3 megabytes in size. Other numbers of tickets scale proportionately (you can run a JUnit test and generate your own numbers). This is such a small amount of overhead that Cushy can be proactive.

CAS is a very important application, but on modern hardware it is awfully small and cheap to run. Since it was first developed there have been at least 5 generations of new chip technology that now run what was never a big application to begin with.

So to take the next logical step, start with the previous ticketRegistry.xml configuration and duplicate the XML elements that currently call a function in the RegistryCleaner every few minutes. In the new copy of the XML elements, call the "timerDriven" function in the (Cushy)ticketRegistry bean every few minutes. Now Cushy will not wait for shutdown but will back up the ticket objects regularly just in case the CAS machine crashes without shutting down normally. When CAS restarts after a crash, it can load a fairly current copy of the ticket objects which will satisfy the 99.9% of the users who did not login in the last minutes before the crash.

The next step should be obvious. Can we turn "last few minutes" into "last few seconds". You could create a full checkpoint of all the tickets every few seconds, but now the overhead becomes significant. So go back to ticketRegistry.xml and set the parameters to call the "timerDriven" function every 10 seconds, but set the "checkpointInterval" parameter on the CushyTicketRegistry object to only create a new checkpoint file every 300 seconds. Now Cushy creates the checkpoint file, and then the next 29 times it is called by the timer it generates an "incremental" file containing only the changes since the checkpoint was written. Incremental files are cumulative, so there is only one file, not 29 separate files. If CAS crashes and restarts, Cushy reads the last checkpoint, then applies the changes in the last incremental, and now it has all the tickets up to the last 10 seconds before the crash. That satisfies 99.99% of the users and it is probably a good place to quit.

What about disaster recovery? The checkpoint and incremental files are ordinary sequential binary files on disk. When Cushy writes a new file it creates a temporary name, fills the file with new data, closes it, and then swaps the new for the old file, so other programs authorized to access the directory can safely open or copy the files while CAS is running. Feel free to write a shell script or Pearl or Python program to use SFTP or any other program or protocol to back up the data offsite or to the cloud.

Some people use JPATicketRegistry and store a copy of the tickets in a database to accomplish the same single server restart capability that Cushy provides. If you are happy with that solution, stick with it. Cushy doesn't require the database, it doesn't require JPA, and it may be easier to work with.

Before you configure a cluster, remember that today a server is typically a virtual machine that is not bound to any particular physical hardware. Ten years ago moving a service to a backup machine involved manual work that took time. Today there is VM infrastructure and automated monitoring and control tools. A failed server can be migrated and restarted automatically or with a few commands. If you can get the CAS server restarted fast enough that almost nobody notices, then you have solved the problem that clustering was originally designed to solve without adding a second running node.

You may still want a cluster.

CushyClusterConfiguration

If you use the JPATicketRegistry, then you configure CAS to know about the database in which tickets are stored. None of the nodes knows about the cluster as a whole. The "cluster" is simply one or more CAS servers all configured to backup tickets into the same database.

If you use Ehcache or one of the other object replication "cache" technologies, then there is typically an option to use an automatic node discovery mechanism based on multicast messages. That would be a good solution if you have only the one production CAS cluster, but it becomes harder to configure if you have separate Test and Development clusters that have to have their own multicast configuration.

It seems to be more reliable to configure each node to know the name and URL of all the other machines in the same cluster. However, a node specific configuration file on each machine is difficult to maintain and install. You do not want to change the CAS WAR file when you distribute it to each machine, and Production Services wants to churn out identical server VMs with minimal differences.

In the 1980's before the internet, 500 universities worldwide were connected by BITNET. The technology required a specific local configuration file for each campus, but maintaining 500 different configurations was impossible. So they created a single global file that defined the entire network from no specific point of vew, and a utility program that, given the identity of a campus somewhere in the network, could translate that global file to the configuration data that campus needed to install to participate in the network. CushyClusterConfiguration does the same thing for your global definition of many CAS clusters.

CushyClusterConfiguration (CCC) provides an alternative approach to cluster configuration, and while it was originally designed for CushyTicketRegistry it also works for Ehcache. Instead of defining the point of view of each individual machine, the administrator defines all of the CAS servers in all of the clusters in the organization. Production, Functional Test, Load Test, Integration Test, down to the developers desktop or laptop "Sandbox" machines.

CCC is a Spring Bean that is specified in the CAS Spring XML. It only has a function during initialization. It reads in the complete set of clusters, uses DNS (or the hosts file) to obtain information about each CAS machine referenced in the configuration, it uses Java to determine the IP addresses assigned to the current machine, and then it tries to match one of the configured machines to the current computer. When it finds a match, then that configuration defines this CAS, and the other machines in the same cluster definition can be used to manually configure Ehcache or CushyTicketRegistry.

CCC exports the information it has gathered and the decisions it has made by defining a number of properties that can be referenced using the "Spring EL" language in the configuration of properties and constructor arguments for other Beans. This obviously includes the TicketRegistry, but the ticketSuffix property can also be used to define a node specific value at the end of the unique ticketids generated by beans configured by the uniqueIdGenerators.xml file.

There is a separate page to explain the design and syntax of CCC.

Front End or CushyFrontEndFilter

Front End devices know many protocols and a few common server conventions. For everything else they expose a simple programming language. The Filter contains the same logic written in Java.

We begin by assuming that the CAS cluster has been configured by CushyClusterConfiguration or its equivalent, and that one part of configuring the cluster was to create a unique ticket suffix for every node and feed that value to the beans configured in the uniqueIdGenerators.xml file.

After login, the other CAS requests all operate on tickets. They generate Service Tickets and Proxy Granting Tickets, validate tickets, and so on. The first step is to find the ticket that is important to this request. There are only three places to find the ticketid that defines an operation:

1. In the ticket= parameter at the end of the URL for validation requests.
2. In the pgt= parameter for a proxy request.
3. In the CASTGC Cookie for browser requests.

A validate request is identified by having a particular "servletPath" value ("/validate", "/serviceValidate, "proxyValidate", "/samlValidate"). The Proxy request has a different path ("/proxy"). Service Ticket create requests come from a browser that has a CASTGC cookie. If none of the servletPath values match and there is no cookie, then this request is not related to a particular ticket and can be handled by any CAS server.

If you program this into the Front End, then the request goes directly to the right server without any additional overhead. With only the Filter, a request goes to some randomly chosen CAS Server which may have to forward the request to another server, forward back the response, and handle failure if the preferred server goes down.

There is a separate page to describe Front End programming for CAS.

CushyTicketRegistry and a CAS Cluster

Picking back up where we left off from the Standalone Server discussion, the names of each checkpoint and incremental files are created from the unique node names each server in the cluster, so they can all coexist in the same disk directory. The simplest Cushy communication option is "SharedDisk". When this is chosen, Cushy expects that the other nodes are writing their full backup and incremental files to the same disk directory it is using. If Cushy receives a request that the Front End should have sent to another node, then Cushy assumes some node or network failure has occurred, loads the other node's tickets into memory from its last checkpoint and incremental file in the shared directory, and then processes the request on behalf of the other node.

Of course you are free to implement SharedDisk with an actual file server or NAS, but technically Cushy doesn't know or care how the files got to the hard drive. So if you don't like real shared disk technology, you can write a shell script somewhere to wake up every 10 seconds copy the files between machines using SFTP or whatever file transfer mechanism you like to use. You could also put the 3 megabyte files on the Enterprise Service Bus if you prefer architecture to simplicity.

SharedDisk is not the preferred Cushy communication mechanism. Cushy is, after all, part of CAS where the obvious example of communication between computers is the Service Ticket validation request. Issue an HTTPS GET to /cas/serviceValidate with a ServiceTicket and get back a bunch of XML that describes the user. So with Cushy, one node can issue a HTTPS GET to /cas/cluster/getCheckpoint on another node and it gets back the current checkpoint file for that CAS server.

Obviously you need security for this important data. CAS security is based on short term securely generated Login and Service Tickets. So every time CAS generates a new checkpoint file it also generates a new "dummyServiceTicketId" that controls access to that checkpoint file and all the incrementals generated until there is a new checkpoint. So the full request is "/cas/cluster/getCheckpoint?ticket=..."  where the dummyServiceTicketId is appended to the end.

How do the other nodes get the dummyServiceTicketId securely? Here we borrow a trick from the CAS Proxy Callback. Each CAS node is a Web server with an SSL Certificate to prove its identity. So when a node generates a new checkpoint file, and a new dummyServiceTicketId, it issues an HTTPS GET to all the other configured CAS nodes using URL
/cas/cluster/notify?nodename=callernodename&ticket=(dummyServiceTicketId).

Thanks to https: this request will not transmit the parameters unless the server first proves its identity with its SSL Certificate. Then the request is sent encrypted so the dummyServiceTicketId is protected. Although this is a GET, there is no response. It is essentially a "restful" Web Service request that sends data as parameters.

Notify does three things:

1. It tells the other node there is a new checkpoint ready to pick up immediately
2. It securely provides the other node with the dummyServiceTicketId needed to read files for the next few minutes.
3. It is a general declaration that the node is up and healthy. When a node starts up it sends its first /cluster/notify to all nodes with the &reboot=yes parameter to announce that it is live again.

Notify is only done every few minutes when there is a new checkpoint. Incrementals are generated all the time, but they are not announced. Each server is free to poll the other servers periodically to fetch the most recent incremental with the /cas/cluster/getIncremental request (add the dummyServiceTicketId to prove you are authorized to read the data).

CAS is a high security application, but it always has been. The best way to avoid introducing a security problem is to model the design of each new feature on something CAS already does, and then just do it the same way.

Since these node to node communication calls are modeled on existing CAS Service Ticket validation and Proxy Callback requests, they are configured into CAS in the same place (in the Spring MVC configuration, details provided below).

Are You Prepared?

Everything that can go wrong will go wrong. We plan for hardware and software failure, network failure, and disaster recovery. To do this we need to know how things will fail and how they will recover from each type of problem.

JPA is pretty straight forward. CAS depends on a database. To plan for CAS availability, you have to plan for database availability. At this point you have not actually solved any problem, but you have redefined it from a CAS issue to a database issue. Of course there is now an additional box involved, and you now have to look at network failures between the CAS servers and the database. However, now the CAS programmers can dump the entire thing on the DBAs and maybe they will figure it out. Unfortunately, you are probably not their most important customer when it comes to planning recovery.

The other CAS clustering techniques (JBoss Cache, Ehcache, Memcached) are typically regarded as magic off the shelf software products that take care of all your problems automatically and you don't have to worry about them. Again you haven't actually solved the problem, but now you really have transformed it into something you will never understand and so you just have to cross your fingers and hope those guys know what they are doing.

Even it you do not understand Java programming, CushyTicketRegistry performs a sequence of steps described here that you can understand. It writes a file on disk, and from that point on everything is file transfer. You can use the built-in Web support, or replace it with something else. From that point on every type of node failure or network failure produces predictable behavior. Since the file transfer is being retried periodically, every type of hardware recovery also produces predictable results. This is something you can understand and take into consideration when you plan out the scenarios.

Why another Clustering Mechanism?

You can use JPA, but CAS doesn't really have a database problem.

• CAS tickets all timeout after a number of hours. They have no need for long term persistence.
• There are no meaningful SQL operations in CAS. Nobody will generate reports based on tickets.
• CAS has no transactional structure or need for a conventional commit operation.

JPA also weaves its own generated code into the methods exposed by the objects it manages. This causes the application (CAS) to fail in unpredictable and unavoidable ways if the database goes down or if network access to the database is interrupted.

There are a number of non-database central object server technologies available. There are no existing CAS TicketRegistry implementations for any of them, and the central server remains a problem.

JBoss Cache has proven unreliable, and it is terribly complex to configure with multicast addresses and complex network timeout and other parameters.

Ehcache appears to be the most commonly used CAS replication technology. It is fairly simple to configure, and it uses RMI calls to transmit tickets, a built in Java technology that is about as simple as Cushy HTTP. It can store tickets on local disk. It is the obvious alternative to CushyTicketRegistry and deserves special consideration.

Ehcache Compared to CushyTicketRegistry

CushyClusterConfiguration will configure either EhcacheTicketRegistry or CushyTicketRegistry, so it is certainly no easier to configure one or the other.

CushyFrontEndFilter works for both Ehcache and CushyTicketRegistry, so any benefits there can apply equally to both systems if you reconfigure Ehcache to exploit them.

With Front End support, every 10 seconds or so Ehcache replicates all the tickets that have changed in the last 10 seconds, while Cushy transmits a file with all of the ticket changes since the last full checkpoint. Then every few minutes Cushy generates a full checkpoint that Ehcache does not use. So Ehcache transmits a lot less data.

Ehcache uses RMI and does not seem to have any security, so it depends on the network Firewall and the good behavior of other computers in the machine room. Cushy encrypts data and verifies the identity of machines, so it cannot be attacked even from inside the Firewall.

Cushy generates regular files on disk that can be copied using any standard commands, scripts, or utilities. This provides new disaster recovery options.

Ehcache is designed to be a "cache". That is, it is designed to be a high speed, in memory copy of some data that has a persistent authoritative source on some server. That is why it has a lot of configuration for "LRU" and object eviction, because it assumes that lost objects are reloaded from persistent storage. You can use it as a replicated in memory table, but you have to understand if you read the documentation that that is not its original design. Cushy is specifically designed to be a CAS TicketRegistry.

Cushy models its design on two 40 year old concepts. A common strategy for backing disks up to tape was to do a full backup of all the files once a week, and then during the week to do an incremental backup of the files changed since the last backup. The term "checkpoint" derives from a disk file into which an application saved all its important data periodically so it could restore that data an pick up where it left off after a system crash. These strategies work because they are too simple to fail. More sophisticated algorithms may accomplish the same result with less processing and I/O, but the more complex the logic the more vulnerable you become if the software, or hardware, or network failure occurs in a way that the complex sophisticated software did not anticipate.

Ehcache is a large library of complex code designed to merge changes to shared data across multiple hosts. Cushy is a single source file of pure Java written to be easily understood.

Replicating the entire TicketRegistry instead of just replicating individual tickets is less efficient. The amount of overhead is predictable and you can verify that the extra overhead is trivial. However, remember this is simply the original Cushy 1.0 design which was written to prove a point and is aggressively "in your face" pushing the idea of "simplicity over efficiency". After we nail down all the loose ends, it is possible to add a bit of extra optimization to get arbitrarily close to Ehcache in terms of efficiency.

Ticket Chains (and Test Cases)

A TGT represents a logged on user. It is called a Ticket Granting Ticket because it is used to create Service and Proxy tickets. It has no parent and stands alone.

When a user requests it, CAS uses the TGT to create a Service Ticket. The ST points to the TGT that created it, so when the application validates the ST id string, CAS can follow the chain from the ST to the TGT to get the Netid and attributes to return to the application. Then the ST is discarded.

However, when a middleware application like a Portal supports CAS Proxy protocol, the CAS Business Logic layer trades an ST (pointing to a TGT) in and turns it into a second type of TGT (the Proxy Granting Ticket or PGT). The term "PGT" exists only in documents like this. Internally CAS just creates a second TGT that points to the login TGT.

If the Proxy application accesses a backend application, it calls the /proxy service passing the TGT ID and gets back a Service Ticket ID. That ST points to the PGT that points to the TGT from which CAS can find the Netid.

So when you are thinking about Ticket Registries, or when you are designing JUnit test cases, there are four basic arrangements to consider:

1. a TGT
2. a ST pointing to a TGT
3. a PGT pointing to a TGT
4. a ST pointing to a PGT pointing to a TGT

This becomes an outline for various cluster node failure tests. Whenever one ticket points to a parent there is a model where the ticket pointed to was created on a node that failed and the new ticket has to be created on the backup server acting on behalf of that node. So you want to test the creation and validation of a Service Ticket on node B when the TGT was created on node A, or the creation of a PGT on node B when the TGT was created on node A, and so on.

What Cushy Does at Failure

While other TicketRegistry solutions combine tickets from all the nodes, a Cushy cluster operates as a goup of standalone CAS servers. The Front End or the Filter routes requests to the server that can handle them. So when everything is running fine, the TicketRegistry that CAS uses is basically the same as the DefaultTicketRegistry module that works on standalone servers.

So the interesting things occur when one server goes down or when network connectivity is lost between the Front End and a node, or between one node and another.

If a node fails, or the Front End cannot get to it and thinks it has failed, then requests start to arrive at CAS nodes for tickets that they do not own and did not create. File sharing or replication gives every node a copy of the most recent checkpoint and incremental file from that node, but normally the strategy of "Tickets on Request" does not open or process the files until they are needed. So the first request restores all the tickets for the other node to memory under the Secondary TicketRegistry object created at initialization to represent the failed node.

Since the rule is that the other node "owns" its own tickets, you cannot make any permanent changes to the tickets in the Secondary Registry. These tickets will be passed back as needed to the CAS Business Logic layer, and it will make changes as part of its normal processing thinking that the changes it makes are meaningful. In reality, when the other node comes back it will reload its tickets from the point of failure and that will be the authoritative collection representing the state of those tickets. In practice this doesn't actually matter.

If CAS on this node creates a new Service Ticket or Proxy Granting Ticket related to a Login TGT created originally by the other node, then: The new Ticket belongs to the node that created it and that node identifier is added to the end of the ticket ID. So the new ST is owned by and is validated by this node even though the Login TGT used to create it comes from the Secondary Registry of the failed node.

Service Tickets are created and then in a few milliseconds they are deleted when the application validates them or they time out after a few seconds or minutes. They do not exist long enough to raise any issues.

Proxy Granting Tickets, however, can remain around for hours. So the one long term consequence of a failure is that the login TGT can be on one server, but a PGT can be on a different server that created it while the login server was temporarily unavailable. This requires some thought, but you should quickly realize that everything will work correctly today. In future CAS releases there will be an issue if a user adds additional credentials (factors of authentication) to an existing login after a PGT is created. Without the failure, the PGT sees the new credentials immediately. With current Cushy logic, the PGT on the backup server is bound to a point in time snapshot of the original TGT and will not see the additional credentials. Remember, this only occurs after a CAS failure. It only affects the users who got the Proxy ticket during the failure. It can be "corrected" if the end user logs out and then logs back into the middleware server.Cushy 2.0 will consider addressing this problem automaticallyThe PGT ends up with its own private copy of the TGT which is frozen in time at the moment the PGT was created. Remember, this is normal behavior for all existing TicketRegistry solutions and none of the other TicketRegistry options will ever "fix" this situation. At least Cushy is aware of the problem and with a few fixes to the Ticket classes Cushy 2.0 might be able to do better.

There is also an issue with Single Sign Out. If a user logs out during a failure of his login server, then a backup server processes the Single Log Out normally. Then when the login server is restored to operation, the Login TGT is restored from the checkpoint file into memory. Of course, no browser now has a Cookie pointing to that ticket, so it sits unused all day and then in the evening it times out and a second Single Sign Out process is triggered and all the applications that perviously previously were told the user logged out are not contacted a second time with the same logout information. It is almost unimaginable that any application would be written so badly it would care about this, but it should be mentioned.

While the login server is down, new Service Tickets can be issued, but they cannot be meaningfully added to the "services" table in the TGT that drives Single Sign Out. After the login server is restored, if the user logs out to CAS the only applications that will be notified of the logout will be applications that received their Service Tickets from the logon server. Cushy regards Single Sign Out as a "best effort" service and cannot at this time guarantee processing for ST's issued during a node or network failure.

Again, Cushy 2.0 may address this problem.

Cushy CAS Cluster

In this document a CAS "cluster" is just a bunch of CAS server instances that are configured to know about each other. The term "cluster" does not imply that the Web servers are clustered in the sense that they share Session objects (JBoss "clustering"). Nor does it depend on any other type of communication between machines. In fact, a Cushy CAS cluster could be created from a CAS running under Tomcat on Windows and one running under JBoss on Linux.

To the outside world, the cluster typically shares a common virtual URL simulated by the Front End device. At Yale, CAS is "https://secure.its.yale.edu/cas" to all the users and applications. The "secure.its.yale.edu" DNS name is associated with an IP address managed by the BIG-IP F5 device. It holds the certificate, terminates the SSL, then examines requests and based on programming called iRules it forwards requests to any of the configured CAS virtual machines.

Each virtual machine has a native DNS name and URL. It is these "native" URLs that define the cluster because each CAS VM has to use the native URL to talk to another CAS VM. At Yale those URLs follow a pattern of "https://vm-foodevapp-01.web.yale.internal:8443/cas". 

Internally, Cushy configuration takes a list of URLs and generates a cluster definition with three pieces of data for each cluster member: a nodename like "vmfoodevapp01" (the first element of the DNS name with dashes removed), the URL, and the ticket suffix that identifies that node (the F5 prefers the ticket suffix to the an MD5 hash of the IP address of the VM"services" table in the TGT of the machine that is down.When that machine comes back up it resumes controlling the old TGT of the logged in user, and when the user logs off the Single Sign Out processing will occur only for servers that that machine knows about, and will omit services to which the user connected while the server that owned the TGT was down. Cushy provides a "best effort" Single Sign Out experience, and Cushy 1.0 cannot do better than this.

There are a few types of network failure that work differently from node failure.

If one CAS node is unable to connect to another CAS node for a while, even though the other node is up, then it marks the other node as being "unhealthy" and waits patiently for the other node to send a /cluster/notify. The other node will send a Notify every time it generates a new Checkpoint, and when one of those Notify messages gets through then the two nodes will reestablish communication.

If the Front End is unable to get to a CAS Node, but the other server can get to it, then what happens next depends on whether the CushyFrontEndFilter is also installed. Having both the programmed Front End and also the Filter is a bit like suspenders and a belt, but if the Front End is doing its job then the Filter has nothing to do. However, in this particular case the Filter will see a request for a ticket owned by another node and will attempt to forward it to the node indicated in the request. If it succeeds then CAS has automatically routed traffic around the point of failure. However, remember that if the node actually goes down then there will be two connect timeout delays, one where the Front End determines the node is down and then a second where the Filter verifies that it is down.

Without the Filter then the current node receives a request for a ticket it does not own, loads tickets into its Secondary Registry for that node, and processes the request. What is different is that if the node is really up and the two nodes can connect, then this CAS node will continue to receive Notify requests and new checkpoint and incremental files from the other node even as it is also processing requests for that node sent to it by the Front End. Cushy is designed to handle this situation (because even in a normal failure the other node can come up just as you are in the middle of handling a request for it).

Configuration

In CAS the TicketRegisty is configured using the WEB-INF/spring-configuration/ticketRegistry.xml file.

...

Then add a second timer driven operation to the end of the file to call the "timerDriven" method of the CushyTicketRegistry object on a regular basis (say once every 10 seconds) to trigger writing the checkpoint and incremental files.

There is a separate page that describes CushyClusterConfiguration in detail.

You Can Configure Manually

Although CushyClusterConfiguration makes most configuration problems simple and automatic, if it does the wrong thing and you don't want to change the code you can ignore it entirely. As will be shown in the next section, there are three properties, a string and two Properties tables) that are input to the CusyTicketRegistry bean. The whole purpose of CushyClusterConfiguration is to generate a value for these three parameters. If you don't like it, you can use Spring to generate static values for these parameters and you don't even have to use the clusterConfiguration beanthe CushyTicketRegistry object on a regular basis (say once every 10 seconds) to trigger writing the checkpoint and incremental files.

There is a separate page that describes CushyClusterConfiguration in detail.

You Can Configure Manually

Since CushyClusterConfiguration only generates strings and Property tables that are used by CushyTicketRegistry, if you prefer you can generate those strings and tables manually in the CAS configuration file for each server.

Other Parameters

Typically in the ticketRegistry.xml Spring configuration file you configure CushyClusterConfiguration as a bean with id="clusterConfiguration" first, and then configure the usual id="ticketRegistry" using CusyTicketRegistry. The clusterConfiguration bean exports some properties that are used (through Spring EL) to configure the Registry bean.

...