...
Four years ago Yale implemented a "High Availability" CAS cluster using JBoss Cache to replicate tickets. After that, the only CAS crashes were caused by failures of JBoss Cache. Red Hat failed to diagnose or fix the problem. We considered replacing JBoss Cache with Ehcache, but there is a more fundamental problem here. It should not be possible for any failure of the data replication mechanism to crash all of the CAS servers at once. Another choice of cache might be more reliable, but it would suffer from the same fundamental structural problem.
Having been burned by software so complicated that the configuration files were almost impossible to understand, Cushy was developed to accomplish the same thing in a way so simple it could not possibly fail.
The existing CAS TicketRegistry solutions must be configured to replicate tickets to the other nodes and to wait for this activity to complete, so that any node can validate a Service Ticket that was just generated a few milliseconds ago. Waiting for the replication to complete is what makes CAS vulnerable, but it seemed like an obvious solution because Ehcache and JBoss Cache promised that it would work. Once it didn't work, it was obvious that someone would at least ask the question whether there was another way to do this. If there was, then the entire TicketRegistry strategy could be reconsidered.
It is easier and more efficient to send the request to the node that already has the ticket and can process it rather than struggling to get the ticket to every other node in advance of the next request.
It turns out that most of the modern network Front End devices that distribute requests to members of a cluster are smart enough today to program a relatively simple amount of CAS protocol and locate the ticketid for a request. If you activate the CAS feature that puts a node identifier on the end of the ticketid, then requests can be routed to the node that created the ticket. If you cannot get your network administrator to program your Front End device, then CushyFrontEndFilter does the same thing not as efficiently as it could be done in the network box, but more efficiently than the processing of requests using "naked" Ehcache without the Filter.
...
The number of tickets CAS holds grows during the day and shrinks over night. At Yale there are fewer than 20,000 ticket objects in CAS memory, and Cushy can write all those tickets to disk in less than a second generating a file around 3 megabytes in size. Other numbers of tickets scale proportionately (you can run a JUnit test and generate your own numbers). This is such a small amount of overhead that Cushy can be proactive.
CAS is a very important application, but on modern hardware it is awfully small and cheap to run. Since it was first developed there have been at least 5 cycles of new chip technology applied to what was never a big application to begin with.
So to take the next logical step, start with the previous ticketRegistry.xml configuration and duplicate the XML elements that currently call a function in the RegistryCleaner every few minutes. In the new copy of the XML elements, call the "timerDriven" function in the (Cushy)ticketRegistry bean every few minutes. Now Cushy will not wait for shutdown but will back up the ticket objects regularly just in case the CAS machine crashes without shutting down normally. When CAS restarts after a crash, it can load a fairly current copy of the ticket objects which will satisfy the 99.9% of the users who did not login in the last minutes before the crash.
...
Notify is only done every few minutes when there is a new checkpoint. Incrementals are generated all the time, but they are not announced. Each server is free to poll the other servers periodically to fetch the most recent incremental with the /cas/cluster/getIncremental request (add the dummyServiceTicketId to prove you are authorized to read the data).
CAS is a high security application, but it always has been. The best way to avoid introducing a security problem is to model the design of each new feature on something CAS already does, and then just do it the same way.
Since these node to node communication calls are modeled on existing CAS Service Ticket validation and Proxy Callback requests, they are configured into CAS in the same place (in the Spring MVC configuration, details provided below).
Note: Yes, this sort of thing can be done with GSSAPI, but after looking into configuring Certificates or adding Kerberos, it made sense to keep it simple and stick with the solutions that CAS was already using to solve the same sort of problems in other contexts.
Are You Prepared?
...
Are You Prepared?
Everything that can go wrong will go wrong. We plan for hardware and software failure, network failure, and disaster recovery. To do this we need to know how things will fail and how they will recover from each type of problem.
...
Replicating the entire TicketRegistry instead of just replicating individual tickets is less efficient. The amount of overhead is predictable and you can verify that the extra overhead is trivial. However, remember this is simply the original Cushy 1.0 design which was written to prove a point and is aggressively "in your face" pushing the idea of "simplicity over efficiency". After we nail down all the loose ends, it is possible to add a bit of extra optimization to get arbitrarily close to Ehcache in terms of efficiency.
Basic Principles
- CAS is very important, but it is also small and cheap to run.
- Emphasize simplicity over efficiency as long as the cost remains trivial.
- The Front End gets the request first and it can be told what to do to keep the rest of the work simple. Let it do its job.
- Hardware failure doesn't have to be completely transparent. We can allow one or two users to get a bad message if everything works for the other 99.9% of the users. Trying to do better than this is the source of most 100% system failures.
Ticket Chains (and Test Cases)
A TGT represents a logged on user. It is called a Ticket Granting Ticket because it is used to create Service and Proxy tickets. It has no parent and stands alone.
When a user requests it, CAS uses the TGT to create a Service Ticket. The ST points to the TGT that created it, so when the application validates the ST id string, CAS can follow the chain from the ST to the TGT to get the Netid and attributes to return to the application. Then the ST is discarded.
...
Ticket Chains (and Test Cases)
A TGT represents a logged on user. It is called a Ticket Granting Ticket because it is used to create Service and Proxy tickets. It has no parent and stands alone.
When a user requests it, CAS uses the TGT to create a Service Ticket. The ST points to the TGT that created it, so when the application validates the ST id string, CAS can follow the chain from the ST to the TGT to get the Netid and attributes to return to the application. Then the ST is discarded.
However, when a middleware application like a Portal supports CAS Proxy protocol, the CAS Business Logic layer trades an ST (pointing to a TGT) in and turns it into a second type of TGT (the Proxy Granting Ticket or PGT). The term "PGT" exists only in documents like this. Internally CAS just creates a second TGT that points to the login TGT.
...
Without the Filter then the current node receives a request for a ticket it does not own, loads tickets into its Secondary Registry for that node, and processes the request. What is different is that if the node is really up and the two nodes can connect, then this CAS node will continue to receive Notify requests and new checkpoint and incremental files from the other node even as it is also processing requests for that node sent to it by the Front End. Cushy is designed to handle this situation (because even in a normal failure the other node can come up just as you are in the middle of handling a request for it).
Cushy CAS Cluster
In this document a CAS "cluster" is just a bunch of CAS server instances that are configured to know about each other. The term "cluster" does not imply that the Web servers are clustered in the sense that they share Session objects (JBoss "clustering"). Nor does it depend on any other type of communication between machines. In fact, a Cushy CAS cluster could be created from a CAS running under Tomcat on Windows and one running under JBoss on Linux.
To the outside world, the cluster typically shares a common virtual URL simulated by the Front End device. At Yale, CAS is "https://secure.its.yale.edu/cas" to all the users and applications. The "secure.its.yale.edu" DNS name is associated with an IP address managed by the BIG-IP F5 device. It holds the certificate, terminates the SSL, then examines requests and based on programming called iRules it forwards requests to any of the configured CAS virtual machines.
Each virtual machine has a native DNS name and URL. It is these "native" URLs that define the cluster because each CAS VM has to use the native URL to talk to another CAS VM. At Yale those URLs follow a pattern of "https://vm-foodevapp-01.web.yale.internal:8443/cas".
Internally, Cushy configuration takes a list of URLs and generates a cluster definition with three pieces of data for each cluster member: a nodename like "vmfoodevapp01" (the first element of the DNS name with dashes removed), the URL, and the ticket suffix that identifies that node (the F5 prefers the ticket suffix to the an MD5 hash of the IP address of the VMprocesses the request. What is different is that if the node is really up and the two nodes can connect, then this CAS node will continue to receive Notify requests and new checkpoint and incremental files from the other node even as it is also processing requests for that node sent to it by the Front End. Cushy is designed to handle this situation (because even in a normal failure the other node can come up just as you are in the middle of handling a request for it).
Configuration
In CAS the TicketRegisty is configured using the WEB-INF/spring-configuration/ticketRegistry.xml file.
...