Content Comparison

• Recover tickets after reboot without JPA, or a separate server, or a cluster (even works on a standalone server)
• Recover tickets after a crash, except for the last few seconds of activity that did not get to disk.
• No dependency on any external library. Pure Java using only the standard Java SE runtime.
• All source in one class. A Java programmer can read it and understand it.
• Can also be used to cluster CAS servers, if the Front End is programmed to route requests with an understanding of CAS protocol.
• Cannot crash CAS ever, no matter what is wrong with the network or other servers.
• A completely different and simpler approach to the TicketRegistry. Easier to work with and extend.
• Probably less efficient than the other TicketRegistry solutions, but it has a constant predictable overhead you can measure and verify it is trivial"price out".

CAS is a Single SignOn solution. Internally the function of CAS is to create, update, and delete a set of objects it calls "Tickets" (a word borrowed from Kerberos). A Logon Ticket object is created to hold the Netid when a user logs on to CAS. A partially random string is generated to be the login ticket-id and is sent back to the browser as a Cookie and is also used as a "key" to locate the logon ticket object in a table. Similarly, CAS creates Service Tickets to identity a logged on user to an application that uses CAS authentication.

CAS stores its tickets in a customer administrator selectable component called a TicketRegistry. CAS provides one implementation of the TicketRegistry for single-server configurations, and at least four alternatives that can be used to share tickets among several CAS servers operating in a network cluster. This document describes a new implementation called CushyTicketRegistry that is simple, provides added function for the standalone server, and yet also operates in clustered configurations.

Four years ago Yale implemented a "High Availability" CAS cluster using JBoss Cache to replicate tickets. After that, the only CAS crashes were caused by failures of the ticket replication mechanism. Red Hat failed to diagnose or fix the problem. We considered replacing JBoss Cache, but there is a more fundamental problem here. It should not be possible for any failure of the data replication mechanism to crash CAS. One of the other mechanisms might fail less frequently, but we were reluctant to switch from one "big magic black box" library to another.

Obviously copying It will always be necessary to copy data to backup locations is necessary if the system is a backup location in order to recover from a crash. CAS, however, depends on data replication for its normal functioning. It was designed to expect The previous TicketRegistry alternatives had to copy data to all servers very, very fast because CAS made no assumptions about the intelligence of the network Front End to randomly distribute requests to any available CAS server. Since the next request may depend on data created by the previous requests, the current CAS design requires that data objects be copied to all the other servers in the cluster before the next request arrives to any Web server.

In the last decade network Front End devices have gotten smarter. They understand many of the common protocols of widely used servers. For example, they know about the JSESSIONID generated by most Java J2EE Web Servers. CAS protocol does not come built in to them, but they are programmable and it is fairly simple to program them to route requests intelligently.

Consider the popular BIG-IP F5. Like any other Front End device it will have to hold the SSL Certficate, manage an IP address that the outside world regards as the CAS server, read in the first line (GET or POST) and the headers. Then it executes some custom code specifically for the CAS IP address to decide what to do. An intelligent decision can be made with two items from the first line of the request: the "path" part of the URL (/cas/login, /cas/serviceValidate, etc.) and the parameters in the query string following the "?" in the URL (particularly the ticket= and pgt= parameters). We also need one Header value (the login CASTGC Cookie).

CAS has always had the ability to put a node name identifying the specific server at the end of each ticket ID string. This code makes configuration of the unique node name easier and required. Then depending on the value of the path in the URL, a CAS ticket can be located from the CASTGC cookie or the ticket= or pgt= parameters. From the ticket find the node that created and is responsible for managing that ticket, and pass the request to that node, unless that node is down. With this change, a CAS cluster can operate as a loosely associated set of standalone servers and data replication is only required to recover from server failuredevice. Today it is possible to use the capability of modern Front End equipment to simplify and relax the data replication requirements.

Since CAS has to use SSL, any Front End that distributes requests among CAS servers has to hold the SSL Certificate and manage the SSL protocol. This is more complicated than you may think, because if you want to accepts User Certificates installed in the browser (or in plug in devices based on User Certificate protocol), then it is the Front End and not CAS that has to manage the trusted Certificate Authorities, the signature validation, the ASN.1 decode, and the Credentials To Principal Resolution functions that CAS handles when it talks directly to browsers. Compared to that, intelligent routing of requests based on ticket ID suffix is fairly simple.

Although we think of CAS from the end user's point of view as a Single Sign On system with "logged in users", under the covers CAS processes HTTP Web requests to create, read, update, and delete Tickets. If you read the CAS protocol carefully, you realize that every CAS request (except the initial login) is a Ticket operation in which the primary piece of data is a Ticket ID string. It can be the value of the CASTGC cookie, or the ticket= parameter on a /cas/serviceValidate, or the pgt= parameter sent by a proxy. The ticket string is in different places, but it is always there.

CAS generates a new Ticket ID string in a specific class. This is important because CAS is not secure unless the Ticket ID is large enough and random enough that it cannot be guessed or brute forced. This code has the ability to put a "node name" identifier suffix on the end of each Ticket ID, although in many CAS installations nobody changes the default which is to end the ticket with "-CAS". However, if every CAS server generates a unique Ticket ID suffix value that the Front End can use, then programming the Front End to find the ticket in one of the three places it can be, extract the string following the third "-" character in the ID, and using that value to select a particular server is fairly simple.

Do not make the mistake of assuming this is a "CAS Session". It is a substitute for a session for the Logon CASTGC Cookie value, but an application validates Service Tickets as they come in from browsers, and although the application thinks it is talking to a single CAS URL (that actually points to the Front End) each request may go to a different specific server since each Service Ticket has its own specific issuing node.

Even if you don't use Cushy, automating the configuration of the Ticket ID suffix and adding intelligent routing to your Front End can vastly simplify the configuration of all the other TicketRegistry solutions. For other solutions, this is a simplification. For Cushy it is a requirement.

"Cushy" stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale". This summarizes what it is and how it works.

There are two big ideas.

1. Modern Front End devices (like the BIG-IP F5) can be programmed to understand CAS protocol and make smarter decisions about how to route requests.
2. If you clone the Collection that points to all the ticket objects, you can make a copy of the entire current state of the CAS server to a disk file with a single writeObject statement. That can be restored to memory with a single readObject statement.

...

1. All the complexity is handled automatically by Java!

After that the design of Cushy almost writes itself. It is clear that this is an entirely different approach to the TicketRegistry. It is so simple it has HAS to work. It will definitely . Whether it would be efficient enough to actually use in production was the question that could not be answered until the code was written and tested. It was clear that it would be useful to smaller CAS installations and may scale up. It is small CAS deployments, particularly single servers, even if it turned out to be unworkable for large institutions. It was not a lot of code, and might be finished in two weeks (it took four). Cushy was written simply because it was possible, and it turned out better than expected.

...

after the first few days there was enough working to run some test cases and benchmark the results. It turned out to be much better than expected. Still to understand the code, begin with the small size and work up.

The Standalone Server

For a single CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in-memory Java table keyed by the ticket id string. Suppose you change the name of the Java class in the Spring ticketRegistry.xml file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry source code, so everything works the same as it did before, until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when the application CAS restarts and users all have to login again. Cushy detects the shutdown and using a single Java writeObject statement it saves all the ticket objects in the Registry to a file on disk (called the "checkpoint" file). When CAS restarts, Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown (although a few tickets may have expired). No user even notices that CAS restarted unless they tried to access CAS during the restart.

The number of tickets CAS holds grows during the day and shrinks over night. At Yale there are fewer than 20,000 ticket objects in CAS memory, and Cushy can write all those tickets to disk in less than a second generating a file around 3 megabytes. Other numbers of tickets scale proportionately (you can run a JUnit test and generate your own numbers). This is such a small amount of overhead that Cushy can be proactive.

So to take the next logical step, start with the previous ticketRegistry.xml configuration and duplicate the statements in ticketRegistry.xml XML elements that currently call a function in the RegistryCleaner every few minutes. In the new statementscopy of the elements, call the "timerDriven" function in the (Cushy)ticketRegistry bean every few minutes. Now Cushy will not wait for shutdown but will back up the ticket objects regularly just in case the CAS machine crashes without shutting down normally. When CAS restarts after a crash, it can load a fairly current copy of the ticket objects which will satisfy the 99.9% of the users who did not login in the last minutes before the crash.

The next step should be obvious. Can we turn "last few minutes" into "last few seconds". You could create a full checkpoint of all the tickets every few seconds, but now the overhead becomes significant. So go back to ticketRegistry.xml and set the parameters to call the "timerDriven" function every 10 seconds, but set the "checkpointInterval" parameter on the CushyTicketRegistry object to only create a new checkpoint file every 300 seconds. Now Cushy creates the checkpoint file, and then the next 29 times it is called by the timer it generates an "incremental" file containing only the changes since the checkpoint was written. Incremental files are cumulative, so there is only one file, not 29 separate files. If CAS crashes and restarts, Cushy reads the last checkpoint, then applies the changes in the last incremental, and now it has all the tickets up to the last 10 seconds before the crash. That satisfies 99.99% of the users and it is probably a good place to quit.

...

Cushy can be configured node by node, but Yale Production Services did not want to manage individual machine specific files. So Cushy adds a cluster configuration class.   Actually, you configure every CAS cluster you have in your enterprise (desktop sandbox, development, test, stress test, production, ...). When CAS starts up the configuration class figures out which of the configured clusters includes this the current server machine. It generates parameters that configure the CushyTicketRegistry. It also generates and feeds a server specific "ticket ID suffix" string to the Spring XML that configures ticket ID generation.

...

SharedDisk is not the preferred Cushy communication mechanism. Cushy is, after all, part of CAS where the obvious example of communication between computers is the Service Ticket validation request. Issue an HTTPS GET to /cas/serviceValidate with a ServiceTicket and get back a bunch of XML that describes the user. Issue an So with Cushy, one node can issue a HTTPS GET to /cas/cluster/getCheckpoint and you get on another node and it gets back the current checkpoint file for that CAS server.

Obviously you need security for this important data. CAS security is based on short term securely generated Login and Service Tickets. So every time CAS generates a new checkpoint file it also generates a new "dummyServiceTicketId" that controls access to that checkpoint file and all the incrementals generated until there is a new checkpoint. So the full request is "/cas/cluster/getCheckpoint?ticket=... and add on "  where the dummyServiceTicketId is appended to the end.

How do the other nodes get the dummyServiceTicketId securely. ? Here we borrow a trick from the CAS Proxy Callback. Each CAS node is a Web server with an SSL Certificate to prove its identity. So when a node generates a new checkpoint file, and a new dummyServiceTicketId, it calls issues an HTTPS GET to all the other configured CAS nodes it its cluster at their using URL
/cas/cluster/notify?nodename=callernodename&ticket=(dummyServiceTicketId) address. The request won't go through unless the certificate verifies the server, the data is encrypted, and this message .

Thanks to https: this request will not transmit the parameters unless the server first proves its identity with its SSL Certificate. Then the request is sent encrypted so the dummyServiceTicketId is protected. Although this is a GET, there is no response. It is essentially a "restful" Web Service request that sends data.

Notify does three things:

1. It tells the other node there is a new checkpoint ready to pick up immediately
2. It securely provides the other node with the dummyServiceTicketId needed to read files for the next few minutes.
3. It is a general declaration that the node is up and healthy. When a node starts up it sends its first /cluster/notify to all nodes with the &reboot=yes parameter to announce that it is live again.

Incrementals are unimportant and are not worth announcing. However, after getting a Notify, other nodes will typically call back every 10 seconds (or whatever interval you configure) to read a new incremental using Notify is only done every few minutes when there is a new checkpoint. Incrementals are generated all the time, but they are not announced. Each server is free to poll the other servers periodically to fetch the most recent incremental with the /cas/cluster/getIncremental ?ticket=request (add the dummyServiceTicketId ) request.Of course, to prove you are authorized to read the data).

Since these node to node communication calls are all are modeled on existing CAS Service Ticket validation and Proxy Callback requests, they are configured into CAS at in the same place (using in the Spring MVC ) and in the same way that the /serviceValidate and /samlValidate processing is configuredconfiguration).

Note: Yes, this sort of thing can be done with GSSAPI, but after looking into configuring Certificates or adding Kerberos, it made sense to keep it simple and stick with the solutions that CAS was already using to solve the same sort of problems in other contexts.

...

JPA is pretty straight forward. CAS depends on a database. To plan for CAS availability, you have to plan for database availability. At this point you have not actually solved any problem, but you have redefined the problem it from a CAS issue to a database issue. Of course there is now an additional box involved, and you now have to look at network failures between the CAS servers and the database. However, now you the CAS programmers can dump the entire thing on the DBAs and maybe they will figure it out. Unfortunately, you are probably not their most important customer when it comes to planning recovery.

The other CAS clustering techniques (JBoss Cache, Ehcache, Memcached) are typically regarded as magic off the shelf software products that take care of all your problems automatically and you don't have to worry about them. Again you haven't actually solved the problem, but now you really have transformed it into something you will never understand and so you just have to cross your fingers and hope those guys know what they are doing.

...

In Cushy, the only connection between the CAS mainline function (the part of CAS that responds to Web requests and services users and applications) is that references to objects are occasionally copied from one in memory collection to another. This is an object copy operation that can't fail unless you run out of memory. Separate from that, on a timer driven basis collections of objects completely separate from the objects used by CAS request processing are periodically written to disk with a single Java writeObject statement. This is a disk I/O operation that can't fail unless the disk fills up or dies. Separate from that, on a network request driven basis, copies of those files are then send to other CAS nodes. If there is a network problem, the HTTP GET fails with an I/O error, this the operation is aborted completelycancelled, then the servers try again retried 10 or 15 seconds later. Each step places an absolute boundary between itself and the other steps. None of them can interfere with CAS mainline services. There are no queues or operations to clog and back up into the running code. 

Each of these three steps (memory to memory, memory to disk, disk to network) is independent and runs under different threads. Network problems do not block the code that writes the file, file problems do not block the code that copies references to objects. Nothing can crash CAS.

Comparison of Cushy and previous cluster technologies:

• With Cushy the only thing you have to configure is the URL of the other node. There are no multicast addresses, no ports, no timeout values, no retry counts. There is no library, no drivers, no database.
• The part of Cushy that creates disk files from tickets or restores tickets from disk files is easy and can't really fail. So what is left is the problem to get a disk file from one machine to another. If you can do that, Cushy is going to work. Cushy does it automatically if you use HTTP, and any problems are easy to diagnose.
• Cushy keeps the tickets for each node separate from the tickets for every other node. 
• Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiencyIf the network has a problem, Cushy on the other nodes uses the last copy of the files it received from each of the other nodes. That is probably good enough for 99.9% of the users. It will update as soon as it can talk to the other nodes. Meanwhile, it continues to generate its own files on disk whether the other nodes are able to read them or not.
• Other systems break down if the communication between nodes or to the database fails while the Front End is still distributing requests round-robin. If the ST did not get replicated, it cannot be validated by any node except the one that issued it. Adding the Cushy routing logic to the Front End protects against this type of problem whether you use Cushy or any other TicketRegistry.
• If anything goes wrong with Cushy, it is just an HTTP request. You know how to diagnose them.
• Cushy keeps the tickets for each node separate from the tickets for every other node. That is also simpler.
• Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiency.

Two examples in the form of a fable -

Suppose the Rapture happens and all your users have been good users and they are all transported to Heaven leaving their laptops and tablets behind. Activity ceases on the network, so all the other TicketRegistry systems have nothing to do. Cushy, however, is driven by the number of tickets in the Registry and not as much by the amount of activity. So it continues to generate and exchange checkpoint files until 8 hours after the Rapture when the logins all timeout.

Suppose (and this one we have all seen) someone doesn't really understand how applications are supposed to work with CAS, and they write their code so they get a new Service Ticket for every Web page the user accesses. CAS now sees a stream of requests to create and validate new Service Tickets. The other TicketRegistry systems replicate the Service Ticket and then immediately send a message to all nodes to delete the ticket they just created. Cushy instead just wakes up after 10 seconds and finds that all this ticket activity has mostly cancelled out. The incremental file will contain an increasing number of deleted Ticket IDs, until the next checkpoint resets it to empty and it starts growing again. If you turn on the option to ignore Service Tickets all together (because you don't really need to replicate them if you have programmed your Front End), Cushy can ignore this mess entirely.

Basic Principles

1. CAS is very important, but it is also small and cheap to run.
2. Emphasize simplicity over efficiency as long as the cost remains trivial.
3. The Front End gets the request first and it can be told what to do to keep the rest of the work simple. Let it do its job.
4. Hardware failure doesn't have to be completely transparent. We can allow one or two users to get a bad message if everything works for the other 99.9% of the users. Trying to do better than this is the source of most 100% system failures.

...

Again, Cushy 2.0 may address this problem.

Cushy CAS Cluster

In this document a CAS "cluster" is just a bunch of CAS server instances that are configured to know about each other. The term "cluster" does not imply that the Web servers are clustered in the sense that they share Session objects (JBoss "clustering"). Nor does it depend on any other type of communication between machines. In fact, a Cushy CAS cluster could be created from a CAS running under Tomcat on Windows and one running under JBoss on Linux.

...