...
Four years ago Yale implemented a "High Availability" CAS cluster using JBoss Cache to replicate tickets. After that, the only CAS crashes were caused by failures of the ticket replication mechanism. We were disappointed that a mechanism nominally designed to improve availability should be a source of failure. We considered switching from JBoss Cache to an alternate library performing essentially the same service, but it was not clear that any other option would solve this problemall the problems.
General object replication systems are necessary for shopping cart applications that handle thousands of concurrent users spread across a number of machines. That is not really the CAS problem. CAS has a relatively light load that could probably be handled by a single server, but it needs to be available all the time, even during disaster recovery when there may be unexpected network communication problems. It also turns out that CAS tickets violate some of the restrictions that general object replication systems place on applicationsapplication objects.
CushyTicketRegistry is a much simpler solution for clustering than any of the other available systems, but it also new alternative you can plug into the TicketRegistry component of CAS. It adds useful availability features to a single standalone CAS server, but it also provides an entirely different approach to clustering two or more CAS servers for reliability. It solves the specific requirements of CAS. It is as simple as possible because it doesn't have to be a general purpose tool, and because CAS is a high value system with low resource utilization that can afford simplicity over optimizationis specifically designed to the requirements of CAS and no other application.
CAS is based on the Spring Framework, which means that internal components are selected and connected to each other using XML text files. The ticketRegistry.xml file has to configure some object that implements the TicketRegistry interface. An administrator chooses one of at least 5 alternatives. The simplest class, which keeps tickets in memory on a single standalone server, is called DefaultTicketRegistry. CushyTicketRegistry is a much enhanced version of that code.
Suppose that you start with the standard single server CAS configuration and simply . but change the class name in the XML file from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Everything works the same as before, until you shut down the CAS server. The old TicketRegistry loses all the tickets, and therefore everyone has to login again. Cushy detects the shutdown and saves all the ticket objects to a file on disk, using a single Java writeObject statement. Unless that file is deleted while CAS is down, then when CAS restarts Cushy loads all the tickets from that file into memory (although if CAS was down for a long time some may have timed out) and then CAS picks up and then CAS picks up where it left off. Users do not have to login again, and no user notices that CAS rebooted unless they tried to access CAS while it was down.
With this single modification in place it was possible to measure the cost of the operation. It turns out that the largest number of tickets normally encountered at Yale could be written to disk in less than a second, and the file was only 3MB. That is such a small cost that you don't have to wait just until shutdown. So the next step was to duplicate the existing configuration of the RegistryCleaner in the standard CAS Spring XML file to create a second timer driven operation. Every so often (2-5 minutes for example) Cushy does the same writeObject statement that is does at normal shutdown. Now, if the CAS machine crashes without a shutdown, it can come back up with almost the same state it had at the crash (except for changes between the last timer driven backup and the crash).This turns out to be well over 99% of what you want. Users log in to CAS all the time, so in any three minute period there are 10,000 unchanged logons from earlier in the day and perhaps a hundred new logons. So the next step is to pick up the remaining 1% without any measurable increase in cost. More frequently (say every 10 seconds) Cushy can write an "incremental" or "differential" file that contains the changes since the last checkpoint (mostly the new logins, but there is a chance of a logoff). This is a much smaller file and it typically takes only a few milliseconds to writeIf you examine the standard default ticketRegistry.xml configuration file you will see a few extra XML elements that configure a timer driven component called the RegistryCleaner that runs periodically to delete expired tickets. If you copy these XML statements to call the "timerDriven" method of CushyTicketRegistry, then in the regular intervals you just selected Cushy will write the same complete copy of the tickets to disk. Now if the CAS server crashes instead of shutting down normally, Cushy can restore the most recently written set of tickets, missing only the ones created between the last backup and crash.
Backing up all the tickets to disk doesn't use a lot of processing time, but it is probably not something you would do every 5 seconds. So Cushy provides a much quicker alternative that can provide complete coverage. Cushy can track the new and deleted tickets. It can write a second file called the "incremental" that contains only the new or deleted tickets since the last full checkpoint file with all the tickets was written. Typically the incremental takes only a few milliseconds to write, so you can write it as frequently as you need. Then if CAS crashes it uses one readObject to read the last full checkpoint, and then another operation to read the last incremental and add back in the tickets from the last seconds before the crash.before the crash.
Occasionally the source of the crash was a problem that prevents bringing CAS up on the same host computer. The checkpoint and incremental files are plain old ordinary disk files. They can be stored on local disk on the CAS machine, or they can be stored on a file server, or on a SAN, NAS, or some other highly available disk technology. The farther away they are the safer they are in terms of disaster recovery, but then the elapsed time to write the 3 megabytes may be a few milliseconds longer. So rather than slowing CAS down, you should let it write the file to local disk, then a shell script or other program can copy the file a second later to a remote safer location. So there may be no need for a cluster. Certainly the CAS function is small enough and simple enough that it doesn't need to be load balanced. It is certainly possible to configure a server with enough processing power to handle the largest CAS load on a single machineIf the CAS machine is unbootable, you can bring up a copy of CAS from the remote backup on other hardware.
The idea of using a cluster for availability made sense ten years ago when servers were physical machines and recovery involved manual intervention. Today servers run on VMs in a highly managed environment and backup VMs can be spun off automatically. It may be possible to design a system so that the backup comes up automatically and so quickly that you don't need a cluster at all. Cushy supports this profile and strategy.
However, if you still insist on building a CAS cluster, then consider the small number of very specific programming problems that any CAS cluster must solve:
...