...
"Cushy" stands for "Clustering Using Serialization to disk and Https transmission of files between servers, written by Yale". This summarizes what it is and how it works.
Cushy is designed for small and medium sized CAS installations. It might or might not work on massively large systems. There is a JUnit test you can configure to generate arbitrarily large numbers of tickets and time basic operations. If the results are not satisfactory, use one of the previous TicketRegistry options.
The Standalone Server
For a simple single standalone CAS server, the standard choice is the DefaultTicketRegistry class which keeps the tickets in an in memory Java table keyed by the ticket id string. Suppose you simply change the class name from DefaultTicketRegistry to CushyTicketRegistry (and add a few required parameters described later). Cushy was based on the DefaultTicketRegistry code, so everything works the same as before until you have to restart CAS for any reason. Since the DefaultTicketRegistry only has an in memory table, all the ticket objects are lost when the application restarts and users have to login again. Cushy detects the shutdown and saves all the ticket objects to a file on disk, using a single Java writeObject statement on the entire collection. Unless that file is deleted while CAS is down, then when CAS restarts Cushy reloads all the tickets from that file into memory and restores all the CAS state from before the shutdown. No user even notices that CAS restarted unless they tried to access CAS during the restart.
...
The cluster will still run in a modern VM infrastructure. This means that individual CAS server node outages should be measured in minutes instead of hours.
In any clustered application, all requests go to a single network address ("https://secure.its.yale.edu/cas") that points to a Front End machine. Ten years ago that Front End was dumb and simply distributed the requests round-robin across the set of back end servers. Today, Front End machines, such as the BIG-IP F5, are much smarter and they can be programmed with enough understanding of the CAS protocol so that they only round robin the initial login of new users. After that, they should be able to route requests for your tickets, whether from the browser or from applications validating a ticket, to the node you logged into which is also the node that created the ticket. Tickets only have to be replicated to protect against system failure, which allows replication to occur in seconds instead of millseconds.
The first draft of Cushy configured each server individually. At Yale, production services did not like the idea of maintaining unique configuration files for each machine. So on request the CushyClusterConfiguration class was written to simplify things. With this class you configure all the CAS clusters that you have anywhere in the network: sandbox clusters on a developers laptop, formal development, functional test, load test, and production clusters in the machine room. When CAS comes up the CushyClusterConfiguration class examines the machine it is running on and matches it to one of the machines in one of the cluster definitions. It then assigns values to properties that configure the CushyTicketRegistry and other parts of the CAS Spring configuration, especially the unique ticket ID generation objects.
When CushyTicketRegistry comes up with a cluster definition, it creates a secondary object for every other node in the same cluster. This secondary object manages communication with the node and, should the node fail, loads and holds a copy of the tickets created by that node immediately before the failure.
The simplest communication option is SharedDiskif a request arrives at the CAS virtual IP address, then the login ticketid is in the CASTGC Cookie HTTP header, the Service Ticket ID is in the ticket= parameter in the query string of a validate request, or the Proxy ticket ID is in the pgt= parameter of the query string in a /cas/proxy request. CAS has always had the ability to identify the node that created the ticket by a suffix added to all ticket ID strings. Cushy adds a formal methodology to enforce this.
Cushy can be configured node by node, but Yale Production Services did not want to configure machines individually. So Cushy adds a configuration class to which you configure the cluster. Actually, you configure every CAS cluster you have in your enterprise (desktop sandbox, development, test, stress test, production, ...). When CAS starts up the configuration class figures out which cluster this machine is a member of, and it configures that cluster and this machine. If also feeds a "ticket ID suffix" string to the CAS components that generate ticket IDs so that the Front End will route tickets properly.
How does Cushy handle clustering? At startup, it creates a "secondary" TicketRegistry that will contain a shadow copy of ticket for each of the other nodes in the cluster. However, as long as the network and nodes are healthy, Cushy only needs access to or a copy of the full checkpoint and incremental file for each node in the network. It does not open the files to restore tickets until there is a failure.
The file names are created from the node names of the CAS servers, so they can all coexist in the same directory. The simplest Cushy communication option is "SharedDisk". When this is chosen, Cushy expects that the other nodes are writing their full backup and incremental files to the same disk directory it is using. If Cushy receives a request that the Front End should have sent to another node, then Cushy assumes some failure has occurred, loads the other node's tickets into memory, and processes the request on behalf of the other node.
Of course you are free to implement SharedDisk with an actual file server or NAS, but technically Cushy doesn't know or care how the files got to the hard drive. So if you don't like real shared disk technology, you can write a shell script somewhere to wake up periodially periodically and copy the files between machines using SFTP or whatever file transfer mechanism you like to use. You could , for example, also put the 3 megabytes megabyte file on the Enterprise Service Bus if you prefer architecture to simplicity.
However, Cushy provides a built-in data transfer solution based on simple HTTPS GET requests. After all, CAS runs on a Web server, so the one thing CAS can reasonably assume is available is HTTP. Web Servers and they are very good about sending the current copy of small files over the network to clients. The checkpoint file is small, and the incremental file is smaller. Everyone understands how an HTTP GET works. So unless you configure Shared Disk"SharedDisk", Cushy running in cluster mode uses HTTP HTTPS GET to retrieve a copy of the most recent full checkpoint or incremental file from every other node in the cluster and put puts the copy in its work directory on the local hard disk of the machine.
Everything that can go wrong will go wrong. It is easy to plan for a server crashing. However, suppose you maintain multiple redundant data centers and the fiber connection is broken between centers, or a main router breaks somewhere in the network. Everything is up, but some machines cannot talk to each other. The Front End may believe a CAS server is down while other CAS servers can get to it, or the Front End may be able to talk to all servers but they may not be able to talk to each other. What about disaster recovery?
...
1) Any system that seeks to replicate tickets has a concurrency problem if there are multiple threads (like the request threads maintained by any Web Server) that can change the content of an object while another thread has triggered replication of the object. CAS has some collections in its TicketGrantingTicket object that can be changed by one Web request while another request is trying to serialize the ticket for replication to another system. CAS 3 was sloppy about this. CAS 4 added the "synchronized" attribute to methods so at least the CAS API is protected from threading problems. However, when tickets get passed to a black box cache mechanism for replication, then under the covers they are "serialized" to a stream of bytes, and serialization is not synchronized with updates unless you provide write a trivial change to protect it, and that change is not yet in CAS 4.0. As a result, any of the ticket replication technologies has a very, very small chance of throwing a ConcurrentModificationException. Cushy doesn't solve this problem yet, because it doesn't change the Ticket classes that have the bug, but it does provide a small amount of transparent pure Java code where a fix can be validated.
...
3) It is not possible to fix the previous problem in the TicketRegistry alone because the Ticket classes do not expose a method that allows the Registry to reconnect the copy of the Proxy or Service Ticket to the real TGT after it arrivesTGT already in the registry. Cushy mostly "solves" the problem because every full checkpoint (every 5 minutes or so) fixes the broken pointers, but Cushy is still stuck with the problem in tickets added by incrementals . That is a very small percent of the tickets (while with other replication options all the tickets have broken pointers that stay broken), but adding a method that allows it to fix the tickets would be helpfulbetween full checkpoints. It would be better to modify the Ticket classes so that even tickets added by incrementals could be reconnected to the real TGT instead of their private copy. In all the other replication systems, the problem is never solved and generally cannot be solved (because they hide the moment when a ticket is replicated).
The big difference here is that Cushy is designed 100% to satisfy the needs of CAS, and so we can discuss and fix those specific problems. The larger off the shelf generic libraries provide no place to fix problems specific to CAS and up to this point nobody seems to have noticed or fixed the problems.
...
- Existing cluster technologies maintain the image of a single pool of shared tickets. Cushy exploits modern programmable Front End network devices (such as the BIG-IP F5) to distribute initial CAS logons across different members of the cluster, but then to route subsequent CAS requests to the node that handled the specific user logon unless that node crashes. Each Cushy node maintains its own set of tickets.
- Existing cluster technologies try to replicate individual tickets (although the nature of Java's writeObject drags along copies of additional associated tickets). Cushy replicates a batch of tickets at regular time intervals (say every 10 seconds) and less frequently it replicates a copy of the entire collection of tickets.
- Existing cluster technologies use complex logic and databases or complex network configuration. Cushy uses HTTP that everyone understands, although you can replace this with shared files or your own trivial programs. As a result you can know how things work and how they will respond to any type of network or system failure.
- Existing cluster technologies require a cluster. Cushy does something useful on a single machine, and its clustering capability is simply an extension of that simple design.Existing cluster technologies are general purpose off the shelf libraries designed to handle any application. Cushy was written to handle CAS tickets. There are unresolved problems when CAS tickets are replicated using generic replication. In its initial distribution as a TicketRegistry component, Cushy cannot solve bugs in other CAS components, but because it exposes 100% of the logic as simple Java it provides the framework to resolve these problems when you start to use the new features of CAS 4an extension of that simple design.
- Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiency.
...
So when Cushy figures out what cluster this computer is in and assigns each node a name, it generates the TicketSuffix value and feeds it to the ticket ID generation logic on each node. In the simplest case, the suffix is just the node name. The F5, however, likes to identity hosts by the MD5 hash of their IP address.
Every CAS request except the initial login comes with one or more tickets located in different places in the request. A modern programmable Front End device like the BIG-IP F5 can be programmed to understand the CAS protocol and to locate the important ticket. The node is the suffix (the string after the third "-" character). There is a sequence of tests and you stop at the first match:
...
- After receiving a Service Ticket ID, an application opens its own HTTPS session to CAS, presents the ticket id , and if in a "validate" request. If the id is valid CAS passes back the Netid, and in certain requests can pass back additional attributes. The suffix on the ticket= parameter identifies the CAS server that created the ticket and has it in memory without requiring any high speed replication.
- When a middleware server like a Portal has obtained a CAS Proxy Granting Ticket, it requests CAS to issue a Service Ticket by making a /proxy call. Since the middleware is not a browser, it does not have a Cookie to hold the PGT. So it passes it explictly explicitly in the pgt= parameter. This looks like the previous case, but it is really much more like the next case.
- After a user logs in, CAS creates a Login TGT that points to the Netid and attributes and writes the ticket id of the TGT to the browser as a Cookie. The Cookie is scoped to the URL of the CAS application as seen from the browser point of view. At Yale this is "https://secure.its.yale.edu/cas" and so whenever the browser sees a subsequent URL that begins with this string, it appends the CASTGC Cookie with the TGT ID. CAS uses this to find the TGT object and knows that the user has already logged in. This rule sends a browser back to the CAS node the user is logged into.
- If the first three tests fail, this request is not associated with an existing logged in user. CAS has a bug/feature that it depends on Spring Web Flow and stores data during login in Web Flow storage which in turn depends on the HTTPSession object maintained by the Web Server (Tomcat, JBoss, ...). You can cluster JBoss or Tomcat servers to share HTTPSession objects over the network, but it is simpler if you program the Front End so that if the user responds in a reasonable amount of time, the login form with the userid and password is send back to the Web Server that wrote the form it to the browser in response to the browser's original HTTP GET. This is called a "sticky session" by Front Ends, and it just has to apply while the user is typing in Netid and passwordand the F5 does it automatically if you just check a box. You don't need to write code.
- Otherwise, if this is a brand new request to login to CAS or if the CAS Server selected by one of the previous steps has failed and is not responding to the Front End, then send the request to any available CAS server.
...