...
CAS is a Single SignOn solution. Internally, it creates a set of objects called Tickets. There is a ticket for every logged on user, and short term Service Tickets that exist while a user is being authenticated to an application. The Business Layer of CAS creates tickets by, for example, validating your userid and password in a back end system like Active Directory. The tickets are stored in a plug in component called a Ticket Registry.
CAS is configured with Spring XML. This sets values that define the CAS node. Its only memory ("state") is the set of Tickets in the Ticket Registry.
For a single CAS server, the Ticket Registry is just a in memory table of tickets (a Java "Map" object) keyed indexed by the ticket ID string. When more than one CAS server is combined to form a cluster, then an administrator chooses one of several optional Ticket Registry solutions that allow the CAS servers to share the tickets.
One clustering option is to use JPA, the standard Java service to map objects to tables in a relational database. All the CAS servers share a database, which means that any CAS node can fail but the database has to stay up all the time or CAS stops working. Other solutions use generic object "caching" solutions (Ehcache, JBoss Cache, Memcached) where CAS puts the tickets into what appears to be a common container of Java objects and, under the covers, the cache technology ensures that the new tickets are copied to all the other nodes.
JPA makes CAS dependent on a database. It doesn't really use the database for any real SQL stuff, so you could you almost any database system. However, queries or reports. You can use any database, but the database is a single point of failure, so you need it to be reliable. At Yale CAS is nearly the first thing that has to come up during disaster recovery, but if it uses JPA then you have to bring up the database (or have a special standalone CAS configuration for disaster recovery only). If you already have a 24x7x365 database managed by professionals who can guarantee availability, this is a good solution. If not, then this is an insurmountable prerequisite for bringing up an application like CAS that doesn't really need database.
The various cache (in memory object replication) solutions should also work. Unfortunately, they some have massively complex configuration parameters with multicast network addresses and timeouts.timeout values to determine node failure.They also tend to be better at detecting a node that is dead and does not respond than they are at dealing with nodes that are sick and accept a message but then never really get to processing it and responding. They operate entirely in memory, so at least one node has to remain up while the others reboot in order to maintain the content of the cache. While node failure is well defined, the status of objects is ambiguous if the network is divided into two segments by a linkage failure, the two segments operate independently for a while, and then connection is reestablished.
...
The name explains what it does. Java has a built in operation called named writeObject that writes a binary ("serialized") version of Java objects to disk. If you You can use it on a complex single object, like but if you pass it a list of all the tickets in the Registry, then it creates a disk file with all the tickets in the listor table of objects then it copies everything in the list and captures all the relationships between the objects. Later on you can use readObject to turn the disk file back into a from the same program, from a different JVM, or from a different computer and restore to memory an exact copy of the original list , or table and all the objects in the list are automatically connected back and forth with references identical to the original objects. Java calls this mechanism "Serialization". Using just one statement and letting Java do all the work and handle all the complexity makes this easy.The other mechanisms (JPA or the cache technologies) operate on single tickets. They it contains. This is a very complex process, but Java handles all the complexity. To the program that uses it, it is a single statement.
JPA and the various "cache" technologies try to write individual tickets to the database or replicate them across the network. Obviously this is vastly more efficient than periodically copying all the tickets to disk. Except that from node to node. They may also write additional tickets connected to the ticket you intended to copy. Obviously it is more efficient to write only the changed tickets rather than writing all the tickets. However, at Yale (a typical medium sized university), the entire Registry of less than 20,000 tickets can be written to a disk file in 1 second and it produces a file about 3 megabytes in size. That is a trivial use of modern multicore server hardware, and copying 3 megabytes of data over the network every 5 minutes, or even every minute, is a trivial use of network bandwidth. So Cushy is less efficient, but in a way that is predictable and insignificant, in exchange for code that is simple and easy to completely understand.
Once the tickets are a file on disk, the Web server provides an obvious way (HTTPS GET) to transfer them from one server to another. Instead of using complex multicast sockets with complex error recovery, you are using a simple technology everyone understands to accomplish a trivial function. You can immediately understand the consequences of any network failure and eventual network recovery.
Cache solutions go memory to memory. Adding an intermediate disk file wasn't an obvious step, but once you think of it it has some added benefits. If you reboot the CAS server, the local disk file allows CAS to immediately restore the tickets and therefore its state from before the reboot. Serializing the tickets to disk will work no matter how badly the network or other nodes are damaged, and it is the only step that involves the existing CAS code. Although the second step, transferring the file from one server to another, is accomplished with new code that runs in the CAS Web application, it does not touch a single existing CAS object or class. So whatever unexpected problems the network might create, they affect only the independent file transfer logic leaving normal CAS function untouched. And while the cache solutions require complex logic to reconcile cache on different machines after communication between nodes is restored, Cushy retransmits the entire set of tickets every few minutes after which everyone is guaranteed to be back in synchronization.
...
Back in the 1960's a "checkpoint" was a copy of the important information from a program written on disk so if the computer crashed the program could start back at almost the point it left off. If a CAS server saves its tickets to a checkpoint disk file, reboots, and then reads restores the tickets from the file back into memory it is back to the same state it had before rebooting. If you transfer the file to another computer and bring CAS up on that machine, you have moved the CAS server from one machine to another. Java's writeObject and readObject guarantee the state and data are completely saved and restored.
...
JPA and the cache technologies try to maintain the image of a single big common bucket of shared tickets.This seems like it is fairly simple, but it creates synchronization problems between servers while serving no useful CAS purpose. Cushy maintains a separate TicketRegistry for each CAS server that holds all the tickets created by that server. It then replicates a copy of each TicketRegistry to all the other servers in the cluster. No server has to look at the Registry of another server until the other server fails and the Front End starts routing its requests to the other nodes.
You could configure a Cushy cluster to only make full checkpoints files containing all the tickets. The cost of a checkpoint is small, but it is large enough that you might be reluctant to schedule them frequently enough to provide the best protection. So between full checkpoints, Cushy creates and transmits a sequence of "incremental" change files that each have all the changes since the last full checkpoint. In the Spring XML configuration file you set the time between incrementals and the time between checkpoints. The choice is up to you, but a reasonable suggestion is to exchange incrementals every 5-15 seconds and checkpoints every 3-15 minutes.
Each incremental has a small number of new Login (TGT) tickets and maybe a few unclaimed service tickets. However, because we do not know whether any previous incremental was or was not processed, it is necessary to transmit the list of every ticket that was deleted since the last full checkpoint, and that will contain the ID of lots of Service Tickets that were created, was necessary when the network Front End device simply accepted HTTP requests and assigned them to CAS servers is a round robin manner. Today network Front End devices are programmable and they can make decisions based on specific CAS logic. This allows each CAS server to own its own private slice of the problem.
When a new user is redirected to CAS, then the Front End can randomly choose a server. However, after the user logs in and is assigned a Cookie, the Front End should always route subsequent requests to the server that issued the cookie. That means that Service Tickets and Proxy Tickets are issued by the CAS server you logged into. The Front End can also be programmed to recognize validation requests (/validate, /serviceValidate, etc.) and route those requests to the server that issued the ticket identified by the ticket= parameter. Configuration for the BIG-IP F5 will be provided. If you do not have a smart Front End device, then use a different Ticket Registry technology.
With an intelligent Front End, there is no need for a Ticket Registry that simulates a big shared pool of tickets. Each node has its own registry with its own logged in users and the tickets they create. No other node needs to access these tickets, unless the node that owns them fails. Then any other node, or all the other nodes, handle requests until the failed node is restarted.
You could configure a Cushy cluster to only make full checkpoints files containing all the tickets. The cost of a checkpoint is small, but it is large enough that you might be reluctant to schedule them frequently enough to provide the best protection. So between full checkpoints, Cushy creates and transmits a sequence of "incremental" change files that each have all the changes since the last full checkpoint. In the Spring XML configuration file you set the time between incrementals and the time between checkpoints. The choice is up to you, but a reasonable suggestion is to exchange incrementals every 5-15 seconds and checkpoints every 3-15 minutes.
Each incremental has a small number of new Login (TGT) tickets and maybe a few unclaimed service tickets. However, because we do not know whether any previous incremental was or was not processed, it is necessary to transmit the list of every ticket that was deleted since the last full checkpoint, and that will contain the ID of lots of Service Tickets that were created, validated, and deleted within a few milliseconds. That list is going to grow, and its size is limited by the fact that we can start over again after each full checkpoint.
A Service Ticket is created and then is immediately validated and deleted. Trying to replicate Service Tickets to the other nodes before the validation request comes in is an enormous problem that screws up the configuration and timing parameters for all the other Ticket Registry solutions. Cushy doesn't try to do replication at this speed. Instead, it has CAS configuration elements that ensure that each Ticket ID contains an identifier of the node that created it, and it depends on a front end smart enough to route any of the ticket validation requests to the node that created the ticket and already has it in memory. Then replication only is needed for crash recovery.
Note: If the front end is not fully programmable it is a small programming exercise to be considered in Cushy 2.0 to forward the validation request from any CAS node to the node that owns the ticket and then pass back the results of the validation to the app.
Ticket Names
As with everything else, CAS has a Spring bean configuration file (uniqueIdGenerators.xml) to configure how ticket ids are generated. If you accept the defaults, then tickets have Note: Replicating Service Tickets between nodes is almost never useful. The "p:excludeSTFromFiles" parameter in the Spring configuration XML causes Cushy to ignore Service Tickets, which keeps the deleted tickets list small and limits the growth of incrementals, if you prefer a very long time between full checkpoints.
Ticket Names
As with everything else, CAS has a Spring bean configuration file (uniqueIdGenerators.xml) to configure how ticket ids are generated. The default class generates tickets in the following format:
type - num - random - nodename
where type is "TGT" or "ST", num is a ticket sequence number, random is a large random string like "dmKAsulC6kggRBLyKgVnLcGfyDhNc5DdGKT", and the suffix at the end of the ticket is identified as a nodename.
In vanilla CAS the nodename typically comes from the cas.properties file, but Cushy requires every node in the cluster to have a unique name and even when you are using real clustering many CAS locations leave the "nodename" suffix on the ticket id to its default value of "-CAS". Cushy adds a smarter configuration bean described below and enforces the rule that the end of the ticket really identifies the node that created it and therefore owns it.
How it Fails (Nicely)
The Primary + Warm Spare Cluster
One common cluster model is to have a single master CAS server that normally handles all the requests, and a normally idle backup server (a "warm spare") that does nothing until the master goes down. Then the backup server handles requests while the master is down.
During normal processing the master server is generating tickets, creating checkpoints and increments, and sends them to the backup server. The backup server is generating empty checkpoints with no tickets because it has not yet received a request.
Then the master is shut down or crashes. The backup server has a copy in memory of all the tickets generated by the master, except for the last few seconds before the crash. It can handle new logins and it can issue Service Tickets against logins previously processed by the master, using its copy of the master's registry.
Now the master comes back up and, for this example, let us assume that it resumes its role as master (there are configurations where the backup becomes the new master and so when the old master comes back it becomes the new backup. This is actually easier for Cushy).
The master restores from disk a copy of its old registry and over the network it fetches a copy of the registry from the backup. It now has access to all the login or proxy tickets created by the backup while it was down, and it can issue Service Tickets based on those loginsthe end of the ticket is identified as a nodename.
In vanilla CAS the nodename typically comes from the cas.properties file and defaults to "CAS". Cushy requires each node in the cluster to have a unique node name. The configuration of the CushyClusterConfiguration bean makes this somewhat easy (as described below) and it also generates the clusterConfiguration.getTicketSuffix property that can be used to plug a real node name into the uniqueIdGenerators.xml file:
<bean id="ticketGrantingTicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator">
<constructor-arg index="0" type="int" value="50" />
<constructor-arg index="1" value="#{clusterConfiguration.getTicketSuffix()}" />
</bean>
How it Fails (Nicely)
The Primary + Warm Spare Cluster
One common cluster model is to have a single master CAS server that normally handles all the requests, and a normally idle backup server (a "warm spare") that does nothing until the master goes down. Then the backup server handles requests while the master is down.
During normal processing the master server is generating tickets, creating checkpoints and increments, and sends them to the backup server. The backup server is generating empty checkpoints with no tickets because it has not yet received a request.
Then the master is shut down or crashes. The backup server has a copy in memory of all the tickets generated by the master, except for the last few seconds before the crash. It can handle new logins and it can issue Service Tickets against logins previously processed by the master, using its copy of the master's registry.
Now the master comes back up and, for this example, let us assume that it resumes its role as master (there are configurations where the backup becomes the new master and so when the old master comes back it becomes the new backup. Cushy works either way).
What happens next depends on how smart the Front End is. If it has been programmed to route requests based on the suffix of the tickets in the login cookie, then users who logged into the backup server during the failure continue to use the backup server, while new users all go back to the master. If the Front End is programmed to route all requests to the master as long as the master is up, then it appears that when the master came up the backup server "failed over to the master".
Lets be careful here. The Front End things a node has failed if it cannot get to the node. It then routes all requests to other nodes. However, if the failure is in a router between the Front End and the node in question, then the nodes themselves may be able to connect to each other and exchange checkpoint and incremental files.
In this sense, if the Front End begins to route requests for tickets owned by the backup server to the newly rebooted master, then the master and backup are still able to talk to each other and the master gets the latest checkpoint file from the backup server, but from that point on the master is servicing all the requests on behalf of the backup server as if the backup server failed.
However, the failure has left some minor issues that are not important enough to be problems. Because each server is the owner of its own tickets and registry, each has Read-Only access to the tickets of the other server. (Strictly speaking that is not true. You can temporarily change tickets in your copy of the other node's registry, but when the other node comes back up and generates its first next checkpoint, whatever changes you made will be replaced by a copy of the old unmodified ticket). So the master is unaware of CAS logouts that occurred while it was down and although it can process a logout for a user that logged into the backup while it was down, it really has no way to actually delete the login ticket. Since no browser has the TGT ID in a cookie any more, nobody will actually be able to use the zombie TGT, but the ticket is going to sit around in memory until it times out.There are a few more consequences to Single Sign Out that will be explained in the next sectionbe replaced by a copy of the old unmodified ticket).
This means that the master is unaware of things the backup server did that should have modified its tickets. For example, if a user logs out of CAS on the backup, then when the master comes back the Logon Ticket that was supposed to be deleted is restored. Of course, no browser has that ticketid as a cookie, so the ticket will sit there unused until it times out. The same thing happens in reverse to users who logged into the backup server while the master is down. When the master comes up and handles the logout it cannot really delete the Logon Ticket that belongs to the backup server, so the ticket sits on the backup server unused until it times out. This is an example of the principle that Cushy is designed to handle the specific problem of CAS Tickets in a way that is simple but adequate to CAS needs.
A Smart Front End
A programmable front end Front End is configured to send Validate requests to the CAS server that generated the Service Ticket, /proxy requests to the CAS server that generated the PGT, other requests of logged on users to the CAS server they logged into, and login requests based on standard load balancing or similar configurations. Each ticket has a suffix that indicates which CAS server node generated it.adequate to Cushy needs if it can route requests based on four rules:
- If the URL "path" is a validate request (/cas/validate, /cas/serviceValidate, etc.) then route to the node indicated by the suffix on the value of the ticket= parameter.
- If the URL is a /proxy request, route to the node indicated by the suffix of the pgt= parameter.
- If the request has a CASTGC cookie, then route to the node indicated by the suffix of the TGT that is the cookie's value.
- Otherwise, or if the node selected by 1-3 is down, choose a CAS node using whatever round robin or master-backup algorithm previously configured.
...