Content Comparison

...

Existing cluster technologies maintain the image of a single pool of shared tickets. Cushy exploits modern programmable Front End network devices (such as the BIG-IP F5) to distribute initial CAS logons across different members of the cluster, but then to route subsequent CAS requests to the node that handled the specific user logon unless that node crashes. Each Cushy node maintains its own set of tickets.
Existing cluster technologies try to replicate individual tickets (although the nature of Java's writeObject drags along copies of additional associated tickets). Cushy replicates a batch of tickets at regular time intervals (say every 10 seconds) and less frequently it replicates a copy of the entire collection of tickets.
Existing cluster technologies use complex logic and databases or complex network configuration. Cushy uses HTTP that everyone understands, although you can replace this with shared files or your own trivial programs. As a result you can know how things work and how they will respond to any type of network or system failure.
Existing cluster technologies require a cluster. Cushy does something useful on a single machine, and its clustering capability is simply an extension of that simple design.
Existing cluster technologies are general purpose off the shelf libraries designed to handle any application. Cushy was written to handle CAS tickets. There are unresolved problems when CAS tickets are replicated using generic replication. In its initial distribution as a TicketRegistry component, Cushy cannot solve bugs in other CAS components, but because it exposes 100% of the logic as simple Java it provides the framework to resolve these problems when you start to use the new features of CAS 4.
Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiency.

Thou Shall Not

...

Basic Principles

CAS is very important, but it is also small and cheap to run.
Emphasize simplicity over efficiency as long as the cost to run remains trivial.
Assume the network front end is programmable.
Trying for perfection is the source of most total system failures. Allow The Front End gets the request first and it can be told what to do to keep the rest of the work simple. Let it do its job.
Hardware failure doesn't have to be completely transparent. We can allow one or two users to get a temporary error message when a CAS server fails, but make it impossible for any single failure to crash everything.

A Bit More Detail on CAS Tickets

When the user logs in, CAS creates a Logon Ticket (the Ticket Granting Ticket or TGT because it can be used to generate other tickets). You can usually get away with believing that the TGT contains the login userid and user attributes, but there is really a chain of objects. The TGT points to an Authentication that points to a Principal that points to the username. The TGT can also contain a collection of Attributes used to generate SAML responses. In most cases you can ignore this chain of objects, unless you are writing or trying to understand a JUnit Test.

In CAS 3 the TGT is fairly stable once it is created. There is a table of pairs of Service Ticket ID strings and Service objects used by Single Sign Off that gets a new entry every time a Service Ticket is created, but otherwise the TGT doesn't change. With CAS 4 things threaten to become more interesting. With multiple factors of authentication and the possibility of adding new factors to an existing logon, the TGT will become a more interesting and active object once these features are implemented.

The simplest next step occurs when a user who has logged in and has a TGT decides to access an application that redirects the browser to CAS to obtain a Service Ticket. A Service Ticket object is created. It points to the TGT of the logged in user, and it contains the Service URL of the application. It exists for a few milliseconds before the application connects back to CAS to validate the ST ID string.

At validation, the Business Logic looks up the ST ID string in the Ticket Registry and gets the Service Ticket object. It points to the TGT object, and from that the validation code can obtain the userid (from the Authentication and Principal objects) and the Attributes (if this is a SAML validation). Then the ST is deleted.

A more complicated situation occurs when the application is a Proxy service, like a Portal. Then the CAS Business Logic trades a Service Ticket object in and generates a new TGT object in return (the Proxy TGT is called a Proxy Granting Ticket or PGT to distinguish it). A PGT is a form of TGT except that it points to the real TGT that contains the userid and attributes, and in the PGT if you follow the chain of Authentication and Principal objects you will end up with the Service URL in the place where a TGT has the userid.

The PGT can be used to obtain Service Tickets. When this happens, the ST points to the PGT which in turn points to the TGT that "contains" the userid.

So when you are thinking about Ticket Registries, or when you are designing JUnit test cases, there are four things to think about:

a TGT
a ST pointing to a TGT
a PGT pointing to a TGT
a ST pointing to a PGT pointing to a TGT

In various node failure scenarios, at one of the "pointing to" breaks you can jump from the current node's TicketRegistry to a backup shadow TicketRegistry copy of tickets belonging to a failed node. For example, the ST could point to the PGT and TGT in the failed node's registry, or the ST could point to a local PGT that then points to a TGT in the failed node's registry. Create the possibilities and verify that they work, but also remember that these have to work in order for the design to handle failures properly.

Keep It Simple

Back in the 1960's a "checkpoint" was a copy of the important information from a program written on disk so if the computer crashed the program could start back at almost the point it left off. If a CAS server saves its tickets to a checkpoint disk file, reboots, and then restores the tickets from the file back into memory it is back to the same state it had before rebooting. So the basic idea of Cushy is about 50 years old.

Writing the checkpoint file is fairly trivial, so would typically configure the time between checkpoints in minutes.

Between checkpoints you can write the changed objects to a separate file. You could write a sequence of files with changes and require that they be applied in order, but that it more complicated. The amount of data between full checkpoints is small, so it makes sense to maintain a single file with all the accumulated changes. Whenever you see a new file you can apply it to get a current set of tickets without worrying about whether previous versions of the file have been applied or if any version has been missed.

A user requests a Service Ticket. CAS generates the ST and writes its ID string back to the browser. The browser then forwards the ID string on to the application, which opens a separate HTTPS session to CAS to validate the ID and return the netid and possible attributes. This can be handled by replicating the Service Ticket across the network in milliseconds, or even worse by requiring that it has been replicated before you return the ID string to the browser. It is much simpler to program the network Front End device (by writing an iRule for the F5) to recognize the /cas/validate, /cas/serviceValidate, and similar path elements and route all such requests to the node whose id is appended to the ticket= parameter. Then you can replicate in seconds instead of milliseconds and that make everything much easier.

Ticket Names

As with everything else, CAS has a Spring bean configuration file (uniqueIdGenerators.xml) to configure how ticket ids are generated. The default class generates tickets in the following format:

type - num - random - nodename

where type is "TGT" or "ST", num is a ticket sequence number, random is a large random string like "dmKAsulC6kggRBLyKgVnLcGfyDhNc5DdGKT", and the suffix at the end of the ticket is identified as a nodename.

In vanilla CAS the nodename typically comes from the cas.properties file and defaults to "CAS". Cushy requires each node in the cluster to have a unique node name. The configuration of the CushyClusterConfiguration bean makes this somewhat easy (as described below) and it also generates the clusterConfiguration.getTicketSuffix property that can be used to plug a real node name into the uniqueIdGenerators.xml file:

How it Fails (Nicely)

The Primary + Warm Spare Cluster

One common cluster model is to have a single master CAS server that normally handles all the requests, and a normally idle backup server (a "warm spare") that does nothing until the master goes down. Then the backup server handles requests while the master is down.

During normal processing the master server is generating tickets, creating checkpoints and increments, and sending them to the backup server. The backup server is generating empty checkpoints with no tickets because it has not yet received a request.

Then the master is shut down or crashes. The backup server has a copy in memory of all the tickets generated by the master, except for the last few seconds before the crash. When new users log in, it creates new Login Tickets in its own Ticket Registry. When it gets a request for a new Service Ticket for a user who logged into the master, it creates the ST in its own registry (with its own nodename suffix) but connects the ST to the Login Ticket in its copy of the master's Ticket Registry.

Remember the CAS Business Logic is used to a Ticket Registry maintaining what appears to be a large collection of tickets shared by all the nodes. So the Business Logic is quite happy with a Service Ticket created by one node pointing to a Login Ticket created by another node.

Now the master comes back up and, for this example, let us assume that it resumes its role as master (there are configurations where the backup becomes the new master and so when the old master comes back it becomes the new backup. Cushy works either way).

What happens next depends on how smart the Front End is. If it has been programmed to route requests based on the suffix of the tickets in the login cookie, then users who logged into the backup server during the failure continue to use the backup server, while new users all go back to the master. If the Front End is programmed to route all requests to the master as long as the master is up, then it appears that when the master came up the backup server "failed over to the master".

When the master comes up it reloads its old copy of its ticket registry from before the crash, and it gets a copy of the tickets generated by the backup server while it was down. When it subsequently gets requests from users who logged into the backup server, it resolves those requests using its copy of that TGT.

This leaves a few residual "issues" that are not really big problems and are deferred until Cushy 2.0. Because each server is the owner of its own tickets, and its Ticket Registry is the authoritative source of status on its own tickets, other nodes cannot make permanent changes to another node's tickets during a failover.

This means that the master is unaware of things the backup server did while it was down that should have modified its tickets. For example, if a user logs out of CAS while the backup server is in control, then the Cookie gets deleted and all the normal CAS logoff processing is done, but the Login Ticket (the TGT) cannot really be deleted. That ticket belongs to the master, and when the master comes back up again it will be in the restored Registry. However, it turns out that CAS doesn't really have to delete the ticket. Since the cookie has been deleted, nobody is going to try and use it. It will simply sit around until it times out and is deleted later on.

A more serious problem occurs for Single Sign Out of people who logged into the backup server while the master is down in systems where the Front End processor is not programmed to route requests intelligently. When the master reboots and starts handling all new requests, they have a TGT is that is "frozen" to the state it was in when the master rebooted. The master can subsequently create new Service Tickets from that TGT, but Single Sign Out will not know to log them off from those services when the user logs off. The current solution is to use Front End programming. Cushy 2.0 may add intelligent TGT migration and merging after a CAS server reboots.

A Smart Front End Cluster

A programmable Front End is adequate to Cushy needs if it can route requests based on four rules:

If the URL "path" is a validate request (/cas/validate, /cas/serviceValidate, etc.) then route to the node indicated by the suffix on the value of the ticket= parameter.
If the URL is a /proxy request, route to the node indicated by the suffix of the pgt= parameter.
If the request has a CASTGC cookie, then route to the node indicated by the suffix of the TGT that is the cookie's value.
Otherwise, or if the node selected by 1-3 is down, choose any CAS node

So normally all requests go to the machine that created and therefore owns the ticket, no matter what type of ticket it is. When a CAS server fails, requests for its tickets are assigned to one of the other servers.

When a CAS server receives a request for a ticket owned by another node, it fully activates the other nodes shadow Ticket Registry. It then looks up the ticket in that registry and returns it to the CAS Business Logic. A node may not have a copy of tickets issued in the last few seconds, so one or two users may see an error.

Cushy can issue a Service Ticket that points to a Login Ticket owned by the failed node. More interestingly, it can issue a Proxy Granting Ticket pointing to the Login Ticket on the failed node. In both cases the new ticket has the suffix and is owned by the node that created it and not by the node that owns the login.

Again, the rule that each node owns its own registry and all the tickets it created and the other nodes can't successfully change those tickets has certain consequences.

If you use Single Sign Off, then the Login Ticket maintains a table of Services to which you have logged in so that when you logout or when your Login Ticket times out in the middle of the night then each Service gets a call from CAS on a published URL with the Service Ticket ID you used to login so the application can log you off if it has not already done so. In fail-over mode a backup server can issue Service Tickets for a failed nodes TGT, but it cannot successfully update the Service table in the TGT, because when the failed node comes back up it will restore the old Service table along with the old TGT.
If the user logs out and the Services are notified by the backup CAS server, and then the node that owned the TGT is restored along with the now undead copy of the obsolete TGT, then in the middle of the night that restored TGT will timeout and the Services will all be notified of the logoff a second time. It seems unlikely that anyone would ever write a service logout so badly that a second logoff would be a problem. Mostly it will be ignored.

You have probably guessed by now that Yale does not use Single Sign Out, and if we ever enabled it we would only indicate that it is supported on a "best effort" basis in the event of a CAS node crash.

CAS Cluster

In this document a CAS "cluster" is just a bunch of CAS server instances that are configured to know about each other. The term "cluster" does not imply that the Web servers are clustered in the sense that they share Session information. Nor does it depend on any other type of communication between machines. In fact, a CAS cluster could be created from a CAS running under Tomcat on Windows and one running under JBoss on Linux.

To the outside world, the cluster typically shares a common virtual URL simulated by the Front End device. At Yale, CAS is "https://secure.its.yale.edu/cas" to all the users and applications. The "secure.its.yale.edu" DNS name is associated with an IP address managed by the BIG-IP F5 device. It terminates the SSL, then examines requests and based on programming called iRules it forwards requests to any of the configured CAS virtual machines.

Each virtual machine has a native DNS name and URL. It is these "native" URLs that define the cluster because each CAS VM has to use the native URL to talk to another CAS VM. At Yale those URLs follow a pattern of "https://vm-foodevapp-01.web.yale.internal:8443/cas".

Internally, Cushy configuration takes a list of URLs and generates a cluster definition with three pieces of data for each cluster member: a nodename like "vmfoodevapp01" (the first element of the DNS name with dashes removed), the URL, and the ticket suffix that identifies that node (at Yale the F5 likes the ticket suffix to be an MD5 hash of the DNS name).

Sticky Browser Sessions

An F5 can be configured to have "sticky" connections between a client and a server. The first time the browser connects to a service name it is assigned any available back-end server. For the next few minutes, however, subsequently requests from that client to the same service are forwarded to whichever server the F5 assigned to handle the first request.

While a user is logging in to CAS with the form that takes userid and password, or any other credentials, there is no Ticket. No cookie, no ticket=, none of the features that would trigger the first three rules of the programmable intelligent Front End. CAS was designed (for better or worse) to use Spring Webflow which keeps information in the Session object during the login process. For Web Flow to work, one of two things must happen:

The browser has to POST the Userid/Password form back to the CAS server that sent it the form (which means the front end has to use sticky sessions based on IP address or JSESSIONID value).
You have to use real Web Server clustering so the Web Servers all exchange Session objects based on JSESSIONID.

Option 2 is a fairly complex process of container configuration, unless you have already solved this problem and routinely generate JBoss cluster VMs using some canned script. Sticky sessions in the front end are somewhat easier to configure, but any sticky session rule MUST apply only after the first three rules (ticket= suffix, pgt= suffix, or CASTGC suffix) have been tested and found not to apply.

There is another solution, but it involves a CAS modification. Yale made a minor change to the CAS Web Flow to store data that Web Flow saves in the Session object also in hidden fields of the login form (because it is not secure information). Then there is a check at the beginning of the Web Flow for a POST arriving at the beginning of the flow, allowing it to jump forward to the step of the Flow that handles the Form submission.

What is a Ticket Registry

This is a rather detailed description of one CAS component, but it does not assume any prior knowledge.

Layers

Web applications are traditionally defined in three layers. The User Interface generates the Web pages, displays data, and processes user input. The Business Logic validates requests, verifies inventory, approves the credit card, and so on. The back-end "Persistence" layer talks to a database. CAS doesn't sell anything, but it has roughly the same three layers.

In CAS the "User Interface" layer has two jobs. The part that talks to real users handles login requests through the Spring Web Flow services. However, CAS also accepts Web requests from the applications that are trying to validate a Service Ticket and get information about the user. This is also part of the UI layer, and it is handled by the Spring MVC framework.

Cushy extends this second part of the UI so that node to node communication within the cluster also flows through MVC.

The Business Logic layer of CAS verifies the userid and password or any other credentials, and it creates and deletes the TGT and ST objects.It also validates Service Tickets and deletes them after use.

The Persistence layer implements the TicketRegistry interface. In the simplest case of a single CAS server using the DefaultTicketRegistry, the tickets are stored in an in memory table and there is no back end database or network I/O. JPA stores the tickets in a database. The "cache" solutions trigger network I/O.

Cushy stores the tickets in memory, just like the DefaultTicketRegistry. Periodically it backs the table up to a file on disk, but that is not part of the CAS request processing flow, so the checkpoint files and HTTP file transfer are not part of the application layers.

Spring Configuration

Many applications have their own custom configuration file. Spring is a Java framework that provide a much more powerful configuration environment that is, necessarily, somewhat more complicated. Consider the TicketRegistry layer and interface.

The CAS Business Logic configuration requires that some Java class be loaded into memory, that an object of that class be created and then configured with parameters, and that its name be "ticketRegistry". The object also has to implement the TicketRegistry Java interface. CAS provides a number of classes that can do the job, including DefaultTicketRegistry that holds the tickets in memory and has no important configuration parameters.

In the CAS WAR file (the web application file deployed to web servers) there is a ticketRegistry.xml file where, by CAS convention, the class that implements the TicketRegistry interface should be configured.

The Serialization Problem of Current CAS

JPA is the current technique for creating Java objects from a database query, updating objects, and committing changes back to the database. To support JPA, the TGT and ST Java objects have "annotations" to define the names of tables and columns that correspond to each object and field. If you don't use JPA, these annotations are ignored. If you use JPA, then it automatically generates additional Java code that is added to every ticket object to track when it is used and updated. JPA is the only TicketRegistry solution that doesn't use serialization (other than DefaultTicketRegistry that does nothing).

The "cache" (Ehcache, JBoss Cache, Memcached) JASIG TicketRegistry modules have no annotations and few expectations. They use ordinary objects (sometimes call Plain Old Java Objects or POJOs). They require the objects to be serializable because, like Cushy, they use the Java writeObject statement to turn any object to a stream of bytes that can be held in memory, stored on disk, or sent over the network.

...

bad message if everything works for the other 99.9% of the users. Trying to do better than this is the source of most 100% system failures.

Ticket Chains (and Test Cases)

A TGT represents a logged on user. It is called a Ticket Granting Ticket because it is used to create Service and Proxy tickets. It has no parent and stands alone.

When a user requests it, CAS uses the TGT to create a Service Ticket. The ST points to the TGT that created it, so when the application validates the ST id string, CAS can follow the chain from the ST to the TGT to get the Netid and attributes to return to the application. Then the ST is discarded.

However, when a middleware application like a Portal supports CAS Proxy protocol, the CAS Business Logic layer trades an ST (pointing to a TGT) in and turns it into a second type of TGT (the Proxy Granting Ticket or PGT). The term "PGT" exists only in documents like this. Internally CAS just creates a second TGT that points to the login TGT.

If the Proxy application accesses a backend application, it calls the /proxy service passing the TGT ID and gets back a Service Ticket ID. That ST points to the PGT that points to the TGT from which CAS can find the Netid.

So when you are thinking about Ticket Registries, or when you are designing JUnit test cases, there are four basic arrangements to consider:

a TGT
a ST pointing to a TGT
a PGT pointing to a TGT
a ST pointing to a PGT pointing to a TGT

This becomes an outline for various cluster node failure tests. Whenever one ticket points to a parent there is a model where the ticket pointed to was created on a node that failed and the new ticket has to be created on the backup server acting on behalf of that node. So you want to test the creation and validation of a Service Ticket on node B when the TGT was created on node A, or the creation of a PGT on node B when the TGT was created on node A, and so on.

Front End Programming

CAS Ticket IDs have four sections:

type - num - random - suffix

where type is "TGT" or "ST", num is a ticket sequence number, random is a large random string like "dmKAsulC6kggRBLyKgVnLcGfyDhNc5DdGKT", and the suffix at the end is configured in the XML.

There are separate XML configurations for different types of tickets, but they all look alike and they all occur in the uniqueIdGenerators.xml file. With cushy the suffix is tied to the TicketSuffix property generated by the CushyClusterConfiguration:

So when Cushy figures out what cluster this computer is in and assigns each node a name, it generates the TicketSuffix value and feeds it to the ticket ID generation logic on each node. In the simplest case, the suffix is just the node name.

Every CAS request except the initial login comes with one or more tickets located in different places in the request. A modern programmable Front End device like the BIG-IP F5 can be programmed to understand the CAS protocol and to locate the important ticket. The node is the suffix (the string after the third "-" character). There is a sequence of tests and you stop at the first match:

If the Path part of the URL is a validate request (/cas/validate, /cas/serviceValidate, /cas/proxyValidate, or /cas/samlValidate) then look at the ticket= parameter in the query string part of the URL
If the Path part of the URL is a /cas/proxy request, then look at the pgt= parameter in the query string.
If the request has a CASTGC cookie, then look at the cookie value.
If a request has been seen from this browser in the last 5 minutes, then send it to the same node it was previously sent to.
Otherwise, or if the node selected by 1-4 is down, choose any CAS node

That is the code, now here is the explanation:

After receiving a Service Ticket ID, an application opens its own HTTPS session to CAS, presents the ticket id, and if the id is valid CAS passes back the Netid, and in certain requests can pass back additional attributes. The suffix on the ticket= parameter identifies the CAS server that created the ticket and has it in memory without requiring any high speed replication.
When a middleware server like a Portal has obtained a CAS Proxy Granting Ticket, it requests CAS to issue a Service Ticket by making a /proxy call. Since the middleware is not a browser, it does not have a Cookie to hold the PGT. So it passes it explictly in the pgt= parameter. This looks like the previous case, but it is really much more like the next case.
After a user logs in, CAS creates a Login TGT that points to the Netid and attributes and writes the ticket id of the TGT to the browser as a Cookie. The Cookie is scoped to the URL of the CAS application as seen from the browser point of view. At Yale this is "https://secure.its.yale.edu/cas" and so whenever the browser sees a subsequent URL that begins with this string, it appends the CASTGC Cookie with the TGT ID. CAS uses this to find the TGT object and knows that the user has already logged in.
If the first three tests fail, this request is not associated with an existing logged in user. CAS has a bug/feature that it depends on Spring Web Flow and stores data during login in Web Flow storage which in turn depends on the HTTPSession object maintained by the Web Server (Tomcat, JBoss, ...). You can cluster JBoss or Tomcat servers to share HTTPSession objects over the network, but it is simpler if you program the Front End so that if the user responds in a reasonable amount of time, the login form with the userid and password is send back to the Web Server that wrote it to the browser. This is called a "sticky session" by Front Ends, and it just has to apply while the user is typing in Netid and password.
Otherwise, if this is a brand new request to login to CAS or if the CAS Server selected by one of the previous steps has failed and is not responding to the Front End, then send the request to any available CAS server.