...
Of course, these node to node communication calls are all configured into CAS at the same place (using Spring MVC) and in the same way that the /serviceValidate and /samlValidate processing is configured.
Note: Yes, this sort of thing can be done with GSSAPI, but after looking into configuring Certificates or adding Kerberos, it made sense to keep it simple and stick with the solutions that CAS was already using to solve the same sort of problems in other contexts.
Are You Prepared?
Everything that can go wrong will go wrong. We plan for hardware and software failure, network failure, and disaster recovery. To do this we need to know how things will fail and how they will recover from each type of problem.
...
The other CAS clustering techniques (JBoss Cache, Ehcache, Memcached) are typically regarded as magic off the shelf software products that take care of all your problems automatically and you don't have to worry about them. Again you haven't actually solved the problem, but now you really have transformed it into something you will never understand and so you just have to cross your fingers and hope these those guys know what they are doing.
Cushy is simple enough that anyone can figure out how it works and, therefore, the consequences of any type of failure. So now you really can plan for each type of failure and design the system to do the best thing. Cushy was specifically written for the people who feel more comfortable knowing exactly how stuff works, and how it fails.
Why another Clustering Mechanism?
...
In Cushy, the only connection between the CAS mainline function (the part of CAS that responds to Web requests and services users and applications) is that references to objects are occasionally copied from one in memory collection to another. SeparatelySeparate from that, on a timer driven basis collections of objects are periodically written to disk with a single Java writeObject statement. SeparatelySeparate from that, on a network request driven basis, copies of those files are then send to other CAS nodes. If there is a network problem, the HTTP GET fails with an I/O error, this operation is aborted completely, then the servers try again 10 or 15 seconds later. Each step places an absolute boundary between itself and the other steps. None of them can interfere with CAS mainline services. There are no queues or operations to clog and back up into the running code.
Comparison of Cushy and previous cluster technologies:
- The other CAS cluster mechanisms are designed so the CAS servers all share a common pool of tickets. The problem then is to get a copy of a new or modified ticket to all the other servers before they receive a request that requires it. A Cushy cluster is really a connected group of standalone CAS servers. The user may initially login to a randomly chosen server, but once she logs on the Front End routes all requests for that user to that particular server.
- The other CAS cluster mechanisms try to replicate individual tickets. Cushy replicates the ticket registry as a whole, either as a full backup file or as cumulative incremental changes.
- Other mechanisms are driven by individual addTicket or deleteTicket operations. Cushy notes these operations, but it goes to the network on a regular timer driven basis and if an operation fails it retries after the next timer interval.
- For other mechanisms you configure a highly available database or multicast addresses, timeouts, and recovery parameters. Cushy uses HTTP, and you already have that from the Web server CAS is already running in. The only configuration is the URLs of the machines in the cluster.
- A lot of CAS users configure JPA just so they can reboot CAS without losing the tickets. Cushy does that without the database or cluster.
- Cushy is probably With Cushy the only thing you have to configure is the URL of the other node. There are no multicast addresses, no ports, no timeout values, no retry counts. There is no library, no drivers, no database.
- The part of Cushy that creates disk files from tickets or restores tickets from disk files is easy and can't really fail. So what is left is the problem to get a disk file from one machine to another. If you can do that, Cushy is going to work. Cushy does it automatically if you use HTTP, and any problems are easy to diagnose.
- Cushy keeps the tickets for each node separate from the tickets for every other node.
- Cushy is probably less efficient than other technologies, but if it uses less that 1% of one core of a modern server then, given the relative importance of CAS in most institutions, reducing that to a quarter of 1% is not worthwhile if you have to give something up to get the efficiency.
...
This becomes an outline for various cluster node failure tests. Whenever one ticket points to a parent there is a model where the ticket pointed to was created on a node that failed and the new ticket has to be created on the backup server acting on behalf of that node. So you want to test the creation and validation of a Service Ticket on node B when the TGT was created on node A, or the creation of a PGT on node B when the TGT was created on node A, and so on.
Front End Programming
CAS Ticket IDs have four sections:
type - num - random - suffix
where type is "TGT" or "ST", num is a ticket sequence number, random is a large random string like "dmKAsulC6kggRBLyKgVnLcGfyDhNc5DdGKT", and the suffix at the end is configured in the XML.
There are separate XML configurations for different types of tickets, but they all look alike and they all occur in the uniqueIdGenerators.xml file. With cushy the suffix is tied to the TicketSuffix property generated by the CushyClusterConfiguration:
...
Users at Yale know that CAS is "https://secure.its.yale.edu/cas". In reality, DNS resolves secure.its.yale.edu to 130.132.35.49 and that is a Virtual IP address (a VIP) on the BIG-IP F5. There has to be special configuration for that particular IP address. For example, the F5 has to hold the SSL Certificate for "secure.its.yale.edu" and it has to manage the SSL protocol.
CAS is not the only application that appears to be on secure.its.yale.edu. So when a GET or POST arrives at that IP address, the F5 has to look at the URL that follows it. If the URL begins with /cas then it will be handled by one of the configured IP addresses for one of the VMs that run the CAS code. So the F5 is already making decisions based on part of the URL.
Sometimes a new user should be randomly assigned to any available server, but then subsequent requests should go to the same server. This can be done by JSESSIONID if it is a J2EE server. Making decisions based on this is more complicated because the F5 has to maintain tables. It is so common, however, that this is built in logic in the F5 that you can turn on by clicking a checkbox.
The Cushy requirements are simpler and cleaner. Routing is done based on data in the request. The data is either in the URL or in an HTTP Header. The data explicitly identifies the server to which the request should be routed. There must be a bit of special programming because locating the routing information depends on knowing the CAS protocol.
First, however, we need to understand the format of CAS ticketids because that is where the routing information comes from.
CAS Ticket IDs have four sections:
type - num - random - suffix
where type is "TGT" or "ST", num is a ticket sequence number, random is a large random string like "dmKAsulC6kggRBLyKgVnLcGfyDhNc5DdGKT", and the suffix at the end is configured in the uniqueIdGenerators.xml file.
There are separate XML configurations for different types of tickets, but they all look alike and they all occur in the uniqueIdGenerators.xml file. With cushy the suffix is tied to the TicketSuffix property generated by the CushyClusterConfiguration:
<bean id="ticketGrantingTicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator">
<constructor-arg index="0" type="int" value="50" />
<constructor-arg index="1" value="#{clusterConfiguration.getTicketSuffix()}
...
So when Cushy figures out what cluster this computer is in and assigns each node a name, it generates the TicketSuffix value and feeds it to the ticket ID generation logic on each node. In the simplest case, the suffix is just the node name. The F5, however, likes to identity hosts by the MD5 hash of their IP address.
Every CAS request except the initial login comes with one or more tickets located in different places in the request. A modern programmable Front End device like the BIG-IP F5 can be programmed to understand the CAS protocol and to locate the important ticket. There is a sequence of tests and you stop at the first match:
- If the Path part of the URL is a validate request (/cas/validate, /cas/serviceValidate, /cas/proxyValidate, or /cas/samlValidate) then look at the ticket= parameter in the query string part of the URL
- If the Path part of the URL is a /cas/proxy request, then look at the pgt= parameter in the query string.
- If the request has a CASTGC cookie, then look at the cookie value.
- If a request has been seen from this browser in the last 5 minutes, then send it to the same node it was previously sent to.
- Otherwise, or if the node selected by 1-4 is down, choose any CAS node
That is the code, now here is the explanation:
...
" />
</bean>
So when Cushy figures out what cluster this computer is in and assigns each node a name, it generates the TicketSuffix value and feeds it to the ticket ID generation logic on each node. In the simplest case, the suffix is just the node name. The F5, however, likes to identify hosts by the MD5 hash of their IP address.
Every CAS request except the initial login comes with one or more tickets located in different places in the request. There is a sequence of tests and you stop at the first match:
- If the Path part of the URL is a validate request (/cas/validate, /cas/serviceValidate, /cas/proxyValidate, or /cas/samlValidate) then look at the ticket= parameter in the query string part of the URL
- If the Path part of the URL is a /cas/proxy request, then look at the pgt= parameter in the query string.
- If the request has a CASTGC cookie, then look at the cookie value.
- If a request has a JSESSIONID, then the user is in the middle of login and send it back to the same node.
- Otherwise, or if the node selected by 1-4 is down, choose any CAS node
That is the code, now here is the explanation:
- After receiving a Service Ticket ID, an application opens its own HTTPS session to CAS, presents the ticket id in a "validate" request. If the id is valid CAS passes back the Netid, and in certain requests can pass back additional attributes. The suffix on the ticket= parameter identifies the CAS server that created the ticket and has it in memory without requiring any high speed replication.
- When a middleware server like a Portal has obtained a CAS Proxy Granting Ticket, it requests CAS to issue a Service Ticket by making a /proxy call. Since the middleware is not a browser, it does not have a Cookie to hold the PGT. So it passes it explicitly in the pgt= parameter.
- After a user logs in, CAS creates a Login TGT that points to the Netid and attributes and writes the ticket id of the TGT to the browser as a Cookie. The Cookie is scoped to the URL of the CAS application as seen from the browser point of view. At Yale this is "https://secure.its.yale.edu/cas" and so whenever the browser sees a subsequent URL that begins with this string, it appends the CASTGC Cookie with the TGT ID. CAS uses this to find the TGT object and knows that the user has already logged in. This rule sends a browser back to the CAS node the user is logged into.
- If the first three tests fail, this request is not associated with an existing logged in user. CAS has a bug/feature that it depends on Spring Web Flow and stores data during login in Web Flow storage which in turn depends on the HTTPSession object maintained by the Web Server (Tomcat, JBoss, ...). You can cluster JBoss or Tomcat servers to share HTTPSession objects over the network, but it is simpler if you program the Front End so that if the user responds in a reasonable amount of time, the login form with the userid and password is send back to the Web Server that wrote the form it to the browser in response to the browser's original HTTP GET. This is called a "sticky session" and the F5 does it automatically if you just check a box. You don't need to write code.
- Otherwise, if this is a brand new request to login to CAS or if the CAS Server selected by one of the previous steps has failed and is not responding to the Front End, then send the request to any available CAS server.
Lets be clear about what is happening here.
Neither CAS nor the F5 is maintaining a "session" as that term is commonly used. Requests are being routed based on data in the HTTP headers. When an application validates Service Tickets that it receives, every Service Ticket validation request can be routed to a different specific CAS VM based on the suffix of different ticket= parameters. In any normal meaning of "session" requests from the same client would go to the same server. However, the purpose of this logic is to do exactly what sessions normally do, select a specific VM from the pool, it is just that this selection is based on data in the request unrelated to either the identity of the client or any previous traffic from that client.
If you look at the rather silly example from the F5 manual, it is clear that the device is designed to be able to do this sort of thing:
set uri [HTTP::uri]
if { $uri ends_with ".gif" }
{ pool my_pool }
elseif { $uri ends_with ".jpg" }
{ pool your_pool }
CAS does not require the F5 to create any new table. The pool of servers associated with /cas is already part of the F5 configuration. The coding is just to select a specific server in the pool based on data in a field of each request.
The logic depends on the CAS protocol, which is fairly stable and does not change between minor releases.
What Cushy Does at Failure
...
- A Primary server gets all the requests until it fails. Then a Backup "warm spare" server gets requests. If the Primary comes back up relatively quickly, then Cushy will work best if Front End resumes routing all request to the Primary as soon as it becomes available again.
- Users are assigned to login to a CAS Servers Server on a round-robin or load balanced basis. After a user logs in, the suffix on the login, proxy, or service tickets in the URL or headers of an HTTP request route the request to that server.
Each CAS server in the cluster has a shadow object representing the TicketRegistry of each of the other nodes. In normal operation, that object contains no ticket objects. There is no need to extract objects from the other node's files until a failure occurs and a request for one of those tickets arrives. Then Cushy restores the tickets from the file into memory (Just In Time) and processes requests on behalf of the failed node.
However, every new ticket Cushy creates belongs to the node that created it. During a node failure, the new Service Tickets or Proxy Granting Tickets created for users logged into the failed node are created by and belong to the backup node. They each get a ticket ID that has the suffix of the backup node. They live forever in the Ticket Registry of the backup node. They just happen to be associated with and point to a TGT in the secondary shadown registry on the backup node associated with the failed login node.
So while the failed node is down, and even after it comes back up again, requests associated with tickets created by the backup node are routed to the backup node by the Front End. However, after the failed node comes back new requests for new tickets associated with the login TGT will go back to being processed by the original node.
...