...
Cushy stores tickets in an in-memory table. It writes tickets to a disk file with a single writeObject Java statement. It transfers files from machine to machine using an HTTPS GET. So far, everything seems to be rather simple. Cushy started that way, but then it became clear that there were a small number of optimizations that really needed to be made even if they added a slight amount of complexity to the code.
Notify
Once every 5-15 minutes a node generates a new full checkpoint file. It also generates a new dummy ServiceTicketId that acts as the password that other nodes will present to request the files over HTTPS. It then does a "Notify" operation. It generates a HTTPS GET to the /cas/cluster/notify URL on every other CAS node in the cluster. This request is routed by Spring MVC to the CacheNotifyController class provided by the Cushy package. A node also does a Notify immediately after it reboots to inform the other nodes that it is back up and to provide them with the password needed to communicate until the next checkpoint.
...
When using "Tickets on Request", there are two basic rules. First, you don't have complete control unless you are synchronized on the Secondary Registry object that corresponds to that node and set of files. Secondly, in order to work in both HTTPS and SharedDisk mode, the processing is coordinated by the modified date on the files. When a file is turned into object in memory, then the objects have the same "modified date" as the file that created or updated them. When the file modified date is later than the objects modified date, then the objects in memory are stale and the file should be restored at the next request.
Sanity check: In a real Shared Disk mode the timestamps on the files are set by the file system, either of the file server or the local disk during HTTP processing (when the /cas/cluster/getCheckpoint or /cas/cluster/getIncremental operation completes). In either case they are set by the same clock. The typical 10 second interval between events (and even a much smaller interval) is much larger than the clock resolution. The important thing here is that we are always comparing one file timestamp with another file timestamp from the same source. This part of the code never uses a timestamp from the local System, so we don't have to worry if clocks are out of sync across systems.
Generally an incremental file if it exists should always be later than a checkpoint. If both files are later than the objects in memory, always restore the checkpoint first.
...