...
- On the local disk of the Shibboleth server VM
- On a Web server of some sort (referenced by a URL).
- On a Subversion server
If they are local disk files then Shibboleth can check their modify date and load a new configuration when the file is updated. If they are on a Web server, then HTTP can be used to check to see Shibboleth reads the files and creates in memory objects at startup time. After that, it can be configured to check each file periodically and to reload the file and replace that part of the configuration if the file is changed. The Subversion access is driven by the version number increasing instead of the date.
The Shibboleth documentation recommends SVN, but because Shibboleth is a Tier 0 application it has to be able to come up with as few prerequisites as possible and the SVN support does not seem to work if the Subversion server has not yet been restored.
Yale Shibboleth configuration is driven by Subversion, but Shibboleth is not the SVN client and it doesn't download files itself. Instead, we control when the files become active through the Jenkins Install job. Jenkins checks out configuration files from SVN and then copies the files to local disk on the Shibboleth servers.Shibboleth is configured to use local disk files, and we have an administratively controlled and logged mechanism to update them. Once installed, the files remain available should Shibboleth be restarted in any Disaster Recovery scenario.
We have configured Shibboleth to check every 30 seconds for a new change timestamp on any local configuration file. When it sees a new version of the file it reads the contents into memory and runs a minimal XML parse. If there is an XML syntax error in the file, it is discarded and the old configuration remains active. Otherwise, once the file has been successfully read then the new configuration replaces the previous configuration.
Metadata is a bit more complicated. Metadata sources are configured in the relying-party.xml file. Each Metadata source is an For local disk files, the last modified date is checked. For HTTP files, an "if-modified-since" check is made. For Subversion, modification is detected when a new version number is checked in.
All three methods of storage work well under normal circumstances, but Shibboleth cannot come up in a disaster recovery situation if the configuration data is not available and options 2 and 3 depend on external servers that may not be up. Although Shibboleth can maintain a local copy of the remote file, it turns out that Shibboleth startup is delayed for an unacceptable amount of time if Subversion is configured but the Subversion server is down.
At Yale the configuration of Shibboleth is controlled by Subversion, but not by making Shibboleth a direct Subversion client. Instead, the Jenkins Install job is the Subversion client, and after the Install job is run Shibboleth expects its files to be on local disk. Having tested the alternatives, this turns out to be the best option. It allows dynamic update, but under the control of an explicit operation (the Jenkins job) and leaves Shibboleth ready to come up without additional external dependencies.
In addition to the three primary configuration files, Shibboleth has a separate mechanism to load Metadata files. A Metadata source can provide information for a single partner or for a group of partners. The group may be an administered federation like InCommon or just a bunch of related definitions, like the 11 new Tenant instances of Workday.
Each Metadata source can be a local file on disk or an HTTP source (but the Subversion option is not available for Metadata). We define InCommon as an HTTP source (with a local file backup of the last fetched copy of the Metadata), and all the other sources are local files installed by Jenkins from Subversion. Each Metadata source is an independent configuration with its own refresh rules. At Yale, we have decided to use three Metadata source models:
Static - Production Metadata for partners that have supplied us with Metadata that we check into SVN and manage directly are handled as individual static files. They are copied from SVN to the local hard drive of the Shibboleth server, but they have no refresh policy. You cannot change a Static Metadata file by itself. You have to change the timestamp on the relying-party.xml file, and when it gets read into memory then Shibboleth automatically reloads all the Metadata files that the relying-party.xml file designatesMetadata files installed in Subversion and only changed by a full Jenkins Install. These individual files are not checked for a change in timestamp. Instead, they are all reloaded from disk when the relying-party.xml file that defined them is changed and reloaded into memory.
Remote- The InCommon Metadata is provided from a remote URL on the InCommon Web server. Once every 8 hours Shibboleth checks for a new version and dowloads it from the server. Shibboleth maintains a local disk copy of the last file downloaded, so if Shibboleth is restarted and the remote server is unavailable it will be able to come up with the previous InCommon Metadata.
Dynamic - Specific Individual Metadata files are stored as local files on disk, but they are configured to be examined once every 5 minutes for a changed timestamp and to be reloaded when they change. Because Shibboleth examines Metadata sources in the order in that are checked every 5 minutes for changes and are individually loaded into memory. This is a new option that was added as part of the 1Q15 general Shibboleth code update and reorganization. However, at this time the files designated to be dynamic are initially empty after each release. There is a new Jenkins type of Install operation that changes only these files. Because Shibboleth examines Metadata sources in the order in which they are configured, and it stops when it finds Metadata for the entity for which it is searching, these dynamic Metadata files are distinguished by their position in the search order.
The "emergency-override" dynamic file comes first in the search, so any Metadata placed in this file overrides an older version configured statically. This file is initially empty and Metadata is placed in it when we have an incident because an existing partner metadata has failed (typically because it has expired or the Certificate and key used by the partner has changed unexpectedly). This provides a safer form of "emergency" fix because only the one Metadata element is replaced instead of the entire Shibboleth configuration.
The "additions" dynamic file comes last in the search, so it cannot be used to change any existing Metadata for any entity. It can only define new Meatadata for new entities. This becomes a relatively safe Standard Change because anything put into this file cannot adversely affect existing configured services.
A new partner may need more than just Metadata. They may need attributes released to them. Fortunately, Shibboleth allows the function of the attribute-filter.xml file to be broken up into multiple files. Existing parteners partners are configured in attribute-filter, and an empty file named "additional-attribute-filter.xml" is initially deployed with every Shibboleth releaseRelease.
Therefore, if a new partner has to be defined to production and cannot wait for the every-other-Thursday Release cycle, the Metadata for that partner can be placed in the metadata/additions.xml file and the attributes to be released can be put in the additonaladditional-attribute-filter.xml file. The two files are updated together. At a normal Release point, information is moved out of the "additions" files and becomes part of the standard configuration files, and the empty additions files are deployed to start the next cycleA Jenkins install of runtype=additions replaces both of these originally empty files with the data for the newly defined partner while guaranteeing by their search order that they cannot interfere with existing services. When the next regularly scheduled Shibboleth Release is ready, the changes move from the additions files to the normal Shibboleth configuration and the additions files are empty again.
If a partner requires a new attribute, however, there is no way to define it outside the every other Thursday system (unless the ECAB authorizes an unscheduled Release).
...
Then the second element of the strategy is to provide a more accurate and complete testing strategy. Currently TEST Shibboleth is connected to the TEST database instances (ACS2, IST2, IDM2, HOP4) and potentially to the TEST AD (yu.yale.net). This provides a service for those who need to use test netids, but it does not actually test what is going to go into production.
It is also true that most partners do not support TEST environments. In fact, the entire InCommon Federation has no concept of TEST and no provision for us to define our TEST Shibboleth.
However, while CAS is bound to a particular well known URL (secure.its.yale.edu/cas), Shibboleth is actually not bound to a URL or server but rather is known by the Public/Private key pair stored in its /usr/local/shibboleth-idp/credentials folder. Create a second instance of Shibboleth running on any server anywhere in the organization and give it a copy of the same credentials files and it will generate a SAML message that will be accepted as legitimate by any of our partners. While applications talk to CAS directly, all communication between Shibboleth and any application goes through the Browser. So if there is a PRE-PROD test environment with a copy of the code we propose to put into production and a copy of the Production credentials, then a Browser on a machine can use it with all the standard production apps by the obvious brute force solution of pointing the hosts file on the Browser client machine to the PRE-PROD VIP whenever the browser is redirected to "auth.yale.edu". The first time it may be necessary to approve the SSL Certificate name mismatch, but after that you have a platform to comprehensively test the exact configuration we intend to put into productionTEST Shibboleth is connected to the TEST database instances (ACS2, IST2, IDM2, HOP4) and potentially to the TEST AD (yu.yale.net). This provides a service for those who need to use test netids, but it does not actually test what is going to go into production. So a new PREPROD instance has been created that uses the production databases and generates SAML signed with the production credentials, but which runs on a separate machine and can be tested prior to installing the changes into real PROD.