...
All three methods of storage work well under normal circumstances, but Shibboleth cannot come up in a disaster recovery situation if the configuration data is not available and options 2 and 3 depend on external servers that may not be up. Although Shibboleth can maintain a local copy of the remote file, it turns out that Shibboleth startup is delayed for an unacceptable amount of time if Subversion is configured but the Subversion server is down.
At So Yale the configuration of Shibboleth is controlled by controls Shibboleth configuration using Subversion, but not by making Shibboleth a direct Subversion SVN client. Instead, the Jenkins Install job is the Subversion client, and after the Install job is run Shibboleth expects its files to be on local disk. Having tested the alternatives, this turns out to be the best option. It allows dynamic update, but under the control of an explicit operation (the Jenkins job) and leaves Shibboleth ready to come up without additional external dependencies.
In addition to the three primary configuration files, Shibboleth has a separate mechanism to load Metadata files. A Metadata source can provide information for a single partner or for a group of partners. The group may be an administered federation like InCommon or just a bunch of related definitions, like the 11 new Tenant instances of Workday.
Each Metadata source can be a local file on disk or an HTTP source (but the Subversion option is not available for Metadata). We define InCommon as an HTTP source (with a local file backup of the last fetched copy of the Metadata), and all the other sources are local files installed by Jenkins from Subversion. Each Metadata source is an independent configuration with its own refresh rules. At Yale, we have decided to use three Metadata source models:
Static - Metadata files installed in Subversion and only changed by a full Jenkins Install. These individual files are not checked for a change in timestamp. Instead, they are all reloaded from disk when the relying-party.xml file that defined them is changed and reloaded into memory.
Remote- The InCommon Metadata is provided from a remote URL on the InCommon Web server. Once every 8 hours Shibboleth checks for a new version and dowloads it from the server. Shibboleth maintains a local disk copy of the last file downloaded, so if Shibboleth is restarted and the remote server is unavailable it will be able to come up with the previous InCommon Metadata.
Dynamic - Individual Metadata files that are checked every 5 minutes for changes and are individually loaded into memory. This is a new option that was added as part of the 1Q15 general Shibboleth code update and reorganization. However, at this time the files designated to be dynamic are initially empty after each release. There is a new Yale Jenkins type of Install operation that changes only these filesShibboleth Install jobs check configuration files out of Subversion and copy them to local disk files on the Shibboleth server. Shibboleth is configured to use option 1 (local disk files and periodically check the modified date), but the file contents are moved from SVN to the local disk through an explicit activity (running the Jenkins Install job) which in turn is driven by Yale Release Policy and Service Now.
Metadata files defining remote Service Provider partners must be either local files or Web URLs. InCommon is a federation with an administrative process for controlling and validating Metadata updates, so that is the only Metadata source that we allow to come from a Web Server, and then we use the Shibboleth option of keeping a local disk copy of the last downloaded InCommon file so Shibboleth can come up when a network connection to InCommon is unavailable. All other Metadata files are managed by us, stored in SVN, and copied to the Shibboleth server local disk by the Jenkins Install job, just like the three primary configuration files.
Metadata files can define a single partner entity, or a list of entities. The list of Metadata sources is configured in the relying-party.xml primary configuration file, and every time that file changes Shibboleth reads in a new copy of all its Metadata sources. Otherwise, individual sources of Metadata can be configured to be periodically checked for an update and new Metadata is read in to replace the old source when the file timestamp changes.
This produces three strategies that Yale uses for Metadata sources:
- Static - Most Yale Metadata files are stored in disk files that are only changed by a full Jenkins configuration install. They are not individually checked for timestamp changes. Instead, when Jenkins Install updates the relying-partly.xml files and changes its timestamp, the new copy of that file is read into memory (even if it didn't really change content) and then all the Metadata files it defines are also refreshed in memory. All these Metadata files refresh together.
- Remote - This option is only used for InCommon. Every 8 hours Shibboleth checks a configured URL to see if the Metadata for InCommon has changed. If so, it downloads a new copy of this information from the remote Web Server, stores a local copy of this file on disk where it can be used as a backup if a network connection to InCommon is not available the next time Shibboleth starts up, and then replaces the in-memory old copy of the InCommon Metadata information.
- Dynamic - Prior to 3/15/2015 Yale did not have any individual Metadata files that could be updated by themselves. In the new configuration, there are three types of individually replaceable Metadata.. Because Shibboleth examines Metadata sources in the order in which they are configured, and it stops when it finds Metadata for the entity for which it is searching, these dynamic Metadata files are distinguished by their position in the search order.
The three types of Dynamic Metadata sources are:
The "emergency-override" dynamic file comes first in the search, so any . Metadata placed in this file overrides an older version configured statically. This file is initially empty and can replace the previous version of production Metadata for any partner. After each new Shibboleth Release, this file is empty. Metadata is placed in it when we have an incident because an existing partner metadata has failed (typically because it has expired or the Certificate and key used by the partner has changed unexpectedly). This provides a safer form of "emergency" fix because only the one Metadata element is logically replaced instead of the entire Shibboleth configuration.
The "additions" dynamic file comes last in the search, so it cannot be used to change any existing Metadata for any entity. It can only define new Meatadata for new entities. This becomes a relatively safe Standard Change because anything put into this file cannot adversely affect existing configured services.A new partner may need more than just Metadata. They may need attributes released to them. Fortunately, Shibboleth allows the function of the attribute-filter.xml file to be broken up into multiple files. Existing partners are configured in attribute-filter, and an empty file named ". A new partner may need more than just Metadata. They may need attributes released to them. Fortunately, Shibboleth allows the function of the attribute-filter.xml file to be broken up into multiple files. Existing partners are configured in attribute-filter, and an empty file named "additional-attribute-filter.xml" is deployed with every Shibboleth Release. Therefore, if a new partner has to be defined to production and cannot wait for the every-other-Thursday Release cycle, the Metadata for that partner can be placed in the metadata/additions.xml file and the attributes to be released can be put in the additional-attribute-filter.xml " is deployed with every Shibboleth Release.
Therefore, if a new partner has to be defined to production and cannot wait for the every-other-Thursday Release cycle, the Metadata for that partner can be placed in the metadata/additions.xml file and the attributes to be released can be put in the additional-attribute-filter.xml file. A Jenkins install of runtype=additions replaces both of these originally empty files with the data for the newly defined partner while guaranteeing by their search order that they cannot interfere with existing services. When the next regularly scheduled Shibboleth Release is ready, the changes move from the additions files to the normal Shibboleth configuration and the additions files are empty again.
If a partner requires a new attribute, however, there is no way to define it outside the every other Thursday system (unless the ECAB authorizes an unscheduled Release)file. A Jenkins install of runtype=additions replaces both of these originally empty files with the data for the newly defined partner while guaranteeing by their search order that they cannot interfere with existing services. When the next regularly scheduled Shibboleth Release is ready, the changes move from the additions files to the normal Shibboleth configuration and the additions files are empty again.
Two of our partners (Salesforce and Cvent) regularly add new AssertionConsumerService URL elements to their existing Metadata file. This happens so frequently that we have the option of replacing these specific production Metadata files with updated copies. There has not yet been any urgency to make such changes outside a normal Release cycle, but we have the ability to respond to the special needs of these two cloud partners if "every other week" becomes an unacceptable delay.
Attribute-Resolver (Queries and Attributes)
...