...
The three primary files can be stored on :
- On the local disk
...
Although the Subversion server option is typically recommended, because it provides control and history and an easy fallback option, Shibboleth and CAS are "Tier 0" applications that have to be up first in a disaster recovery situation and Shibboleth does not initialize if a primary configuration file is associated with a Subversion server that is not yet restored to operation. Our current system ultimately stores the authoritative version of the three primary configuration files in Subversion, but instead of Shibboleth checking the files out itself, new copies of the files are copied to the Shibboleth local disk when a system administrator runs the Jenkins Shibboleth Install job with the "config" option.
...
- of the Shibboleth server VM
- On a Web server of some sort (referenced by a URL).
- On a Subversion server
If they are local disk files then Shibboleth can check their modify date and load a new configuration when the file is updated. If they are on a Web server, then HTTP can be used to check to see if the file is changed. The Subversion access is driven by the version number increasing instead of the date.
The Shibboleth documentation recommends SVN, but because Shibboleth is a Tier 0 application it has to be able to come up with as few prerequisites as possible and the SVN support does not seem to work if the Subversion server has not yet been restored.
Yale Shibboleth configuration is driven by Subversion, but Shibboleth is not the SVN client and it doesn't download files itself. Instead, we control when the files become active through the Jenkins Install job. Jenkins checks out configuration files from SVN and then copies the files to local disk on the Shibboleth servers.Shibboleth is configured to use local disk files, and we have an administratively controlled and logged mechanism to update them. Once installed, the files remain available should Shibboleth be restarted in any Disaster Recovery scenario.
We have configured Shibboleth to check every 30 seconds for a new copy of change timestamp on any local configuration file. When it sees a new version of the file it reads the contents into memory and runs a minimal XML parse. If there is an obvious XML syntax problem and error in the file is unusable, it is ignoreddiscarded and the old configuration remains active. Otherwise, after it once the file has been processed successfully read then the new configuration from the new file replaces the old configuration. The "config" version of the Jenkins Install job checks out the current version of all the configuration files from Subversion. Then an Ant script copies these new copies of the files into the active Shibboleth directories. Within 30 seconds the running Shibboleth sees that all the files have been changed and it reads them into memory.
The only problem is that if you replace all the configuration files at once, a mistake can cause Shibboleth to stop working correctly. The proposed strategy is to design smaller subsets of the "config" Jenkins install that only replace specific configuration files, and to rank the level of risk for each type of change so that the level of approval and processing is appropriate for the change being made. We also need to expand the testing options so that every change can actually be tested in its real environment before it is put into production.
There are a number of ways this can be accomplished. However, to understand the level of risk (and in some cases, the complete absence of any risk) you need to know a bit more about the configuration files.
Attribute-Resolver (Queries and Attributes)
The attribute-resolver.xml file begins with a series of database or LDAP queries. Currently Shibboleth pulls data from Oracle instances (ACS, IST, IDM, HOP), the IDR SQL Server database, the Public LDAP Directory, and the Windows AD LDAP directory.
If any query fails, then Shibboleth stops working unless each query is configured with a "Failover", which can either be another query to another database or it can be a static default value (which can be null) for the returned information. We have learned from the datacenter power failures and other incidents the importance of having an ultimate static default value for everything to keep Shibboleth behaving properly under all conditions. Failures will occur if the instance is down, but they can also be caused by network problems or by changes to the security configuration of the userid that Shibboleth uses to access a database. In one incident, Shibboleth worked in DEV and TEST but then failed in production because the userid it logged into the database with had been granted access to a table in DEV and TEST but did not have that permission in the PROD database instance. This explains why it is important to add a testing stage that EXACTLY duplicates production with the same userids and the same production data because TEST does not guarantee exact fidelity.previous configuration.
Metadata is a bit more complicated. Metadata sources are configured in the relying-party.xml file. Each Metadata source is an independent configuration with its own refresh rules. At Yale, we have decided to use three Metadata source models:
Static - Production Metadata for partners that have supplied us with Metadata that we check into SVN and manage directly are handled as individual static files. They are copied from SVN to the local hard drive of the Shibboleth server, but they have no refresh policy. You cannot change a Static Metadata file by itself. You have to change the timestamp on the relying-party.xml file, and when it gets read into memory then Shibboleth automatically reloads all the Metadata files that the relying-party.xml file designates.
Remote- The InCommon Metadata is provided from a remote URL on the InCommon Web server. Once every 8 hours Shibboleth checks for a new version and dowloads it from the server. Shibboleth maintains a local disk copy of the last file downloaded, so if Shibboleth is restarted and the remote server is unavailable it will be able to come up with the previous InCommon Metadata.
Dynamic - Specific Metadata files are stored as local files on disk, but they are configured to be examined once every 5 minutes for a changed timestamp and to be reloaded when they change. Because Shibboleth examines Metadata sources in the order in which they are configured, and it stops when it finds Metadata for the entity for which it is searching, these dynamic Metadata files are distinguished by their position in the search order.
The "emergency-override" dynamic file comes first in the search, so any Metadata placed in this file overrides an older version configured statically. This file is initially empty and Metadata is placed in it when we have an incident because an existing partner metadata has failed (typically because it has expired or the Certificate and key used by the partner has changed unexpectedly). This provides a safer form of "emergency" fix because only the one Metadata element is replaced.
The "additions" dynamic file comes last in the search, so it cannot be used to change any existing Metadata for any entity. It can only define new Meatadata for new entities. This becomes a relatively safe Standard Change because anything put into this file cannot adversely affect existing configured services.
A new partner may need more than just Metadata. They may need attributes released to them. Fortunately, Shibboleth allows the function of the attribute-filter.xml file to be broken up into multiple files. Existing parteners are configured in attribute-filter, and an empty file named additional-attribute-filter.xml is initially deployed with every Shibboleth release.
Therefore, if a new partner has to be defined to production and cannot wait for the every-other-Thursday Release cycle, the Metadata for that partner can be placed in the metadata/additions.xml file and the attributes to be released can be put in the additonal-attribute-filter.xml file. The two files are updated together. At a normal Release point, information is moved out of the "additions" files and becomes part of the standard configuration files, and the empty additions files are deployed to start the next cycle.
If a partner requires a new attribute, however, there is no way to define it outside the every other Thursday system (unless the ECAB authorizes an unscheduled Release).
Attribute-Resolver (Queries and Attributes)
Attributes are defined and their values are obtained from the configuration in the attribute-resolver.xml file.
The file starts with DataConnectors. A typical connector identifies a database or LDAP directory as a source, and a query (in SQL or LDAP query language) to present to the source. Currently Shibboleth pulls data from Oracle instances (ACS, IST, IDM, HOP), the IDR SQL Server database, the Public LDAP Directory, and the Windows AD LDAP directory.
There are generally three types of queries that make sense:
- A database query can return exactly one row. Then you can think of the row as a user object, and the column names become the properties of the object. All these properties are Strings (Shibboleth does not have a concept of other data types). NOTE: Oracle always returns column names in UPPERCASE and it is a really, really good idea to always capitalize these names in the configuration file whenever they appear. If you fail to use all caps in one of the later configuration elements, it will fail to match and then you get a null instead of a value.
- A database query can return more than one row but only one column. Then you have a "multivalued" property.
- An LDAP query returns the User object from the directory. LDAP User objects have properties some of which are single valued and some of which are multivalued.
One other point. The database query can return no rows, or it can return a row where some of the columns have the value NULL. An LDAP query can return no object, or it can return an object that does not have the property name you are interested in, or the property can exist but have no value. For the most part this doesn't matter unless you have to reference the value of the property in JavaScript. If you write JavaScript, remember to test for the property being "undefined", or null, or empty, which are distinct conditions with distinct tests.
Returning no rows or objects is a normal response to a query. A query fails if it generates an ORAxxxx SQLException or a NamingException. Typically this happens if the database server or directory is down, but it can also happen if the userid and password you are using to login to the server is no longer valid or if permissions have been revoked or were never granted to that user.
Shibboleth regards any query failure as catastrophic, unless there is a "failover" DataConnector. The Failover can be a different query to a different database, or it can be a Static default value.
A Static DataConnector defines one or more properties and values. It is not necessary to define a default value for every property that you could have obtained from the correct execution of the real query, provided that a null or undefined value is acceptable for the other properties. Typically Static default values are -1, 0, "", "undefined" but that is up to you. Because query failure is catastrophic without a Fallback
Every Query must have a Static Fallback DataConnector. It is not important what default values, if any, are provided.
In the new release, the attribute-resolver file has been reorganized to emphasize the Failover relationship, and as part of the testing of the new release we will verify that Shibboleth survives the loss of access to each data source. However, it becomes an ongoing process to ensure that every time a new query is defined, a static Failover is also created and Shib is tested for that failure.
...