General Shibboleth Configuration
Shibboleth is configured with three primary files (attribute-resolver.xml, attribute-filter.xml, and relying-party.xml) and a number of "metadata" files.
The three primary files can be stored:
- On the local disk of the Shibboleth server VM
- On a Web server of some sort (referenced by a URL).
- On a Subversion server
Shibboleth reads the files and creates in memory objects at startup time. After that, it can be configured to check each file periodically and to reload the file and replace that part of the configuration if the file is changed. For local disk files, the last modified date is checked. For HTTP files, an "if-modified-since" check is made. For Subversion, modification is detected when a new version number is checked in.
All three methods of storage work well under normal circumstances, but Shibboleth cannot come up in a disaster recovery situation if the configuration data is not available and options 2 and 3 depend on external servers that may not be up. Although Shibboleth can maintain a local copy of the remote file, it turns out that Shibboleth startup is delayed for an unacceptable amount of time if Subversion is configured but the Subversion server is down.
So Yale controls Shibboleth configuration using Subversion, but not by making Shibboleth a direct SVN client. Instead, the Jenkins Shibboleth Install jobs check configuration files out of Subversion and copy them to local disk files on the Shibboleth server. Shibboleth is configured to use option 1 (local disk files and periodically check the modified date), but the file contents are moved from SVN to the local disk through an explicit activity (running the Jenkins Install job) which in turn is driven by Yale Release Policy and Service Now.
Metadata files defining remote Service Provider partners must be either local files or Web URLs. InCommon is a federation with an administrative process for controlling and validating Metadata updates, so that is the only Metadata source that we allow to come from a Web Server, and then we use the Shibboleth option of keeping a local disk copy of the last downloaded InCommon file so Shibboleth can come up when a network connection to InCommon is unavailable. All other Metadata files are managed by us, stored in SVN, and copied to the Shibboleth server local disk by the Jenkins Install job, just like the three primary configuration files.
Metadata files can define a single partner entity, or a list of entities. The list of Metadata sources is configured in the relying-party.xml (Shib 2) or metadata-providers.xml (Shib 3) primary configuration file, and every time that file changes Shibboleth reads in a new copy of all its Metadata sources. Otherwise, individual sources of Metadata can be configured to be periodically checked for an update and new Metadata is read in to replace the old source when the file timestamp changes.
This produces three strategies that Yale uses for Metadata sources:
- Static - Most Yale Metadata files are stored in disk files that are only changed by a full Jenkins configuration install. They are not individually checked for timestamp changes. Instead, when Jenkins Install updates the relying-partly.xml files and changes its timestamp, the new copy of that file is read into memory (even if it didn't really change content) and then all the Metadata files it defines are also refreshed in memory. All these Metadata files refresh together.
- Remote - This option is only used for InCommon. Every 8 hours Shibboleth checks a configured URL to see if the Metadata for InCommon has changed. If so, it downloads a new copy of this information from the remote Web Server, stores a local copy of this file on disk where it can be used as a backup if a network connection to InCommon is not available the next time Shibboleth starts up, and then replaces the in-memory old copy of the InCommon Metadata information.
- Dynamic - Specific Metadata file sources are configured to be checked periodically for changes and to reload. There is an initially empty metadata file at the start of the search order, one at the end, and two files for frequently changed metadata have been configured to check every minute for changes. However, we can add the parameter to poll and refresh data to any other Metadata file.
Because the search for Metadata stops at the first file (provider) that contains a matching EntityID, there is a strategy for using dynamic metadata files reflected in the runtype parameter on the Jenkins job.3 is a Spring Framework application. Spring provides a generic configuration syntax that can be used by almost any application. It is not reasonable here to try and describe Spring configuration syntax in general. Go to the Spring website for that information.
Every real Spring file contains a <beans> element. In that file there are a sequence of either <bean> elements or else aliases for a particular kind of standard bean. For example, a List bean based on a Java list of objects is a <util:list> element which is translated into into a particular type of <bean>. Every bean has an id= name by which it is referenced.
Each <bean> is an instance of a Java class, and it is configured by providing values for named properties and occasionally a set of constructor arguments. When many beans will all be created with many common arguments, then an incomplete (abstract) <bean> definition is provided with the common arguments, and then other <bean> elements complete the configuration by referencing the partially configured bean with parent= and then add the missing properties to complete the configuration.
Shibboleth has to work with standard SAML configuration files, and they are ordinary XML files with no Spring elements. In general, Spring calls a text, properties, or xml file a "Resource" and the resource can be defined by a path to a disk file, or a URL to a file on a Web server. The Shibboleth configuration files that are standard XML files are a Resource that has to be read and understood by some Bean, which is different from a Spring XML file that contains a <beans> element.
In the Shibboleth conf subdirectory, the attribute-filter.xml and attribute-resolver.xml files are ordinary XML resources while the other files (services.xml, relying-party.xml, metadata-providers.xml) are Spring beans files. Of course, internally the elements in the non-Spring XML files are turned into objects, but that is done by Shibboleth processing and not just standard Spring.
Most Shibboleth configuration files can be sourced from local disk, from a Web server, or from a Subversion repository. However, Shibboleth is a "Tier 0" application that has to come up after any type of power or datacenter failure before anything else comes up, so it cannot depend on external servers. We solve this problem with a Jenkins Install job that checks files out from Subversion but then manually deposits them on the Shibboleth server in the conf subdirectory. We manage the configuration files with Source Control, but they have to be manually transferred to local disk under operations control. Once stored they are available locally when the Shib server boots.
The files in the conf directory are defined by Shibboleth and Yale. There is a separate metadata directory that defines our individual Relying Parties (google.com, yale.service-now.com, salesforce.com, etc.). However, in order for a metadata file to be recognized at all, there has to be an entry pointing to it in the metadata-providers.xml file.
The metadata-provides file defines sources of metadata. Each source is read into memory when Shibboleth starts up, and based on the configuration of the element in metadata-providers it can be "polled" in regular time intervals to see if a new copy of the file is available.
The InCommon federation curates submitted Metadata for hundreds of potential partners. We define a metadata provider for InCommon that reads the latest version of InCommon metadata from the URL they provide. This is cached to a local copy of the file on disk so that after a datacenter failure Shibboleth can come up using the cached copy of the InCommon metadata in the event that network connectivity to the InCommon server has not be restored. There is special logic in the Install job to not delete the cached InCommon metadata file when all the other metadata files are being deleted and replaced.
However, InCommon is the only dynamic metadata file, and we only trust InCommon as a dynamically updated source because the contents are curated. For the other metadata, particularly for critical business functions, each metadata file is managed as a local disk file updated when necessary by the Jenkins Install process.
Shibboleth processes a metadata-provider as a unit. If it is a file on disk, then the file is read and a single XML syntax prevents Shibboleth from using anything in the file. A metadata file can contain a single EntityDescriptor or a list of them. For safety, we maintain separate files and therefore separate metadata-provider definitions for each cloud partner or application. That way we know that a change given to us by Community Force cannot interfere with our configuration of Comcast IPTV, because they are in separate files and a syntax error in either is localized to that one file.
Shibboleth starts with the EntityID of the partner. For example, one of the Comcast application entities is entityID="https://university-dev.ccp.xcal.tv:443/openam". Shib then goes to the list of metadata providers in the xml file and searches the in memory list of elements constructed from that metadata provider for a matching
<EntityDescriptor entityID="https://university-dev.ccp.xcal.tv:443/openam" ...>
It stops when it finds the first matching EntityDescriptor in any metadata provider. Therefore, if the same entityID is in more than one metadata provider source, Shibboleth will use the source that comes first in the list of metadata providers. We use this to provide a maintenance strategy based on a few local-file metadata provider elements that are configured to regularly check for file changes:
The "emergency-override" dynamic file comes first in the search. Metadata placed in this file with an EntityID that matches an existing Metadata entry in a later file will logically replace the previous version of production Metadata for any partner. After each new Shibboleth Release, this file is empty. Metadata can be placed in it when we have an incident because an existing partner metadata has failed (typically because it has expired or the Certificate and key used by the partner has changed unexpectedly). This provides a safer form of "emergency" fix because only the one Metadata element is logically replaced instead of the entire Shibboleth configuration.
...
Runtype "salesforce" and "cvent" are proposed runtypes that change a single Metadata file for the two partners that require frequent updates.
Contents of the Primary Configuration Files
Attribute-Resolver (Queries and Attributes)
Attributes are defined and their values are obtained from the configuration in the attribute-resolver.xml file.
The file starts with DataConnectors. A typical connector identifies a database or LDAP directory as a source, and a query (in SQL or LDAP query language) to present to the source. Currently Shibboleth pulls data from Oracle instances, the IDR SQL Server database, and the Windows AD LDAP directory.Metadata file for the two partners that require frequent updates.
Contents of the Primary Configuration Files
Attribute-Resolver
Shibboleth normally has a single attribute-resolver.xml file, but Yale changed this.
The attribute-provider file has two types of elements. DataConnectors define database or LDAP queries that produce result sets with columns or LDAP User objects with properties. AttributeDefinitions then embed a value of one column or property returned by one of these queries in a SAML attribute statement with a name, friendlyName, and format.
Normally there is one file with the two different types of elements, but that produces more complicated XML because everything has to be decorated xml namespace prefixes. It just seems cleaner to use the capability provided in Shibboleth 3 to create two files, where attribute-resolver-connectors has the DataConnectors and attribute-resolver-definitions has all the AttributeDefinitions. If you don't like this, you can combine the two back together again and then add back all the xml namespace prefixes that can be defaulted if the files are separate.
DataConnectors
There are generally three types of queries that make sense:
...