This is a note to any IDM staff who become responsible for making Shibboleth configuration changes. This reflects the current practice that uses just the /trunk and /tags/Release entries in the SVN repository for the yale-shibboleth-installer. During a transition between releases, new code may be in a slightly different named directory (yale-shibboleth3-installer during the transition from Shib 2 to Shib 3).
Starting with Shibboleth 3, the artifact stored in Artifactory is created by downloading the standard Shibboleth distribution zip file, running the Install script inside that file on a test machine, creating a more or less standard shibboleth-idp directory, and then zipping up the generated directory. This zip artifact is then manually uploaded to Artifactory as shibboleth-idp-3.2.1. No Jenkins Build job is run, and any Jenkins Build jobs for Shibboleth are now regarded as obsolete.
A "Release" of Shibboleth is, therefore, a new set of configuration files stored in the Jenkins Install project under Source Control and has nothing to do with the artifact. A new artifact is generated when we get a new release of code from shibboleth.net. Since we make no modifications to Shibboleth itself, it should be possible to migrate to a new release by downloading a new distribution and generating a new artifact, then switching the artifact.version in the Jenkins Install job parameters.A new release of the Shibboleth IdP can be downloaded from shibboleth.net. It is distributed as a ZIP file.
A Developer unzips this file onto a desktop computer. Then follow the standard install instructions from shibboleth.net to run the build script in the bin subdirectory. Answer the questions giving reasonable answers that match our general configuration (although this is mostly just a sanity check and not an actual requirement). For example, our production EntityId is "https://auth.yale.edu/idp/shibboleth", so that is a good answer to the question about EntityId even though the actual answer does not matter at this point.
The build script creates a shibboleth-idp subdirectory. By default it is in /opt (or c:\opt if you run this on Windows). Zip the contents of this directory (so the zip file contains bin, conf, credentials subdirectories and not a single directory named shibboleth-idp) up into a file shibboleth-idp-3.3.1.zip (changing the version number to match the release). Someone with upload privileges needs to store a copy of this file in Artifactory. Check for the existing shibboleth-idp artifacts to see where it goes and double check the naming.
The Shibboleth Install job will unzip this file to build an initial /apps/shibboleth-idp directory. It will then update that directory with new configuration files that customize it to the Yale environment (and to DEV, TEST, or PROD data).
However, the Install job does not change the /credentials subdirectory. In DEV, TEST, or PROD environments there is an existing /credentials directory with keys that authenticate Shibboleth to applications configured to use the auth, auth-test, or auth-dev Shibboleth IdPs. Generally speaking, developers cannot see or copy the contents of these directories on the official Shib VMs, but it is not difficult to get operations to give you a copy of the DEV directory because nobody in the world should rely on the authenticity of DEV SAML. The default names of files in this directory changed from Shibboleth 2 to 3, but we changed the configuration to continue to support the old Shibboleth 2 names, so this is one case where the default contents generated by a new build script will not work (at least until two files are renamed to match our configuration).
Compatibility
The Shibboleth developers try to maintain support for older versions of configuration files. However, when they change the syntax of a configuration file it is typically for a good reason, and if you understand enough about Shibboleth and SAML you will probably want to pick up the improvements they have made.
For example, the upgrade from Shibboleth 2 to 3 introduced a new way to configure the Subject NameID in the SAML response. In Shibboleth 2 the Subject was regarded as a special case of Attribute, but Shibboleth 3 added a new configuration file that specifically controlled Subject values. The old way of configuring them was screwed up and obscure, so it was worth the effort to manually convert each configuration file to conform to the new syntax and the new documentation. This process also gives you a chance to fix our mistakes as well as implement fixes for the Shibboleth mistakes.
Generally the conversion from one major release number to another (from 2 to 3) is a good time to rework configurations, but minor number updates (from 3.2. to 3.3) is done without creating new VMs or extensive testing, so the conversion should initially use the old configuration, then configuration improvements can be made later.
Install Job
The Jenkins Install job parameters generate an "install.properties" file in the Install project. There is also an install-ALL.properties file for properties that are common to all servers in all environments and should not change much, and separate install-XXX.properties files where XXX is replaced by DEV, TEST, PREPROD, or PROD. A rule is that no parameter should be in more than one properties file, but if you break the rule then install.properties always beats the other files, and install-ALL.properties beats the DEV/TEST/PROD properties files.
...