How Developers Cut a New Release of Shibboleth Configuration

A new release of the Shibboleth IdP can be downloaded from shibboleth.net. It is distributed as a ZIP file.

A Developer unzips this file onto a desktop computer. Then follow the standard install instructions from shibboleth.net to run the build script in the bin subdirectory. Answer the questions giving reasonable answers that match our general configuration (although this is mostly just a sanity check and not an actual requirement). For example, our production EntityId is "https://auth.yale.edu/idp/shibboleth", so that is a good answer to the question about EntityId even though the actual answer does not matter at this point.

The build script creates a shibboleth-idp subdirectory. By default it is in /opt (or c:\opt if you run this on Windows). Zip the contents of this directory (so the zip file contains bin, conf, credentials subdirectories and not a single directory named shibboleth-idp)  up into a file shibboleth-idp-3.3.1.zip (changing the version number to match the release). Someone with upload privileges needs to store a copy of this file in Artifactory. Check for the existing shibboleth-idp artifacts to see where it goes and double check the naming.

The Shibboleth Install job will unzip this file to build an initial /apps/shibboleth-idp directory. It will then update that directory with new configuration files that customize it to the Yale environment (and to DEV, TEST, or PROD data).

However, the Install job does not change the /credentials subdirectory. In DEV, TEST, or PROD environments there is an existing /credentials directory with keys that authenticate Shibboleth to applications configured to use the auth, auth-test, or auth-dev Shibboleth IdPs.  Generally speaking, developers cannot see or copy the contents of these directories on the official Shib VMs, but it is not difficult to get operations to give you a copy of the DEV directory because nobody in the world should rely on the authenticity of DEV SAML. The default names of files in this directory changed from Shibboleth 2 to 3, but we changed the configuration to continue to support the old Shibboleth 2 names, so this is one case where the default contents generated by a new build script will not work (at least until two files are renamed to match our configuration).

Compatibility

The Shibboleth developers try to maintain support for older versions of configuration files. However, when they change the syntax of a configuration file it is typically for a good reason, and if you understand enough about Shibboleth and SAML you will probably want to pick up the improvements they have made.

For example, the upgrade from Shibboleth 2 to 3 introduced a new way to configure the Subject NameID in the SAML response. In Shibboleth 2 the Subject was regarded as a special case of Attribute, but Shibboleth 3 added a new configuration file that specifically controlled Subject values. The old way of configuring them was screwed up and obscure, so it was worth the effort to manually convert each configuration file to conform to the new syntax and the new documentation. This process also gives you a chance to fix our mistakes as well as implement fixes for the Shibboleth mistakes.

Generally the conversion from one major release number to another (from 2 to 3) is a good time to rework configurations, but minor number updates (from 3.2. to 3.3) is done without creating new VMs or extensive testing, so the conversion should initially use the old configuration, then configuration improvements can be made later.

Install Job

The Jenkins Install job parameters generate an "install.properties" file in the Install project. There is also an install-ALL.properties file for properties that are common to all servers in all environments and should not change much, and separate install-XXX.properties files where XXX is replaced by DEV, TEST, PREPROD, or PROD. A rule is that no parameter should be in more than one properties file, but if you break the rule then install.properties always beats the other files, and install-ALL.properties beats the DEV/TEST/PROD properties files.

The structure of the configuration files in the Install project exactly duplicates the subdirectory and file structure of the shibboleth-idp directory created by the Shibboleth installer and saved as the artifact. There are two directories. The files and directories in src/main/installer/resources/updateArtifact are copied exactly as is into the artifact deploy directory (/apps/shibboleth-idp by current convention). The files and directories in src/main/installer/resources/updateArtifactSubstituteVariables are copied but the text files are scanned and variables of the form ${foo.bar} are replaced by the value of the "foo.bar" property from the properties files or Ant script.

The Jenkins Install jobs check out the Install Project files from Source Control using the URL:

• https://svn.its.yale.edu/repos/shibboleth/yale-shibboleth-installer/trunk (for DEV)
• https://svn.its.yale.edu/repos/shibboleth/yale-shibboleth-installer/tags/Release (for TEST or PROD).

(and remember to change the directory name if necessary, for example to yale-shibboleth3-installer if that is the current project).

To work on Shibboleth configuration, use Eclipse to check out the current trunk (from the URL listed above for DEV). Edit changes into the file and commit the changes to the trunk. Run the Jenkins DEV install job and test the changes in the development environment. Generally you continue making changes this way until you need to push something into TEST or PROD, which you try to put off until it is necessary.

To install into TEST or PROD, you have to cut a new Release tag. However, you need to save the previous Release tag as your "rollback" if something goes wrong. So use Eclipse SVN Repository exploring to Refactor - Rename the old Release tag to something else. Then use trunk to create a new Release tag.

If something goes wrong with the new install, rename the /tags/Release to something else for later diagnosis, restore the previous saved tag to the "Release" name, and have the PROD install job rerun.

%{foo.bar}, ${foo.bar}, <property name="foo.bar">, foo.bar=

In the three properties files (install.properties from Jenkins parameters, install-ALL.properties, install-XXX.properties)  a property is given a value by a line of for the form:

foo.bar=/var/log/shibboleth-idp

Once a property is set like this, it cannot be changed in Ant. However, if a property has not be set in any properties file then in the build.xml script executed by Ant then a default value can be provided for properties not set by the files with an Ant statement:

<property name="foo.bar" value="/var/log/shibboleth-idp" />

Properties set either way will be used to replace references of the form "${foo.bar}" when they appear in any file in the src/main/installer/resources/updateArtifactSubstituteVariables directory tree. In particular, if you want the same property to be available during Spring intialization you have to create a line in updateArtifactSubstituteVariables/conf/idp.properties of the form

foo.bar=${foo.bar}

Assuming that foo.bar was assigned a value for the Install job, either in a properties file or a <property> Ant statement, then when the idp.properties file is copied with substitution then Ant will see the "${foo.bar}" after the equals and will substitute the property value of "/var/log/shibboleth-idp". So while the statement above will appear in the Install project, after the Install is done if you go to the artifact deploy directory on the server you will find that the actual content of the /apps/shibboleth-idp/conf/idp.properties file is

foo.bar=/var/log/shibboleth-idp

Yes this is exactly what was in the original properties file, but that is important because while the three install-?.properties files were read in during Jenkins install job processing, now the /apps/shibboleth-idp/conf/idp.properties file is read in during Spring initialization on the Shibboleth server.

All the Shibboleth configuration files can contain variable references that will be replaced by Spring. Spring calls this "EL" or execution language, and it is much more powerful than just variable substitution but the extra features don't matter to us now. Shibboleth already makes extensive use of EL in its own configuration. In Spring EL, the variables are marked by a %{foo.bar} instead of a $.

So if you are coding a new Spring bean or another configuration item that you think needs variable substitution, and you want to do it the way Shib handles all its parameters, then you

• Add a property to one of the Install job properties files.
• Add a new line in the idp.properties file that sets the Spring runtime property to its Install time value (foo.bar=${foo.bar}).
• Reference the run time property in the configuration file using % instead of $ (logdirectory="%{foo.bar}")

Of course, you could decide not to do things the way Shib does them, you could move the configuration file to the updateArtifactSubstituteVariables directory tree and then just reference the property as ${foo.bar}. Then the variable will be replaced by the value at install time and there will be no runtime substitution. It should work the same, but it is not the Shib way to do it.