Content Comparison

This is a note to any IDM staff who become responsible for making Shibboleth configuration changes. This reflects the current practice that uses just the /trunk and /tags/Release entries in the SVN repository for the yale-shibboleth-installer. All other directories and tags are obsolete and will be cleaned up sometime when we are sure nobody cares about the history.A new During a transition between releases, new code may be in a slightly different named directory (yale-shibboleth3-installer during the transition from Shib 2 to Shib 3).

Starting with Shibboleth 3, the artifact stored in Artifactory is created by downloading the standard Shibboleth distribution zip file, running the Install script inside that file on a test machine, creating a more or less standard shibboleth-idp directory, and then zipping up the generated directory. This zip artifact is then manually uploaded to Artifactory as shibboleth-idp-3.2.1. No Jenkins Build job is run, and any Jenkins Build jobs for Shibboleth are now regarded as obsolete.

A "Release" of Shibboleth at Yale is really , therefore, a new version set of the configuration files . The actual Artifact of the Shibboleth application changes very infrequently, but new Relying Parties (Yale applications and cloud partners) are defined every week.

In the current arrangement, all the metadata and configuration files are the same for all three instances of Shibboleth, or as much so as possible. Generally Development versions of Relying Parties tend to point to DEV Shibboleth for their SSO, but PROD or TEST Shibboleth would authenticate them if they pointed to either of them. There are a very small number of parameters that select specific URNs, URLs, or file names that have to be different in each Shibboleth instance (like, for example, the IDP's own Metadata file with its own certificate, or the database URLs). These instance-specific values are set from the DEV, TEST, or PROD .properties file and are substituted into the configuration files and data sources as the Ant script copies files over during each install job.

Because the Artifact (WARs and JARs) don't change, there is no Build job. There is just an Install. While there is a WAR, the Shibboleth configuration files go in the IDP directory outside the Web Server and so are checked into Subversion as files (they are not zipped up) and are copied as individual files by the Ant script the Installer runsstored in the Jenkins Install project under Source Control and has nothing to do with the artifact. A new artifact is generated when we get a new release of code from shibboleth.net. Since we make no modifications to Shibboleth itself, it should be possible to migrate to a new release by downloading a new distribution and generating a new artifact, then switching the artifact.version in the Jenkins Install job parameters.

The Jenkins Install job parameters generate an "install.properties" file in the Install project. There is also an install-ALL.properties file for properties that are common to all servers in all environments and should not change much, and separate install-XXX.properties files where XXX is replaced by DEV, TEST, PREPROD, or PROD. A rule is that no parameter should be in more than one properties file, but if you break the rule then install.properties always beats the other files, and install-ALL.properties beats the DEV/TEST/PROD properties files.

The structure of the configuration files in the Install project exactly duplicates the subdirectory and file structure of the shibboleth-idp directory created by the Shibboleth installer and saved as the artifact. There are two directories. The files and directories in src/main/installer/resources/updateArtifact are copied exactly as is into the artifact deploy directory (/apps/shibboleth-idp by current convention). The files and directories in src/main/installer/resources/updateArtifactSubstituteVariables are copied but the text files are scanned and variables of the form ${foo.bar} are replaced by the value of the "foo.bar" property from the properties files or Ant script.

The Jenkins Install jobs check out the configuration files inInstall Project files from Source Control using the URL:

• https://svn.its.yale.edu/repos/shibboleth/yale-shibboleth-installer/trunk (for DEV)
• https://svn.its.yale.edu/repos/shibboleth/yale-shibboleth-installer/tags/Release (for TEST or PROD).

(and remember to change the directory name if necessary, for example to yale-shibboleth3-installer if that is the current project).

To work on Shibboleth configuration, use Eclipse to check out the current trunk (from the URL listed above for DEV). Edit changes into the file and commit the changes to the trunk. Run the Jenkins DEV install job and test the changes in the development environment. Generally you continue making changes this way until you need to push something into TEST or PROD, which you try to put off until it is necessary.

To install into TEST or PROD, you have to cut a new Release tag. However, you need to save the previous Release tag as your "rollback" if something goes wrong. So use Eclipse SVN Repository exploring to Refactor - Rename the old Release tag to something else. Then use trunk to create a new Release tag.

Request that the Unix Virtualization group run the TEST install job and check the results. When it is time to release, they will run the PROD install job.

If something goes wrong with the new install, rename the /tags/Release to something else for later diagnosis, restore the previous saved tag to the "Release" name, and have the PROD install job rerun.

%{foo.bar}, ${foo.bar}, <property name="foo.bar">, foo.bar=

In the three properties files (install.properties from Jenkins parameters, install-ALL.properties, install-XXX.properties)  a property is given a value by a line of for the form:

foo.bar=/var/log/shibboleth-idp

Once a property is set like this, it cannot be changed in Ant. However, if a property has not be set in any properties file then in the build.xml script executed by Ant then a default value can be provided for properties not set by the files with an Ant statement:

<property name="foo.bar" value="/var/log/shibboleth-idp" />

Properties set either way will be used to replace references of the form "${foo.bar}" when they appear in any file in the src/main/installer/resources/updateArtifactSubstituteVariables directory tree. In particular, if you want the same property to be available during Spring intialization you have to create a line in updateArtifactSubstituteVariables/conf/idp.properties of the form

foo.bar=${foo.bar}

Assuming that foo.bar was assigned a value for the Install job, either in a properties file or a <property> Ant statement, then when the idp.properties file is copied with substitution then Ant will see the "${foo.bar}" after the equals and will substitute the property value of "/var/log/shibboleth-idp". So while the statement above will appear in the Install project, after the Install is done if you go to the artifact deploy directory on the server you will find that the actual content of the /apps/shibboleth-idp/conf/idp.properties file is

foo.bar=/var/log/shibboleth-idp

Yes this is exactly what was in the original properties file, but that is important because while the three install-?.properties files were read in during Jenkins install job processing, now the /apps/shibboleth-idp/conf/idp.properties file is read in during Spring initialization on the Shibboleth server.

All the Shibboleth configuration files can contain variable references that will be replaced by Spring. Spring calls this "EL" or execution language, and it is much more powerful than just variable substitution but the extra features don't matter to us now. Shibboleth already makes extensive use of EL in its own configuration. In Spring EL, the variables are marked by a %{foo.bar} instead of a $.

So if you are coding a new Spring bean or another configuration item that you think needs variable substitution, and you want to do it the way Shib handles all its parameters, then you

• Add a property to one of the Install job properties files.
• Add a new line in the idp.properties file that sets the Spring runtime property to its Install time value (foo.bar=${foo.bar}).
• Reference the run time property in the configuration file using % instead of $ (logdirectory="%{foo.bar}")