This is a note to any IDM staff who become responsible for making Shibboleth configuration changes. This reflects the current practice that uses just the /trunk and /tags/Release entries in the SVN repository for the yale-shibboleth-installer. During a transition between releases, new code may be in a slightly different named directory (yale-shibboleth3-installer during the transition from Shib 2 to Shib 3).
Starting with Shibboleth 3, the artifact stored in Artifactory is created by downloading the standard Shibboleth distribution zip file, running the Install script inside that file on a test machine, creating a more or less standard shibboleth-idp directory, and then zipping up the generated directory. This zip artifact is then manually uploaded to Artifactory as shibboleth-idp-3.2.1. No Jenkins Build job is run, and any Jenkins Build jobs for Shibboleth are now regarded as obsolete.
A "Release" of Shibboleth is, therefore, a new set of configuration files stored in the Jenkins Install project under Source Control and has nothing to do with the artifact. A new artifact is generated when we get a new release of code from shibboleth.net. Since we make no modifications to Shibboleth itself, it should be possible to migrate to a new release by downloading a new distribution and generating a new artifact, then switching the artifact.version in the Jenkins Install job parameters.
The Jenkins Install job parameters generate an "install.properties" file in the Install project. There is also an install-ALL.properties file for properties that are common to all servers in all environments and should not change much, and separate install-XXX.properties files where XXX is replaced by DEV, TEST, PREPROD, or PROD. A rule is that no parameter should be in more than one properties file, but if you break the rule then install.properties always beats the other files, and install-ALL.properties beats the DEV/TEST/PROD properties files.
The structure of the configuration files in the Install project exactly duplicates the subdirectory and file structure of the shibboleth-idp directory created by the Shibboleth installer and saved as the artifact. There are two directories. The files and directories in src/main/installer/resources/updateArtifact are copied exactly as is into the artifact deploy directory (/apps/shibboleth-idp by current convention). The files and directories in src/main/installer/resources/updateArtifactSubstituteVariables are copied but the text files are scanned and variables of the form ${foo.bar} are replaced by the value of the "foo.bar" property from the properties files or Ant script.
The Jenkins Install jobs check out the Install Project files from Source Control using the URL:
- https://svn.its.yale.edu/repos/shibboleth/yale-shibboleth-installer/trunk (for DEV)
- https://svn.its.yale.edu/repos/shibboleth/yale-shibboleth-installer/tags/Release (for TEST or PROD).
(and remember to change the directory name if necessary, for example to yale-shibboleth3-installer if that is the current project).
To work on Shibboleth configuration, use Eclipse to check out the current trunk (from the URL listed above for DEV). Edit changes into the file and commit the changes to the trunk. Run the Jenkins DEV install job and test the changes in the development environment. Generally you continue making changes this way until you need to push something into TEST or PROD, which you try to put off until it is necessary.
To install into TEST or PROD, you have to cut a new Release tag. However, you need to save the previous Release tag as your "rollback" if something goes wrong. So use Eclipse SVN Repository exploring to Refactor - Rename the old Release tag to something else. Then use trunk to create a new Release tag.
If something goes wrong with the new install, rename the /tags/Release to something else for later diagnosis, restore the previous saved tag to the "Release" name, and have the PROD install job rerun.
%{foo.bar}, ${foo.bar}, <property name="foo.bar">, foo.bar=
In the three properties files (install.properties from Jenkins parameters, install-ALL.properties, install-XXX.properties) a property is given a value by a line of for the form:
foo.bar=/var/log/shibboleth-idp
Once a property is set like this, it cannot be changed in Ant. However, if a property has not be set in any properties file then in the build.xml script executed by Ant then a default value can be provided for properties not set by the files with an Ant statement:
<property name="foo.bar" value="/var/log/shibboleth-idp" />
Properties set either way will be used to replace references of the form "${foo.bar}" when they appear in any file in the src/main/installer/resources/updateArtifactSubstituteVariables directory tree. In particular, if you want the same property to be available during Spring intialization you have to create a line in updateArtifactSubstituteVariables/conf/idp.properties of the form
foo.bar=${foo.bar}Assuming that foo.bar was assigned a value for the Install job, either in a properties file or a <property> Ant statement, then when the idp.properties file is copied with substitution then Ant will see the "${foo.bar}" after the equals and will substitute the property value of "/var/log/shibboleth-idp". So while the statement above will appear in the Install project, after the Install is done if you go to the artifact deploy directory on the server you will find that the actual content of the /apps/shibboleth-idp/conf/idp.properties file is
foo.bar=/var/log/shibboleth-idp
Yes this is exactly what was in the original properties file, but that is important because while the three install-?.properties files were read in during Jenkins install job processing, now the /apps/shibboleth-idp/conf/idp.properties file is read in during Spring initialization on the Shibboleth server.
All the Shibboleth configuration files can contain variable references that will be replaced by Spring. Spring calls this "EL" or execution language, and it is much more powerful than just variable substitution but the extra features don't matter to us now. Shibboleth already makes extensive use of EL in its own configuration. In Spring EL, the variables are marked by a %{foo.bar} instead of a $.
So if you are coding a new Spring bean or another configuration item that you think needs variable substitution, and you want to do it the way Shib handles all its parameters, then you
- Add a property to one of the Install job properties files.
- Add a new line in the idp.properties file that sets the Spring runtime property to its Install time value (foo.bar=${foo.bar}).
- Reference the run time property in the configuration file using % instead of $ (logdirectory="%{foo.bar}")
Of course, you could decide not to do things the way Shib does them, you could move the configuration file to the updateArtifactSubstituteVariables directory tree and then just reference the property as ${foo.bar}. Then the variable will be replaced by the value at install time and there will be no runtime substitution. It should work the same, but it is not the Shib way to do it.