Yale Incidental Shibboleth Configuration Notes

Metadata and Attributes are Documented in another Page

From week to week we add new Metadata files and configure the release of attributes to new applications. Occasionally we defined new Attributes or get the same Attribute from a new database. That is the routine maintenance and it is documented in a different page. It is probably a good idea to read the Shibboleth Configuration page first before reading this page because it is more important, and because I assume you understand a little about Spring Bean configuration already.

What is left are all the other Shibboleth configuration files where I have made one change here and one change there for Yale specific reasons and the only time anyone will be interested in these files is if we upgrade to a new release of Shibboleth or if Yale requirements change. That other less important stuff is documented here.

The configuration rules and syntax of all the files are defined in the Shibboleth Wiki pages. However, while the Shibboleth documentation will tell you what was done, this page will explain why Yale did it and what we expect.

If you upgrade a Shibboleth 2 to a Shibboleth 3 the install process turns on several compatibility switches and then Shibboleth 3 reads and uses the old format configuration files. We did not choose that option. The Shibboleth 3 configuration options have improvements, and it made sense to migrate the files to the new syntax manually to get all the new features.

The CAS Integration

Starting with Shibboleth 3, Yale uses the CAS-Shibboleth integration written by Unicon. This is mostly documented in the Unicon project at https://github.com/Unicon/shib-cas-authn3

Shibboleth must authenticate (logon) a user. Normally it will present a login page, the user will enter a userid and password, and Shibboleth will authenticate the user to some backend service (Kerberos, LDAP, ...). We want to use our standard CAS SSO.

In Shibboleth 2 we used an external authentication with the standard CAS Java Client. We defined Java Servlet Filters in the web.xml file that ran before the user got to the Shibboleth application. These filters redirected the user to CAS, then validated the Service Ticket, and stored the userid by wrapping the HTTPServletRequest object with a proxy that returned the Netid when the application (Shibboleth) called the request.getRemoteUser() method. Shibboleth was then configured to use its own RemoteUser authentication module, which assumes the user has been authenticated by the Container (Tomcat, JBoss), although in this case it is a Servlet Filter that did the job.

In Shibboleth 3 we remote the Java CAS Client Filters from web.xml. Unicon adds a new Authentication module to Shibboleth. In addition to Kerberos, LDAP, RemoteUser, and so on there is now a Shibcas option. So CAS authentication is now driven from inside Shibboleth instead of from a Filter that runs before Shib. This allows a tighter integration between SAML options and CAS options.

Do not confuse this with the ability of Shibboleth to use CAS protocol and perform the CAS function to other applications. That is a whole different set of modules and configuration options that we do not use at Yale.

At this time, the configuration files that Unicon distributes through Github were downloaded and placed in the Install Project under Source Control. This includes the jar files containing the Unicon code, which we have not modified. It could be argued that jar files belong in the Artifact in Artifactory rather than being in the Install Project because they are code and not source. This is a legitimate point, but for now the Artifact is vanilla shibboleth.net distributed content, there is no Jenkins Build job, so extra jars are at least for now copied during the Install phase.

Any CAS Client has to be given two parameters:

• "https://secure.its.yale.edu/cas" - You need to tell the client how to get to the CAS Server. The rest of the URL (/login, /serviceValidate, etc.) is part of the CAS protocol and the client can add those pieces itself.
• "https://auth.yale.edu/idp" - You need to tell the client what URL to use as the service= so CAS can redirect the browser back to Shibboleth.

You might think that Shibboleth knows its own URL, but that is not true in general if Shibboleth is behind an F5 front end. The "auth.yale.edu" is the DNS name of a Virtual IP address on the F5, and the F5 forwards anything that comes in for "https://auth.yale.edu/idp" to some "vm-xxxx-01.web.yale.internal" Virtual Machine. The problem is that Shibboleth running on the VM can only query its host for the local hostname, and that will be the internal VM name that may not even be accessible to the browser. The service has to go through the F5, and Shibboleth can only discover the F5 name if you configure it.

Unfortunately, there may be some confusion over port numbers. This would not be a problem if the CAS Shibboleth Integration took the port number it puts in the service= string from the configuration. However, Unicon only configures the server name part and calls a routine to add the port number and context name (/idp) on the end of the configured host name. The routine gets this information from the HTTPServletRequest object, which in turn gets them from the Tomcat container.

This becomes a problem because the port number (or its absence for 80 and 443) used by the F5 VIP may not be the port number used by Tomcat on the virtual machine. In fact, since the F5 holds the certificate and terminates the SSL connection from the browser, and because there is nothing super sensitive in the SAML Response, the connection between the F5 and the VM may be ordinary unencrypted HTTP, so the VM port number may be the Tomcat default of 8080. This can result in a service=https://auth.yale.edu:8080/idp/... which means the F5 has to be configured to accept SSL for auth.yale.edu on both 443 and 8080 (or it may be the default to accept all ports, since I do not know how this ever worked before unless the F5 is ignoring the port number in its processing). If the "tomcat" user on the VM were privileged to bind to port 443, it could be possible to put a self-signed certificate into Tomcat (since the F5 does not validate the Certificates it gets from a machine room computer), and run SSL and the same port number between the F5 and the VM that is used between the Browser and the F5.

The two CAS parameters are different for DEV, TEST, and PROD because there are three CAS servers that might be configured (secure, secure-dev, and auth-dev). In the install-XXX.properties file you will find:

# The CAS Server and the service= hostnames
cas.server.url=https://secure.its.yale.edu
cas.target.url=https://auth.yale.edu

These are Yale parameters. They are plugged into the shibboleth-idp/conf/idp.properties file to generate derived parameters (shibcas.casServerUrlPrefix and shibcas.serverName) that are used by the Unicon integration code.

On a Sandbox when using IdP Initiated Shibboleth login (https://localhost:8080/idp/profile/SAML2/Unsolicited/SSO?providerId=entityID) then you can set the cas.target.user to localhost as well. However, to process a request for an SP initiated login you need to be running a proxy on your desktop. See Testing Shibboleth - Test Cases for more information.

Credentials

The shibboleth-idp/credentials directory of the running Shib server contains files for the private key used to digitally sign SAML messages and the Certificate that contains the public key that partners use to validate the signature. The private key is very, very sensitive because possession of this file would allow someone to impersonate any Yale user. The private key is not part of the Artifact or the Install Project. It is never stored on Artifactory or Source Control. Protection of this key file belongs to Operations.

The Certificate, however, is just a convenient format to distribute a public key. It is self-signed and is never issued by a real CA. The only information it contains is the Public Key. If there was some other widely supported file format for distributing public keys, we might never use Certificates for the purpose. It goes without saying that this has nothing to do with the SSL certificate of the server, and it is not and cannot in general be validated with a Trust chain.

We are in the middle of a shift from old SSL Certificates that used SHA1 digest information to SHA256, because the old SHA1 algorithm is no longer regarded as secure. Shibboleth has an entirely different problem. In SSL Certificates, the digest is used to validate the content of the Certificate itself and verify that it was issued by the CA it claims to come from. SAML certificates are all self-signed, so there is no CA validation. Instead, SAML uses the XML Digital Signature standard to validate that the SAML message was sent by the IdP it claims to have come from. Here a digest is calculated on the XML and not the Certificate. It is still true that SHA1 is now regarded as insecure.

Shibboleth 2 defaulted to using SHA1 and Shibboleth 3 defaults to using SHA256. Unlike SSL, there is no negotiation. Shibboleth will use whichever digest mechanism is configured for the Relying Party to which the message is being sent. Normally it uses the default. If a particular Relying Party cannot support SHA256, then the Metadata of that RP can be changed to force SHA1. It is not clear if this is a real requirement for any RP. SAML has only been widely used for five years now, and SHA256 was universally supported at that time.

Each Yale environment (DEV, TEST, and PROD) has its own private key (and public key certificate, and Metadata file, and EntityID). InCommon and most partners only configure the PROD information and therefore only trust SAML generated by PROD. When the Shibboleth VMs change, then the idp.key and idp.crt files have to be copied from the old DEV to the new DEV, from the old TEST to the new TEST, and from the old PROD to the new PROD.

SAML provides for separate keys for encrypting Assertions and for signing Assertions. Shibboleth as distributed by shibboleth.net generates three sets of files: (signing, encryption, and backchannel). The backchannel appears to be for Web Service requests. In theory it might be important to separate things, but for now in the "real world" Yale encrypts no Attributes at the SAML level. The vendor should supply an https: AssertionConsumerService URL and then the entire SAML Response will be encrypted by the transport layer. Therefore, we use only two files (idp.key and idp.crt) and configure them for use in both signing and encryption (that we don't use).

The name of the files to use for signing and encryption are specified in the conf/idp.properties file. We changed this configuration file from the original distribution (with separate signing and encryption keys) to use just the one pair of keys and, therefore, pair of files.

InCommon itself provides us with a credential that we can use to validate the digital signature on their Metadata file. The "incommon.pem" file is public and is checked into Source Control and is added by the Install Project.

Shibboleth 3 adds a new key and pair of files that are used to encrypt the Cookie used to provide Client-side storage of session data. This is held in the sealer.jks and sealer.kver files. Periodically a chron job generates a new key pair which is added to the "sealer" files, because when a new key is generated the old key remains valid until session timeout guarantees there is no valid data still sealed with the old key.

Generic Spring Bean Parameters

The Yale Install Project stores parameters in the install.properties file (created from the Jenkins Install job parameters), plus the install-ALL.properties and the install-XXX.properties (where XXX is DEV, TEST, or PROD). These properties are loaded into the Maven environment and then the Ant environment of the Install Project and are available only during the execution of the Install job itself. As a result, any configuration file stored in Source Control as part of the resources/updateArtifactSubstituteVariables subdirectory of the Install Project can contain ${property.name} placeholders that will be replaced by the value of a corresponding property as the Ant script in the Install Project copies configuration files into the /apps/shibboleth-idp directory. This produces configuration files that have literal numbers and strings as values.

Shibblleth 3 fully embraces the Spring Framework, which has its own "variable substitution from parameter file" capability called Spring Execution Language. At startup, Spring replaces %{property.name} placeholders in its XML files with the corresponding values from the shibboleth-idp/conf/idp.properties file (and other *.properties files in the conf subdirectory).

Of course, Spring will also use literal values. So we have a choice to do the property substitution at Ant install time or at Spring initialization time. Both should work. When we decide to use the Spring way of doing business, the problem is to get a property value defined in one of the Install Job property files transferred to the idp.properties file used by the Spring runtime. To do this the idp.properties file is itself in the Install Project as resources/updateArtifactSubstituteVariables/conf/idp.properties, and because it is in the SubstituteVariable directory at Install time an ${foo} expressions get replaced. So in Source Control Yale adds lines to that file of the form:

property.name=${property.name}

Then as this file is copied from the SubstituteVariables section of the Install Project to the /apps/shibboleth-idp/conf directory, the "${property.name}" is replaced by a string or number from the Install job properties files. Strictly speaking, it is not necessary for the Spring property name (on the left of the =) to be the same as the Install Project property name (on the right side) but it is simpler to remember when they are. This means that you have to be sensitive to the fact that %{foo} expressions in the resources/updateArtifact directory (which are not replaced by properties) are copied unchanged to shibboleth-idp and are then replaced logically at Spring startup from the values in idp.properties that are replaced during Install.

When in doubt, look at the files after running Install in a Sandbox and compare them to the Install project to see which change and which remain the same.

DataSources

In Shibboleth 2 a Datasource (a database connection pool configuration specifying the database url, jdbc class name, userid, and password) could be configured in a DataConnector element of the attribute-resolver.xml or it could be supplied by the container (the Web Server which is to say Tomcat or JBoss). There was, however, a scary comment in the Wiki that if you use Oracle as your database (which we did) then the Shibboleth provided DataConnector implementation did not work well and they recommended using a container datasource. So we used all ContainerManaged datasources.

A Datasource maintains a pool of connections. Sometimes they get old and are closed at the database end. Sometimes the database goes down and breaks the connection. Before passing a connection to Shib, the Datasource is supposed to check to make sure it is usable. This is not a trivial job, and Apache, JBoss, Spring, and other environments have code (sometimes vendor specific code) to get it right. Shib is not in the generic JDBC business, so you will generally get more reliable behavior if you use code from someone who put a lot of effort into getting it exactly right.

For the first cut, Shibboleth 3 continues to use ContainerManaged datasources. However, we now run under Tomcat instead of JBoss, and Tomcat uses Apache datasource management which is OK but not really as good as JBoss. The solution will be to move from ContainerManaged datasources to a new option that was added by Shib 3 where the Datasources can be managed by Spring.

Tomcat managed datasources are configured in the context.xml file that is copied (with variable substitution) to the /apps/tomcat/current/conf/catalina/localhost/idp.xml file by the Install Project.

Spring managed datasources will be defined as Beans in one of the Bean XML files.

Logging

Logging in Shibboleth 3 is controlled by the conf/logback.xml file. Initially it is treated as a normal configuration file. However, we may want to create a special Jenkins Install runtype to change the logging levels to get more information from a running server, provided that the logging levels can change dynamically.

Services

The conf/services.xml file configures the services and subsystems. In particular, it configures the names and locations of the XML files that configure the other parts of Shibboleth.

It is important enough that Yale removed the commented out sections that are used by services we do not enable so the file is clean.

Yale made on aesthetic change. Normally there is one attribute-resolver.xml file containing both DataConnectors (queries) and AttributeDefinitons. We broke the file into two separate tiles because the two types of attribute configurations are different and because we can assign a different xmlns default to the different types of XML elements if they are in separate files.

Spring <bean> elements creating Yale configured Spring Bean objects have to go somewhere. The two obvious places to put them are global.xml and services.xml. We put them at the end of the services.xml file. The current use is to create "predicates" that return true or false based on whether the application the user is logging into is in a list of EntityIDs.

Relying Party

In Shibboleth 2 the relying-party.xml file also included MetadataProviders. In Shib 3 there is a separate file for metadata-providers.

In SAML, the IdP can digitally sign both the Response and the Attributes, or just the Response. The Attributes can be encrypted or not. Shibboleth has a default. The relying-party.xml file provides syntax to specify specific Yale defaults for each option. Default behavior can be overridden by the Metadata of the RP or by the contents of the Request (if one is sent).

Yale does not encrypt attributes if we can avoid it because it makes debugging impossible and because the entire SAML message is typically protected by SSL. In Shib 2 our default behavior was to digitally sign both the Response and the Attribute Assertion, but now we just sign the Response by default.

The new Shib 3 relying-party.xml file is small and has a new syntax. In theory we could specify defaults for each SAML protocols supported, but currently the only SAML Protocol that anyone uses is SSO:

                <bean parent="SAML2.SSO" p:postAuthenticationFlows="%{idp.postauthenticationflows}" 
                      p:encryptAssertions="false" p:encryptNameIDs="false" p:encryptAttributes="false" />

The postAuthenticationFlows parameter currently has one possible interesting value. If it is set to "attribute-release" then this triggers the Consent dialog, where the user is given a list of attributes that are gooing to be released and has a chance to approve them. Currently we turn on Consent for DEV (because it makes debugging easier) and for Anonymous Relying Parties (where we get a SAML Request from an EntityID for which we have no Metadata).

In the near future:

• We may want to turn off Anonymous Relying Parties. We need to confirm that Metadata has been generated for every party we currently support in production. Scanning Shib 3 logs will give us a chance to look for any remaining legitimate Anonymous RPs and configure them properly. Currently we turn Consent on for Anonymous.

Metadata Providers

The metadata-providers.xml file contains almost exactly the same content as Shibboleth 2. The only change is that Refresh periods have to be expressed as "PT5M" (which means check every 5 minutes) where as in Shibboleth 2 the same thing could be expressed as an integer number of milliseconds.

Although we have not changed the configuration, the defaults appear to have changed. Shib 2 did not check local disk Metadata files unless an explicit refresh time was configured. Shib 3 appears to check the timestamp on files for changes every 5 minutes by default. It does not seem necessary to change this.

Access Control (to Status Page)

Currently the only form of Access Control is by IP address. We modified the conf/access-control.xml file to allow IP address ranges to be inserted from a property file value

                p:allowedRanges="#{ {'127.0.0.1/32', '::1/128', %{allow.status}} }"

 So if the properties files have a line of the form:

allow.status='172.28.187.0/24','172.17.40.0/24'

then a subnet in the DevOps area of ITS and a subnet in the SPS_DC VPN are able to display the status page. This will be expanded on request.

Logo and Footer

In a few messages we replace a dummy logo with a Yale ITS logo. The image is in webapp/images/YaleITS.png and it is used because of some lines in messages/error-messages.properties:

idp.logo = /images/YaleITS.png
idp.logo.alt-text = YaleITS
idp.message = An unidentified error occurred.
idp.footer = Yale University Shibboleth Identity Provider