Metadata and Attributes
...
are Documented in another Page
From week to week we add new Metadata files and configure the release of attributes to new applications. Occasionally we defined new Attributes or get the same Attribute from a new database. That is the routine maintenance and it is documented in a different page. It is probably a good idea to read the Shibboleth Configuration page first before reading this page because it is more important, and because I assume you understand a little about Spring Bean configuration already.
...
Starting with Shibboleth 3, Yale uses the CAS-Shibboleth integration written by Unicon. This is mostly documented in the Unicon project at https://github.com/Unicon/shib-cas-authn3
Shibboleth has many built in ways to authenticate users Shibboleth must authenticate (logon) a user. Normally it will present a login page, the user will enter a userid and password, and Shibboleth will authenticate the user to some backend service (Kerberos, LDAP, ...), but we . We want it to use our standard Yale CAS SSO solution through CAS.
In Shib Shibboleth 2 we used the CAS Client that passes the Netid in the RemoteUser field of the request, and then we select the "RemoteUser" authentication option for Shib. The Unicon code implements the Shibboleth authentication interface and behaves like the other Shibboleth authentication options, but then it redirects the browser to CAS and handles the ticket on return. Potentially this provides additional features, like turning the SAML version of renew=true into the actual CAS renew=true parameter, which you do not get when you use a generic CAS Client that operates as a Servlet Filter unaware of Shibboleth.
Do not get confused with the internal Shibboleth implementation of the CAS protocol. Shibboleth will replace CAS, but we at Yale do not use that option. At this time the CAS protocol support has not been removed from Shibboleth and might even be functional. If it is there we don't use it and it doesn't hurt anything.
Although the link above takes you to a Github location where you could download the Unicon source, we simply download the JAR and Shibboleth configuration files. We do not recompile the source, although you are free to do so.
There are two ways you can manage the integration using Yale project management rules. You could put the Unicon jar files into the Shibboleth artifact, which would probably be the right thing to do since they are code and not configuration text files. Currently, however, we use the second option and check them into Source Control in the resources/updateArtifact/webapps/WEB-INF/lib directory of the Install project even through the jar files are not "source". This is simpler. If we made any significant changes to the Artifact and generate a Build job, then we would move the JAR files.
Any CAS integration has to provide two parameters. You need the URL of CAS itself, and you need a URL or host name for the service string you send to CAS.
You might thing you can get the service= string from the Hostname. The problem is that Yale has an F5 front end that owns the "auth.yale.edu" DNS name which is the public name used to access Shibboleth and has to be the value in the service= string, while Hostname on the Shibboleth VM is going to return "vm-shibprd-01.yale.internal" which is not the right value.
The two CAS parameters are different for DEV, TEST, and PROD because there are three CAS servers that might be configured (secure, secure-dev, and auth-dev). In the install-XXX.properties file you will find:
Code Block | ||
---|---|---|
| ||
# The CAS Server and the service= we pass to it.
cas.server.url=https://auth-dev.yale.edu
cas.target.url=https://auth-dev.yale.edu
###### TEMPORARY VALUE used for SANDBOX or SSH TUNNEL to Shib
#cas.target.url=http://localhost
|
To understand these parameters you need to understand that the names are from the Browser's point of view, not from the point of view of either the CAS VM or the Shib VM. These URLs will be sent to the Browser with an HTTP Redirect and then the Browser will find CAS or Shib itself. In a Sandbox where you are running Shib and maybe CAS on your own desktop under Tomcat, you may be using http instead of https (because it is simpler for testing purposes) and you may be running on port 8080 instead of the standard 80 or 443. More importantly, you can configure your /etc/hosts or \windows\system32\drivers\etc\hosts file to define the hostname as the local machine. Even when using DEV or TEST, you may run an SSH tunnel to the VM, and therefore access the Shib server as if it were localhost:8080.
The cas.server.url parameter is the URL of the CAS server, and the cas.target.url is the Shibboleth server. In DEV they happen to be the same DNS name because it just happens that CAS is https://auth-dev.yale.edu/cas and DEV Shib is https://auth-dev.yale.edu/idp (although they are on different VMs behind the same F5 VIP).
The cas.server.url may need a port number, but you don't add port number to the cas.target.url. I don't know why.
Credentials
The shibboleth-idp/credentials directory of the running Shib server contains files for the private key used to digitally sign SAML messages and the Certificate that contains the public key that partners use to validate the signature. The private key is very, very sensitive because possession of this file would allow someone to impersonate any Yale user. The private key is not part of the Artifact or the Install Project. It is never stored on Artifactory or Source Control. Protection of this key file belongs to Operations.
The Certificate, however, is just a convenient format to distribute a public key. It is self-signed and is never issued by a real CA. The only information it contains is the Public Key. If there was some other widely supported file format for distributing public keys, we might never use Certificates for the purpose. It goes without saying that this has nothing to do with the SSL certificate of the server, and it is not and cannot in general be validated with a Trust chain.
As a result, it does not matter what Crypto digest algorithm is specified in the Certificate. The only thing that matters is the 2048 bit RSA key. No matter what the certificate says, Shibboleth will choose SHA1 or SHA256 for its digital signature based on the other Shibboleth security configuration files. There is a default digest algorithm and then there can be exceptions based on the Metadata or explict configuration based on EntityID. Therefore, this key and certificate only need to be replaced if 2048 bit RSA becomes vulnerable.
I believe that Shibboleth 2 defaulted to SHA1, and I know that Shibboleth 3 defaults to SHA256. The exact syntax for configuring exceptions is not clearly documented in the Wiki, so we hope either that it is not needed or that partners that do need it give us correctly configured Metadata.
Each Yale environment (DEV, TEST, and PROD) has its own private key (and public key certificate, and Metadata file, and EntityID). InCommon and most partners only configure the PROD information and therefore only trust SAML generated by PROD. When the Shibboleth VMs change, then the idp.key and idp.crt files have to be copied from the old DEV to the new DEV, from the old TEST to the new TEST, and from the old PROD to the new PROD.
SAML provides for separate keys for encrypting Assertions and for signing Assertions. Shibboleth as distributed by shibboleth.net generates three sets of files: (signing, encryption, and backchannel). The backchannel appears to be for Web Service requests. In theory it might be important to separate things, but for now in the "real world" Yale encrypts no Attributes at the SAML level. The vendor should supply an https: AssertionConsumerService URL and then the entire SAML Response will be encrypted by the transport layer. Therefore, we use only two files (idp.key and idp.crt) and configure them for use in both signing and encryption (that we don't use).
The name of the files to use for signing and encryption are specified in the conf/idp.properties file. We changed this configuration file from the original distribution (with separate signing and encryption keys) to use just the one pair of keys and, therefore, pair of files.
InCommon itself provides us with a credential that we can use to validate the digital signature on their Metadata file. The "incommon.pem" file is public and is checked into Source Control and is added by the Install Project.
Shibboleth 3 adds a new key and pair of files that are used to encrypt the Cookie used to provide Client-side storage of session data. This is held in the sealer.jks and sealer.kver files. Periodically a chron job generates a new key pair which is added to the "sealer" files, because when a new key is generated the old key remains valid until session timeout guarantees there is no valid data still sealed with the old key.
Generic Spring Bean Parameters
The Yale Install Project stores parameters in the install.properties file (created from the Jenkins Install job parameters), plus the install-ALL.properties and the install-XXX.properties (where XXX is DEV, TEST, or PROD). These properties are loaded into the Maven environment and then the Ant environment of the Install Project and are available only during the execution of the Install job itself. As a result, any configuration file stored in Source Control as part of the resources/updateArtifactSubstituteVariables subdirectory of the Install Project can contain ${property.name} placeholders that will be replaced by the value of a corresponding property as the Ant script in the Install Project copies configuration files into the /apps/shibboleth-idp directory. This produces configuration files that have literal numbers and strings as values.
Shibblleth 3 fully embraces the Spring Framework, which has its own "variable substitution from parameter file" capability called Spring Execution Language. At startup, Spring replaces %{property.name} placeholders in its XML files with the corresponding values from the shibboleth-idp/conf/idp.properties file (and other *.properties files in the conf subdirectory).
Of course, Spring will also use literal values. So we have a choice to do the property substitution at Ant install time or at Spring initialization time. Both should work. When we decide to use the Spring way of doing business, the problem is to get a property value defined in one of the Install Job property files transferred to the idp.properties file used by the Spring runtime. To do this the idp.properties file is itself in the Install Project as resources/updateArtifactSubstituteVariables/conf/idp.properties, and because it is in the SubstituteVariable directory at Install time an ${foo} expressions get replaced. So in Source Control Yale adds lines to that file of the form:
property.name=${property.name}
Then as this file is copied from the SubstituteVariables section of the Install Project to the /apps/shibboleth-idp/conf directory, the "${property.name}" is replaced by a string or number from the Install job properties files. Strictly speaking, it is not necessary for the Spring property name (on the left of the =) to be the same as the Install Project property name (on the right side) but it is simpler to remember when they are. This means that you have to be sensitive to the fact that %{foo} expressions in the resources/updateArtifact directory (which are not replaced by properties) are copied unchanged to shibboleth-idp and are then replaced logically at Spring startup from the values in idp.properties that are replaced during Install.
When in doubt, look at the files after running Install in a Sandbox and compare them to the Install project to see which change and which remain the same.
DataSources
In Shibboleth 2 a Datasource (a database connection pool configuration specifying the database url, jdbc class name, userid, and password) could be configured in a DataConnector element of the attribute-resolver.xml or it could be supplied by the container (the Web Server which is to say Tomcat or JBoss). There was, however, a scary comment in the Wiki that if you use Oracle as your database (which we did) then the Shibboleth provided DataConnector implementation did not work well and they recommended using a container datasource. So that is what we did.
When we migrated to Shibboleth 3 we have initially maintained this convention. Therefore, an external authentication with the standard CAS Java Client. We defined Java Servlet Filters in the web.xml file that ran before the user got to the Shibboleth application. These filters redirected the user to CAS, then validated the Service Ticket, and stored the userid by wrapping the HTTPServletRequest object with a proxy that returned the Netid when the application (Shibboleth) called the request.getRemoteUser() method. Shibboleth was then configured to use its own RemoteUser authentication module, which assumes the user has been authenticated by the Container (Tomcat, JBoss), although in this case it is a Servlet Filter that did the job.
In Shibboleth 3 we remote the Java CAS Client Filters from web.xml. Unicon adds a new Authentication module to Shibboleth. In addition to Kerberos, LDAP, RemoteUser, and so on there is now a Shibcas option. So CAS authentication is now driven from inside Shibboleth instead of from a Filter that runs before Shib. This allows a tighter integration between SAML options and CAS options.
Do not confuse this with the ability of Shibboleth to use CAS protocol and perform the CAS function to other applications. That is a whole different set of modules and configuration options that we do not use at Yale.
At this time, the configuration files that Unicon distributes through Github were downloaded and placed in the Install Project under Source Control. This includes the jar files containing the Unicon code, which we have not modified. It could be argued that jar files belong in the Artifact in Artifactory rather than being in the Install Project because they are code and not source. This is a legitimate point, but for now the Artifact is vanilla shibboleth.net distributed content, there is no Jenkins Build job, so extra jars are at least for now copied during the Install phase.
Any CAS Client has to be given two parameters:
- "https://secure.its.yale.edu/cas" - You need to tell the client how to get to the CAS Server. The rest of the URL (/login, /serviceValidate, etc.) is part of the CAS protocol and the client can add those pieces itself.
- "https://auth.yale.edu/idp" - You need to tell the client what URL to use as the service= so CAS can redirect the browser back to Shibboleth.
You might think that Shibboleth knows its own URL, but that is not true in general if Shibboleth is behind an F5 front end. The "auth.yale.edu" is the DNS name of a Virtual IP address on the F5, and the F5 forwards anything that comes in for "https://auth.yale.edu/idp" to some "vm-xxxx-01.web.yale.internal" Virtual Machine. The problem is that Shibboleth running on the VM can only query its host for the local hostname, and that will be the internal VM name that may not even be accessible to the browser. The service has to go through the F5, and Shibboleth can only discover the F5 name if you configure it.
Unfortunately, there may be some confusion over port numbers. This would not be a problem if the CAS Shibboleth Integration took the port number it puts in the service= string from the configuration. However, Unicon only configures the server name part and calls a routine to add the port number and context name (/idp) on the end of the configured host name. The routine gets this information from the HTTPServletRequest object, which in turn gets them from the Tomcat container.
This becomes a problem because the port number (or its absence for 80 and 443) used by the F5 VIP may not be the port number used by Tomcat on the virtual machine. In fact, since the F5 holds the certificate and terminates the SSL connection from the browser, and because there is nothing super sensitive in the SAML Response, the connection between the F5 and the VM may be ordinary unencrypted HTTP, so the VM port number may be the Tomcat default of 8080. This can result in a service=https://auth.yale.edu:8080/idp/... which means the F5 has to be configured to accept SSL for auth.yale.edu on both 443 and 8080 (or it may be the default to accept all ports, since I do not know how this ever worked before unless the F5 is ignoring the port number in its processing). If the "tomcat" user on the VM were privileged to bind to port 443, it could be possible to put a self-signed certificate into Tomcat (since the F5 does not validate the Certificates it gets from a machine room computer), and run SSL and the same port number between the F5 and the VM that is used between the Browser and the F5.
The two CAS parameters are different for DEV, TEST, and PROD because there are three CAS servers that might be configured (secure, secure-dev, and auth-dev). In the install-XXX.properties file you will find:
Code Block | ||
---|---|---|
| ||
# The CAS Server and the service= hostnames
cas.server.url=https://secure.its.yale.edu
cas.target.url=https://auth.yale.edu
|
These are Yale parameters. They are plugged into the shibboleth-idp/conf/idp.properties file to generate derived parameters (shibcas.casServerUrlPrefix
and shibcas.serverName
) that are used by the Unicon integration code.
On a Sandbox when using IdP Initiated Shibboleth login (https://localhost:8080/idp/profile/SAML2/Unsolicited/SSO?providerId=entityID) then you can set the cas.target.user to localhost as well. However, to process a request for an SP initiated login you need to be running a proxy on your desktop. See Testing Shibboleth - Test Cases for more information.
Credentials
The shibboleth-idp/credentials directory of the running Shib server contains files for the private key used to digitally sign SAML messages and the Certificate that contains the public key that partners use to validate the signature. The private key is very, very sensitive because possession of this file would allow someone to impersonate any Yale user. The private key is not part of the Artifact or the Install Project. It is never stored on Artifactory or Source Control. Protection of this key file belongs to Operations.
The Certificate, however, is just a convenient format to distribute a public key. It is self-signed and is never issued by a real CA. The only information it contains is the Public Key. If there was some other widely supported file format for distributing public keys, we might never use Certificates for the purpose. It goes without saying that this has nothing to do with the SSL certificate of the server, and it is not and cannot in general be validated with a Trust chain.
We are in the middle of a shift from old SSL Certificates that used SHA1 digest information to SHA256, because the old SHA1 algorithm is no longer regarded as secure. Shibboleth has an entirely different problem. In SSL Certificates, the digest is used to validate the content of the Certificate itself and verify that it was issued by the CA it claims to come from. SAML certificates are all self-signed, so there is no CA validation. Instead, SAML uses the XML Digital Signature standard to validate that the SAML message was sent by the IdP it claims to have come from. Here a digest is calculated on the XML and not the Certificate. It is still true that SHA1 is now regarded as insecure.
Shibboleth 2 defaulted to using SHA1 and Shibboleth 3 defaults to using SHA256. Unlike SSL, there is no negotiation. Shibboleth will use whichever digest mechanism is configured for the Relying Party to which the message is being sent. Normally it uses the default. If a particular Relying Party cannot support SHA256, then the Metadata of that RP can be changed to force SHA1. It is not clear if this is a real requirement for any RP. SAML has only been widely used for five years now, and SHA256 was universally supported at that time.
Each Yale environment (DEV, TEST, and PROD) has its own private key (and public key certificate, and Metadata file, and EntityID). InCommon and most partners only configure the PROD information and therefore only trust SAML generated by PROD. When the Shibboleth VMs change, then the idp.key and idp.crt files have to be copied from the old DEV to the new DEV, from the old TEST to the new TEST, and from the old PROD to the new PROD.
SAML provides for separate keys for encrypting Assertions and for signing Assertions. Shibboleth as distributed by shibboleth.net generates three sets of files: (signing, encryption, and backchannel). The backchannel appears to be for Web Service requests. In theory it might be important to separate things, but for now in the "real world" Yale encrypts no Attributes at the SAML level. The vendor should supply an https: AssertionConsumerService URL and then the entire SAML Response will be encrypted by the transport layer. Therefore, we use only two files (idp.key and idp.crt) and configure them for use in both signing and encryption (that we don't use).
The name of the files to use for signing and encryption are specified in the conf/idp.properties file. We changed this configuration file from the original distribution (with separate signing and encryption keys) to use just the one pair of keys and, therefore, pair of files.
InCommon itself provides us with a credential that we can use to validate the digital signature on their Metadata file. The "incommon.pem" file is public and is checked into Source Control and is added by the Install Project.
Shibboleth 3 adds a new key and pair of files that are used to encrypt the Cookie used to provide Client-side storage of session data. This is held in the sealer.jks and sealer.kver files. Periodically a chron job generates a new key pair which is added to the "sealer" files, because when a new key is generated the old key remains valid until session timeout guarantees there is no valid data still sealed with the old key.
Generic Spring Bean Parameters
The Yale Install Project stores parameters in the install.properties file (created from the Jenkins Install job parameters), plus the install-ALL.properties and the install-XXX.properties (where XXX is DEV, TEST, or PROD). These properties are loaded into the Maven environment and then the Ant environment of the Install Project and are available only during the execution of the Install job itself. As a result, any configuration file stored in Source Control as part of the resources/updateArtifactSubstituteVariables subdirectory of the Install Project can contain ${property.name} placeholders that will be replaced by the value of a corresponding property as the Ant script in the Install Project copies configuration files into the /apps/shibboleth-idp directory. This produces configuration files that have literal numbers and strings as values.
Shibblleth 3 fully embraces the Spring Framework, which has its own "variable substitution from parameter file" capability called Spring Execution Language. At startup, Spring replaces %{property.name} placeholders in its XML files with the corresponding values from the shibboleth-idp/conf/idp.properties file (and other *.properties files in the conf subdirectory).
Of course, Spring will also use literal values. So we have a choice to do the property substitution at Ant install time or at Spring initialization time. Both should work. When we decide to use the Spring way of doing business, the problem is to get a property value defined in one of the Install Job property files transferred to the idp.properties file used by the Spring runtime. To do this the idp.properties file is itself in the Install Project as resources/updateArtifactSubstituteVariables/conf/idp.properties, and because it is in the SubstituteVariable directory at Install time an ${foo} expressions get replaced. So in Source Control Yale adds lines to that file of the form:
property.name=${property.name}
Then as this file is copied from the SubstituteVariables section of the Install Project to the /apps/shibboleth-idp/conf directory, the "${property.name}" is replaced by a string or number from the Install job properties files. Strictly speaking, it is not necessary for the Spring property name (on the left of the =) to be the same as the Install Project property name (on the right side) but it is simpler to remember when they are. This means that you have to be sensitive to the fact that %{foo} expressions in the resources/updateArtifact directory (which are not replaced by properties) are copied unchanged to shibboleth-idp and are then replaced logically at Spring startup from the values in idp.properties that are replaced during Install.
When in doubt, look at the files after running Install in a Sandbox and compare them to the Install project to see which change and which remain the same.
DataSources
In Shibboleth 2 a Datasource (a database connection pool configuration specifying the database url, jdbc class name, userid, and password) could be configured in a DataConnector element of the attribute-resolver.xml or it could be supplied by the container (the Web Server which is to say Tomcat or JBoss). There was, however, a scary comment in the Wiki that if you use Oracle as your database (which we did) then the Shibboleth provided DataConnector implementation did not work well and they recommended using a container datasource. So we used all ContainerManaged datasources.
A Datasource maintains a pool of connections. Sometimes they get old and are closed at the database end. Sometimes the database goes down and breaks the connection. Before passing a connection to Shib, the Datasource is supposed to check to make sure it is usable. This is not a trivial job, and Apache, JBoss, Spring, and other environments have code (sometimes vendor specific code) to get it right. Shib is not in the generic JDBC business, so you will generally get more reliable behavior if you use code from someone who put a lot of effort into getting it exactly right.
For the first cut, Shibboleth 3 continues to use ContainerManaged datasources. However, we now run under Tomcat instead of JBoss, and Tomcat uses Apache datasource management which is OK but not really as good as JBoss. The solution will be to move from ContainerManaged datasources to a new option that was added by Shib 3 where the Datasources can be managed by Spring.
Tomcat managed datasources are configured in the context.xml file that is copied (with variable substitution) to the /apps/tomcat/current/conf/catalina/localhost/idp.xml file by the Install Project.However, because Shibboleth 3 is not fully a Spring Framework project, there is an alternative. The datasource can be configured as a Spring Bean. Spring has serious industrial strength support for DataSources that is as good or better than JBoss or Tomcat. Moving the datasources to a Spring Bean configuration is planned for a future (but relatively soon) releaseidp.xml file by the Install Project.
Spring managed datasources will be defined as Beans in one of the Bean XML files.
Logging
Logging in Shibboleth 3 is controlled by the conf/logback.xml file. Initially it is treated as a normal configuration file. However, we may want to create a special Jenkins Install runtype to change the logging levels to get more information from a running server, provided that the logging levels can change dynamically.
...