...
The SAML Single SignOn protocols exchange data between the Application and Shibboleth through the Browser. We do not use the protocols where the Application talks to Shibboleth directly. Therefore, a test environment can be built completely on the desktop computer of the tester. Data can be traced as it passes through the Browser, and traffic can be trapped on the desktop and redirected to the test Shibboleth server.
SAML Messages
Shibboleth generates a SAML message that it sends to any of our Service Providers (box, google, service-now). There is a small amount of important content buried in a large amount of meaningless technical junk. Any testing strategy requires you to know what to test, and that means knowing what is in the message.
The first point about any message is that they have a to: and a from: field. In SAML messages, each end is known by a unique character string they make up called an EntityId, and then configured for this EntityId there is also a network address of a server to which the message is sent. The Google Apps login service has a simple EntityId of "google.com", and its network address when someone logs into their EliApps mailbox is "https://www.google.com/a/yale.edu/acs". However, most parties on the internet use an EntityId that appears to be a network address in itself. The Yale Shibboleth Identity Provider has an EntityId of "https://auth.yale.edu/idp/shibboleth" and the address of its Single Sign-On service is "https://auth.yale.edu/idp/profile/SAML2/POST/SSO". So it is easy unless you look carefully to confuse the EntityId character string with the actual login URL.
The message has a subject. The only requirement is that it be different for every person. It could be a username, employee or student ID number, email address, or just a randomly generated unique string. For most partners it is unimportant, but for the other partners it is the only important thing in the message. So you have to evaluate it on a case by case basis.
Then there are a set of attributes. You can think of them as simply:
FirstName: John
LastName: Doe
Email: John.Doe@yale.edu
eduPersonPrincipalName: jd345@yale.edu
Since most applications ask us for the same half dozen attributes, almost no testing is required. We are already generating the attributes for other applications, so we just configure the new EntityId and then verify that Shibboleth is generating the attributes requested.
Things are more interesting if the application requires a new attribute, or sometimes an existing attribute but with an unusual name. We may ask them to configure their application with the EntityId and network address of our TEST Shibboleth instance. Then we debug their application until we can login. Then we ask them to change their configuration to use the EntityId and network address of production Shibboleth.
The problem occurs when we are making a major change to the Shibboleth Identity Provider itself. We cannot go back and ask all these vendors to change their configuration of Yale. We don't always have contact information. So we have to accept that they are probably configured with the EntityId and network address of our production Shibboleth. Then we have to figure out how to route the request they make for production Shibboleth data to a test machine.
During initial testing all we need to verify is that the new code is generating the same Attribute names and values, and maybe the same Subject value as the old code. This part doesn't actually require that the message actually be sent to the application, of if it is sent it doesn't matter if the application rejects it because of a bad digital signature. The tester simply traps the message that was generated, examines it, and verifies that the attribute names and values are correct.
However, during the final phase of testing there is really no substitute for actually logging in to the application. That can only be done using a message generated with the credentials from our production server, and that typically means that it has to be generate by a pre-production test VM in the machine room that has a copy of the real production key file.
The good news is that all Shibboleth traffic passes through the browser. It is fairly easy to install some simple testing tools on your desktop to intercept traffic to or from auth.yale.edu and to route it to a different network address. It is a bit more complicated to configure the test machine so that it behaves exactly like production. It is easy to tell the pre-production test machine that its EntityId is the same character string as production. It is easy to give it the same key file so it generates the same digital signatures. It is trivial to give it the same access to the same databases.
The problem is faking out the network address. Production Shibboleth is accessed with a URL that begins "https://auth.yale.edu/idp/...". That is not actually the name of the computer or VM that contains the Shibboleth server. It is a DNS name associated with a Virtual IP address exposed by our F5 front-end network device in the machine room. The F5 forwards any messages that it receives to the VM that actually runs Shibboleth.
Today, everybody runs computer services this way. Every company or university has a front end load balancer and firewalls and the other elements of modern network infrastructure. As a result, the Tomcat web server and Shibboleth expect to run in an environment where the network name that they have given the world, the name that everyone uses to get to them, is not actually the name of the computer on which the Identity Provider runs. However, the tricks that make this work depend on the configuration of the front end and network and firewalls.
If you are simply testing a new version of Shibboleth, and you can reproduce all the network structure so the test machine is exactly the same as production Shibboleth, then testing is simple. However, if you are testing a change to the network structure or the Web Server under which Shibboleth runs, then you need to be careful. Dozens of partners out in the cloud need to be testing. Each of them believes that the Shibboleth IdP is on "https://auth.yale.edu/idp" but you need to make some change so that URL is going to go to a different server on the network. You have to be sure that if you are both rerouting the message through the network to a different address than the production server, and at the same time the path through the network to that address is configured differently than the production path, that the combination of the test redirection and the network change do not interact. This will be a particular problem if the resources provided for the test environment are different (less sophisticated, less expensive) than the resources provided for the real production system.
IdP Initiated Logon
About 95% of the applications that use Shibboleth support "IdP Initiated" logon, where the Browser starts with a URL that points to Shibboleth and Shibboleth sends a SAML Response unsolicited to the Application. The starting URL is of the form:
https://auth.yale.edu/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu
When the Browser generates a GET to this URL, the handler for profile/SAML2/Unsolicited/SSO searches the Metadata files for an EntityId that matches the value in the providerId at the end of the URL. There has to be a matching Metadata to supply the URL of the AssertionConsumerService to which the login will be sent, and the hostname in the ACS URL has to exist. Shibboleth also searches the attribute-filter.xml for a policy that releases attributes to this EntityId. It then builds a SAML Response message with the released attributes, writes it to a Form on the Browser screen where JavaScript submits the form and sends everything on to the ACS URL.
The first thing the Application sees is a Form POST from your Browser to the ACS URL containing the SAML Response as data.The Application does not see the IP address or host name of the computer on which Shibboleth ran. The Response will contain the EntityId that was given to Shibboleth by the "idp.entityID= https://auth.yale.edu/idp/shibboleth", which is the first line in the install-XXX.properties file in the Jenkins Install project, but that is just a character string that does not need to have any physical relationship to the actual computer hostname. The EntityId will be used by the Application to find a configured Certificate to validate the digital signature on the Request.
Unless this computer has the actual credentials of PROD Shibboleth (which means that it is either PROD or PREPROD) then it cannot generate a legitimate signature. It will use whatever key it has, but the Application will find the signature faulty and will return an error page and not log the user on. However, if you have been tracing the HTTP messages in the Browser, you will have captured the SAML message sent to the application. You can display it and verify that the Attributes and Subject are correct for that particular Application, and that is most of the testing you typically need to do.
...
IdP Initiated Logon
About 95% of the applications that use Shibboleth support "IdP Initiated" logon, where the Browser starts with a URL that points to Shibboleth and Shibboleth sends a SAML Response unsolicited to the Application. The starting URL is of the form:
https://auth.yale.edu/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu
The "providerId" parmeter in this URL corresponds to what every SAML or Shibboleth document or configuration file calls the EntityID of a Service Provider. When this URL is presented to a Shibboleth server, then the handler for the /profile/SAML2/Unsolicited/SSO path searches the configured Metadata providers for an entry describing the named EntityID. From the Metadata it obtains the default AssertionConsumerService, to which the SAML Response will be sent. Then it searches the attribute-filter.xml for a policy that releases attributes to this EntityId.
If the Consent dialog has been enabled on this Shibboleth server, the user first sees a page listing the attributes that will be sent and their values. This is often good enough for initial testing.
Shibboleth generates a Response with the released Attributes and sends it to the ACS URL. If the Shibboleth IdP does not have the real production /credentials/idp.key file installed, the Service Provider will reject the digital signature. However, if you have SAML Trace enabled in your Browser you will have captured the SAML Response and can examine all the generated XML.
The final test, however, is to really login. To do this, you need production credentials, and that only happens when you test against the PREPROD VM in the machine room. So now you change the IdP initiated URL to point to that server, either with a public URL that goes through the F5, or a private URL that goes through the VPN tunnel to the server VM, or through an SSH tunnel to port 443 on the PREPROD VM.
SP Initiated Login
The user can start by going to the application. A Browser that starts with "yale.service-now.com", "yale.box.com", "mail.bulldogs.yale.edu", or the Comcast IPTV URL will in some cases get immediately sent to Shibboleth and in some cases will push a button or select Yale University from a dropdown list before being Redirected back to Shib.
...