SAML Messages
Shibboleth generates a SAML message (called a "SAML Response") that it sends to eachr of our Service Providers (such as box, google, service-now, archer, and so on). There is a tiny amount of important content in the Response buried in a large amount of meaningless XML junk. If the Service Provider accepts the message, then the user logs on to that service.
...
PREPROD is necessary for final testing where it is not good enough to simply "eyeball" the SAML XML, but you really need to go all the way to login to the Service Provider and verify that you get the right account contents. This isn't necessary for ordinary day to day testing of the configuration of Service Providers, but it is necessary if someone makes a change to Shibboleth itself or do a database like IDR that provides a lot of Shibboleth data.
IdP Initiated Login
An "IdP Initiated" test occurs when you click a URL that points to our Shibboleth server and provides the EntityID of one of our Service Provider partners:
...
At the end, after "providerId=" you add the EntityID of a partner. It is a Yale testing convention that the ID "nobody.yale.edu" is configured to release a very large number of attributes including most of the interesting ones you might want to check. It is the first test to make because is does so much it will point out most errors.
After generating the SAML Response XML, Shibboleth will try to send it to the AssertionConsumerService URL configured in the Metadata for Entity "nobody.yale.edu". This ACS URL is bogus (literally, it is "http://www.yale.edu/bogus") and the message goes nowhere. If you are running on DEV or the Sandbox, before the SAML is sent you will get the "Consent" page where Shibboleth lists all the attributes and values it is about to send, and many problems can be debugged from that page without ever actually sending out the SAML. Otherwise, using the SAML Trace plugin in Firefox you can trap the SAML sent to the bogus address and then inspect it for problems.
Change the EntityID at the end of the URL and you can verify that the right attributes are sent to any particular Service Provider.
SP Initiated
The hard part occurs when you have to go to the Service Provider first, and it has to send you back to Shibboleth. In this case the Service Provider will have generate a SAML Request message that Shibboleth receives. Shibboleth requires that the Request be valid. It performs essentially the same checks that we described above when the Service Provider was validating the SAML Response generated by Shibboleth. I will now copy that list of checks and note the differences.
In order for the SAML Request to work, it has to pass a shorter list of checks:
...
When a browser sends this "Unsolicited/SSO" URL to Shibboleth, the code that receives it does not inspect the request. It does not care that the URL says "localhost:8080". Internally Shibboleth generates a request that might have come from the Service Provider, then it processes it normally, finds the attribute values, generates the SAML Response, and sends it to the Service Provider.
This can only work if the Service Provider will accept a SAML Response that does not actually respond to a real SAML Request it previously generated.
Shibboleth will generate the SAML Response from the databases it is configured to use. It will put its own configured EntityID in the Response, and it will digitally sign it with the shibboleth-idp/credentials/idp.key file it has been given. It will then send the Response using Browser POST protocol through a Form it writes to the Browser to the AssertionConsumerService URL configured in the Metadata for the EntityID identified in the URL.
For initial basic testing, it is not necessary for this Response to actually get to anyone. If it does, it isn't necessarily a problem if it arrives with the wrong credentials and the SP rejects the digital signature. If you have installed the SAML Trace plugin in Firefox, it will capture a copy of the Response and you can review it to make sure all the content seems reasonable.
However, for final 100% coverage testing, then you should use production credentials to generate a SAML Response the Service Provider will actually accept and log you on. Of course, for that to work the URL has to point to PREPROD Shibboleth, either directly through the F5 or through an SSH tunnel from localhost.
Change the EntityID at the end of the URL and you can verify that the right attributes are sent to any particular Service Provider.
SP Initiated
The hard part occurs when you have to go to the Service Provider first, and it has to send you back to Shibboleth. The generation of the Response will be the same as with IdP Initiated login above. The problem is that before the Response is generated, in fact before Shibboleth even sends the Browser to CAS to login, it first validates the SAML Request it got from the Service Provider. Shibboleth rigorously follows the SAML rules, and there is no good way to tell it to relax the checks. So for Shibboleth to even begin processing, it has to pass a subset of the same sort of tests the SP preformed on the Request:
- The EntityID to which the Request is addressed has to match the EntityID configured to this Shibboleth server through the "idp.entityid" property on the Jenkins Install. Because SPs are almost always configured to talk only to PROD, that means the Shibboleth has to be installed with "idp.entityID= https://auth.yale.edu/idp/profile/SAML2/POST/SSO" or "httpsshibboleth".
- The Service Provider will tell the Browser to send the Request (either by POSTing a form or by Redirect on a GET) to the SSO URL that it has in the Metadata we provided it to use for the Yale IdP. That will be a production URL and it will begin with "https://auth.yale.edu/idp/profile/SAML2/Redirect/SSO"). This has to exactly match a URL that Shibboleth constructs from its environment using a utility class that gets some of its information for the Host header and some from the HTTPServeletRequest object passed by Tomcat.
...
- ..." This means that the Browser (or the test computer on which the Browser is running
...
- ) has to trap requests sent to "auth.yale.edu" and send
...
- them somewhere else. How this is done will be described below.
- Shibboleth will validate the digital signature on the Request, but that will always work because the SP is genuine.
- In addition to the EntityID, the XML in the Request contains the SSO URL for Shibboleth configured to the SP in Yale's Metadata "https://auth.yale.edu/idp
...
- /profile/SAML2/POST/SSO" or "https://auth.yale.edu/idp/profile/SAML2/Redirect/SSO". Shibboleth uses some utility classes to generate a URL from the HTTPServletRequest object that Tomcat passed to it, and the generated URL has to exactly match the string in the XML.
The hard part is the last test. You can download the Shibboleth source and find the code that generates the URL and the other code that compares the two strings. Some of the data comes from the HTTP Host header sent to Shibboleth, and some comes from Tomcat. If the two strings don't match, Shibboleth writes an error message to the log containing the two strings it is comparing. You cannot change the string the SP sends in the XML, which will always be "https://auth.yale.edu/idp" is defined to a special network front end device. At Yale this is the "F5" that stands in front of all network services. The host name "auth.yale.edu" resolves to a network address that is on the F5 device. The Certificate that proves that the server is "auth.yale.edu" is installed on the F5, not the Shibboleth server VM. So Shibboleth (and pretty much every other network function) cannot generally expect to find from the computer on which it runs the host name that everyone else in the world uses to get to it. For the Shibboleth code to even know about "auth.yale.edu", then either it has to have this name configured as one of its properties provided by its Install Project, or else it has to get the name from the control blocks that Tomcat provides with each incoming request/profile/SAML2/POST/SSO", so you have to change the way the test service is configured and the network path between the browser and the test server to fix the problem if the non matching string in the log is:
- "http://localhost:8080/" - The Host header contains the network address you are actually using instead of the fake network address you are trying to pretend to use. Change the hosts table or use Charles proxy to get a Host header with "auth.yale.edu".
- "http://auth.yale.edu" - Better, but Tomcat has decided this is not https
- "http://auth.yale.edu:8080" - Same thing, but now you have both the scheme (http) and the port (8080) screwed up.
You can spend many days trying to fix these problems, so the rest of this article is to explain how the problems arise and how networks are set up in general, and then the Client Setup article will explain the tools used to circumvent the problems.
Proxies and Tunnels
The first thing to say is that in almost no modern network is the hostname of the server that the Browser uses to get to Shibboleth ("auth.yale.edu") the actual name of the VM on which Shibboleth runs. In fact, it is almost never possible for the Browser to actually talk to Shibboleth directly. For security reasons, the network addresses and firewalls are set up to block outside access to server VMs, and a front end proxy machine provides the connection between the Browser and the Server. At Yale, that Proxy machine is called the F5.
Proxies have been part of HTTP since the early days of the World Wide Web. Originally they were used when the Internet was slow and it did not make sense for hundreds of computers at Yale to each independently download their own copies of widely used web pages. Today, you can put a proxy:
- On the same computer as the Browser, but this is typically used only for testing.
- Somewhere on campus to keep local copies of commonly used pages (the original use but the internet is faster today and this is not required any more)
- Between the secure environment of the servers in the machine room and the outside world of untrusted requesters.
- On the same machine as the server, because Tomcat runs as an ordinary user and you need the proxy to bind to ports 80 or 443, which require more privilege than Tomcat has.
As a developer testing code, you probably only have control over the first type of Proxy, and if you chose to use it then we recommend Charles Web Debugging Proxy as the simplest tool.
However, if you run a test and the SAML Request send by the SP is being rejected because the generated string does not match the expected URL, then something is going wrong and it could be with any of the other intermediate devices or it could be a Tomcat configuration problem. So you have to understand something about how this stuff works in order to debug it.
The good news is that Tomcat fully expects that one or more proxies will exist between the Browser and the server VM, so it is quite flexible in its configuration. Shibboleth is not as flexible, but the fact that it can work at all means that if you get the proxies configured right then everything will be OK.
When the Browser generates any HTTP request, it creates a Host header. This header contains the protocol, hostname, and port number from the URL that the Browser is using to get to the application. For a request that thinks it is going to production Shibboleth, this is
...