...
Now we get to the last test, and that which is the hardest harder and requires a lot more description than the previous simple testslonger description. The SAML protocol requires a Request to have internal XML fields containing both the EntityID of the IdP include the URL to which the message Request is sent and also the Single Sign On URL to which it is being sent. Assuming the SP uses our production Metadata, it will include in the XML one of two URLs, to also be data in an internal XML field of the Request, and it requires the IdP to check that this field is correct. Depending on whether the Request is sent as a POST or a Redirect, the URL for PROD Shib will be "https://auth.yale.edu/idp/profile/SAML2/POST/SSO" or "https://auth.yale.edu/idp/profile/SAML2/Redirect/SSO", depending on which of two mechanisms it uses to send the Request.
Whatever mechanism you used in the Browser to rewrite or reroute this URL is not going to change the second copy of the same URL in the XML, and the problem is that SAML requires and Shibboleth implements a check that the URL in the XML is the same as the URL that Shibboleth believes was used to deliver the Request.
You can download the Shibboleth source and find the code that generates the URL and the other code that compares the two strings. Some of the data comes from the HTTP Host header sent to Shibboleth, and some comes from Tomcat. If the two strings don't match, Shibboleth writes an error message to the log containing the two strings it is comparing. You cannot change the string the SP sends in the XML, which will always be "https://auth.yale.edu/idp/profile/SAML2/POST/SSO", so you have to change the way the test service is configured and the network path between the browser and the test server to fix the problem if the non matching string in the log is.
Of course, this URL will also have been included in the HTTP headers, and previously some mechanism was used to rewrite or reroute the HTTP flow to the test Shibboleth. That did not change the XML content. So now the problem is to ensure that the URL that the test Shibboleth server receives matches a URL that it reconstructs using the Host header and the HTTPServletRequest object it receives.
Of course, with open source you can download and read the Java code to see exactly how it is done. You can even find the Spring Bean statement that configures the comparison and replace it with new code that relaxes the rules. However, generally speaking if the Host header contains "https://auth.yale.edu" and Tomcat believes that the message arrived over SSL (at least as far as the proxy) then the comparison will work. So you will have a problem when you are trying to handle an SP Initiated Login and either:
- You use Redirector to rewrite the server URL before the Host header is generated instead of using the hosts file or Charles to reroute the request while preserving the "auth.yale.edu" hostname.
- You send the data over an SSH tunnel directly to non SSL port 8080 without any Proxy protocol or Tomcat configuration that says the data was originally SSL.
If the URL in the XML and the URL that Shibboleth generates don't match, Shibboleth writes an error message to the log containing the two strings it is comparing. You know you have to fix your client configuration when one of the two strings reported in the log contains:
- "http://localhost:8080/" - The You have both problems, Redirector has rewritten the Host header contains the network address you are actually using instead of the fake network address you are trying to pretend to use. Change the hosts table or use Charles proxy to get a Host header with "auth.yale.edu".and you have a tunnel to 8080.
- "http://auth.yale.edu" - BetterThe host header is right, but Tomcat has decided doesn't think this is not https.
- "httphttps://auth.yale.edu:8080" - Same thingTomcat knows this is https, but now you have both the scheme (http) and the port (8080) screwed up.
You can spend many days trying to fix these problems, so the rest of this article is to explain how the problems arise and how networks are set up in general, and then the Client Setup article will explain the tools used to circumvent the problems.
- the host header had a port number inserted.
The Client Setup article will provide some configurations that are known to work. Others may also work, but you may take some time with trial and error trying to fix the last problems.
Background on Proxies and Tunnels
The first thing to say is that in almost no modern network is the hostname of the server that the Browser uses to get to Shibboleth ("auth.yale.edu") the actual name of the VM on which Shibboleth runs. In fact, it is almost never possible for the Browser to actually talk to Shibboleth directly. For security reasons, the network addresses and firewalls are set up to block outside access to server VMs, and a front end proxy machine provides the connection between the Browser and the Server. At Yale, that Proxy machine is called the F5.
...
- On the same computer as the Browser, but this is typically used only . The Charles Web Debugging Proxy is a useful tool for testing.
- Somewhere on campus to keep local copies of commonly used pages (the original use but the internet is faster today and this is not required any more)
- Between You can still find a client oriented proxy somewhere in the campus network, but this is no longer common or useful.
- The F5 sits between the secure environment of the servers in the machine room and the outside world of untrusted requesters.On the same machine as the server, because Tomcat runs as an ordinary user and you need the proxy to bind to ports 80 or 443, which require more privilege than Tomcat has
- A front end (Nginx or Apache Web Server) can receive requests on ports 80 and 443 on the Shibboleth VM and they can forward those requests to Tomcat.
As a developer testing code, you probably only have control over the first type of Proxy, and if you chose to use it then we recommend Charles Web Debugging Proxy as the simplest tool.
...
This header can go through the network and the F5 unmodified, or it can be changed and there . There are other network standards for other headers that might be generated by the F5 or any other device that receives, modifies, and forwards the request. The people who configure these network devices know all these conventions, and production Shibboleth receives data through a similar path with a similar set of header modifications.
So if you trap a Service Provider generated SAML Request and decide to forward it to a test Shibboleth server, things will work correctly provided that the Shibboleth server is configured with the same EntityID as production and that when the SAML Request arrives at the test server, passes through Tomcat, and is presented to Shibboleth that the Host header (and any other headers commonly generated by machines like the F5) convince Shibboleth that this request was originally addressed to "https://auth.yale.edu/idp".
Although there will be at any time one or more specific network configurations for accomplishing this result, there are lots of rules and tricks and software that can accomplish the same thing in various ways. The thing to remember is that if you try to login to a Service Provider with a test Shibboleth, and instead of getting a CAS login you get a message saying that the request did not meet security restrictions, and the Shibboleth log contains an error message saying that the message was addressed to "https://auth.yale.edu/idp" but Shibboleth has decided that its own network address is "http://auth.yale.edu/idp" (http not https), or "https://auth.yale.edu:8080/idp" (a port number got added), or "http://localhost:8080/idp" (you didn't fake anything out at all), then you have not set the HTTP headers and the network path right and you have to map out all the intermediate boxes and tunnels through which the request is passing to figure out what is wrong or missing.
The problem is that some of this path may not be under the control of the developer or tester, and if you try and create a path that is completely under your control you now have to learn more about the configuration of Tomcat and HTTP proxy tools, which are normally a problem for someone else.
Some successful recipes will be provided in the Testing Setup document, but if you decide to deviate from them you need to understand the problem described above and figure out new solutionsheaders that proxies frequently generate to indicate processing they have done. There are standards and if you have a couple of days you might find and read through all of them.
However, there are two basic rules. First, if the Browser thinks it is talking to production Shibboleth, and the rerouting is done in the network configuration, then one solution is to leave the original Host header alone. On the other hand, if the Browser knows that the test Shibboleth is really at a different URL, then it might be a good idea to have something in the proxy path that changes the Host header so it has the auth.yale.edu value shown above. In other words, if you know it is right leave it alone, and if you know it is wrong, change it.
It is trivial for Operations to fix the header, but if they are not already doing it, this will be more trouble than it is worth. Configure the client to generate the right header in the first place.
If the header is right coming out of the client, and it is being set incorrectly in the proxy chain, then you have to get Operations to fix it.