Skip to end of metadata
Go to start of metadata

You are viewing an old version of this content. View the current version.

Compare with Current View Version History

« Previous Version 6 Next »

Generally the Developer has little control over the setup of the Server and Network. Operations configures all of the network devices, the operating system, and even the Tomcat server. However, configuration problems can cause test failures, and the only way to fix these issues is to get the configuration fixed. Generally you only have to review this material when there is a major change to the network or server setup.

SAML Messages

The most important part of any Shibboleth login to an application occurs as it generates a "SAML Response" XML message and sends it to the Service Provider. The message contains a large amount of XML boiler plate and a very small amount of important content.

In order for Shibboleth to generate and for the Service Provider to accept the SAML Response, it has to pass a long list of checks:

  • The EntityID of the Service Provider has to be matched in one of the Metadata files configure to Shibboleth
  • The Metadata file has to provide an AssertionConsumerService (ACS) URL to which the Response can be sent.
  • The Digital Signature generated by Shibboleth has to be validated by the Public Key or Certificate configured for Yale in the Service Provider application and the EntityID name that Shibboleth uses to identify itself also has to match. In practice, this means that a Service Provider configure with PROD Shib has to get a message from a server configured with the private key credentials and EntityID of PROD Shib, while a SP that expects a message from TEST needs a message signed with the TEST key and sent with the TEST EntityID.
  • The message will contain the EntityID and ACS URL of the Service Provider. SAML messages have an implied "eyes only" rule, so these two strings have to exactly match the EntityID and ACS URL that the Service Provider expects. Since these two values came from a Metadata file the SP sent us, this check is usually never a problem.
  • The message does not contain any information about the network address or server name of the Shibboleth server. So it is not important that Shibboleth run on any particular machine.
  • The SAML Response must contain any attributes that the Service Provider has told us are required. Generally speaking, across any change to Shibboleth the value of each attribute should not change. If the user changes their name or email address, then Shibboleth will start sending the new attributes, but the attributes should not change if Shibboleth switches to a new database or we put in a new version of Shibboleth.
  • Each SAML message contains a Subject. In most cases the Subject is ignored, but there are a few special cases that also turn out to be really important cases (Google for EliApps, Service-Now) where the Subject is the only field and it is special and must have the same value each time.

The most common Shibboleth testing occurs when we have a new Service Provider to support. They will provide us with Metadata and a list of attributes they need us to supply.

If the attributes are already defined for someone else, all we need to do is to add the Metadata and to add or modify an entry in the attribute-filter.xml to release the attributes to this EntityID. Generally we can set this up in Sandbox or DEV testing, use the IdP Initiated Login URL with this EntityID, and then capture the Response. Correct behavior can be determined by an inspection of the Response. Then the code can move to PREPROD and a real login to the application becomes possible.

Occasionally someone asks for something new, or for a new version of something we are already generating. The Law School asks, "send us the list of groups in the Law School OU that contain this user as a member." We may have previously generated a list of all group membership, but now we need to filter it for just Law School groups and that is new. So now you create the new attribute, release it to this partner, and again manually check the generated SAML message to see if the right values were generated.

Attributes have a formal unique name, a friendlyName (short nickname), and a single value or occasionally a list of values. A simple example (removing all the XML boilerplate) would be:

FirstName: John
LastName: Doe
Email: John.Doe@yale.edu
eduPersonPrincipalName: jd345@yale.edu

Different partners can ask for the same attribute with different names. So "John" can be a FirstName, First_Name, givenName, and so on. So you have to generate both the right name and the right value for each item.

When a new Service Provider appears, it is convenient to initially ask them to configure the EntityID and Public Key Certificate of ourTEST Shibboleth. Then we can verify the attribute list, attribute names, and values and verify a login to the application. When it is all working, then we promote the changes to Production Shibboleth and ask the Service Provider to change EntityID and Certificate configuration to point to PROD. After a few days, we lose our contacts with the person at Yale coordinating the installation and the contact at the Service Provider who is making any changes.

Very few Service Providers bother to have a permanent configuration for our TEST Shibboleth. InCommon doesn't even have a concept of configuring TEST Identity Providers to anyone. So if after a few years we need to upgrade Shibboleth and test the new release, we have to do it with all of our Service Providers configured to use only auth.yale.edu and the production Shibboleth credentials. This explains the need for a PREPROD Shibboleth.

PREPROD is a Shibboleth IdP server on a VM that has the same /credentials subdirectory (and therefore the same private key) as PROD Shibboleth. If it generates a SAML Response, all of our Service Providers will accept it as genuine. It uses the production databases. It has been installed passing a property with the production EntityID. The only difference is that it is not actually at the IP address that the DNS server reports to be "auth.yale.edu" and if a message goes through the F5 to get to it, then it will use a separate (but hopefully equivalent) configuration so it is handled the same as production.

If we can arrange that on one computer in one Browser, all traffic addressed to production Shibboleth goes instead to PREPROD, and all messages back from PREPROD are sent on to the Service Provider, then we can verify that the new code running on PREPROD works before the same code is installed on PROD.

PREPROD is necessary for final testing where it is not good enough to simply "eyeball" the SAML XML, but you really need to go all the way to login to the Service Provider and verify that you get the right account contents. This isn't necessary for ordinary day to day testing of the configuration of Service Providers, but it is necessary if someone makes a change to Shibboleth itself or do a database like IDR that provides a lot of Shibboleth data.

IdP Initiated Login

An "IdP Initiated" test occurs when you click a URL that points to our Shibboleth server and provides the EntityID of one of our Service Provider partners:

http://localhost:8080/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu

The hostname part ("localhost:8080") could reference a Sandbox test Shibboleth running on Tomcat on your desktop, or it could be a test version of Shibboleth running on a VM in the machine room where you have created an SSH tunnel connecting port 8080 on your desktop to the real Server port on the real Server VM.

At the end, after "providerId=" you add the EntityID of a partner. It is a Yale testing convention that the ID "nobody.yale.edu" is configured to release a very large number of attributes including most of the interesting ones you might want to check. It is the first test to make because is does so much it will point out most errors.

When a browser sends this "Unsolicited/SSO" URL to Shibboleth, the code that receives it does not inspect the request. It does not care that the URL says "localhost:8080". Internally Shibboleth generates a request that might have come from the Service Provider, then it processes it normally, finds the attribute values, generates the SAML Response, and sends it to the Service Provider.

This can only work if the Service Provider will accept a SAML Response that does not actually respond to a real SAML Request it previously generated.

Shibboleth will generate the SAML Response from the databases it is configured to use. It will put its own configured EntityID in the Response, and it will digitally sign it with the shibboleth-idp/credentials/idp.key file it has been given. It will then send the Response using Browser POST protocol through a Form it writes to the Browser to the AssertionConsumerService URL configured in the Metadata for the EntityID identified in the URL.

For initial basic testing, it is not necessary for this Response to actually get to anyone. If it does, it isn't necessarily a problem if it arrives with the wrong credentials and the SP rejects the digital signature. If you have installed the SAML Trace plugin in Firefox, it will capture a copy of the Response and you can review it to make sure all the content seems reasonable.

However, for final 100% coverage testing, then you should use production credentials to generate a SAML Response the Service Provider will actually accept and log you on. Of course, for that to work the URL has to point to PREPROD Shibboleth, either directly through the F5 or through an SSH tunnel from localhost.

Change the EntityID at the end of the URL and you can verify that the right attributes are sent to any particular Service Provider.

SP Initiated

The hard part occurs when you have to go to the Service Provider first, and it has to send you back to Shibboleth. The generation of the Response will be the same as with IdP Initiated login above. The problem is that before the Response is generated, in fact before Shibboleth even sends the Browser to CAS to login, it first validates the SAML Request it got from the Service Provider. Shibboleth rigorously follows the SAML rules, and there is no good way to tell it to relax the checks. So for Shibboleth to even begin processing, it has to pass a subset of the same sort of tests the SP preformed on the Request:

  • The EntityID to which the Request is addressed has to match the EntityID configured to this Shibboleth server through the "idp.entityid" property on the Jenkins Install. Because SPs are almost always configured to talk only to PROD, that means the Shibboleth has to be installed with "idp.entityID= https://auth.yale.edu/idp/shibboleth".
  • The Service Provider will tell the Browser to send the Request (either by POSTing a form or by Redirect on a GET) to the SSO URL that it has in the Metadata we provided it to use for the Yale IdP. That will be a production URL and it will begin with "https://auth.yale.edu/idp/..." This means that the Browser (or the test computer on which the Browser is running) has to trap requests sent to "auth.yale.edu" and send them somewhere else. How this is done will be described below.
  • Shibboleth will validate the digital signature on the Request, but that will always work because the SP is genuine.
  • In addition to the EntityID,  the XML in the Request contains the SSO URL for Shibboleth configured to the SP in Yale's Metadata "https://auth.yale.edu/idp/profile/SAML2/POST/SSO" or "https://auth.yale.edu/idp/profile/SAML2/Redirect/SSO". Shibboleth uses some utility classes to generate a URL from the HTTPServletRequest object that Tomcat passed to it, and the generated URL has to exactly match the string in the XML.

The hard part is the last test. You can download the Shibboleth source and find the code that generates the URL and the other code that compares the two strings. Some of the data comes from the HTTP Host header sent to Shibboleth, and some comes from Tomcat. If the two strings don't match, Shibboleth writes an error message to the log containing the two strings it is comparing. You cannot change the string the SP sends in the XML, which will always be "https://auth.yale.edu/idp/profile/SAML2/POST/SSO", so you have to change the way the test service is configured and the network path between the browser and the test server to fix the problem if the non matching string in the log is:

  • "http://localhost:8080/" - The Host header contains the network address you are actually using instead of the fake network address you are trying to pretend to use. Change the hosts table or use Charles proxy to get a Host header with "auth.yale.edu".
  • "http://auth.yale.edu" - Better, but Tomcat has decided this is not https
  • "http://auth.yale.edu:8080" - Same thing, but now you have both the scheme (http) and the port (8080) screwed up.

You can spend many days trying to fix these problems, so the rest of this article is to explain how the problems arise and how networks are set up in general, and then the Client Setup article will explain the tools used to circumvent the problems.

Proxies and Tunnels

The first thing to say is that in almost no modern network is the hostname of the server that the Browser uses to get to Shibboleth ("auth.yale.edu") the actual name of the VM on which Shibboleth runs. In fact, it is almost never possible for the Browser to actually talk to Shibboleth directly. For security reasons, the network addresses and firewalls are set up to block outside access to server VMs, and a front end proxy machine provides the connection between the Browser and the Server. At Yale, that Proxy machine is called the F5.

Proxies have been part of HTTP since the early days of the World Wide Web. Originally they were used when the Internet was slow and it did not make sense for hundreds of computers at Yale to each independently download their own copies of widely used web pages. Today, you can put a proxy:

  • On the same computer as the Browser, but this is typically used only for testing.
  • Somewhere on campus to keep local copies of commonly used pages (the original use but the internet is faster today and this is not required any more)
  • Between the secure environment of the servers in the machine room and the outside world of untrusted requesters.
  • On the same machine as the server, because Tomcat runs as an ordinary user and you need the proxy to bind to ports 80 or 443, which require more privilege than Tomcat has.

As a developer testing code, you probably only have control over the first type of Proxy, and if you chose to use it then we recommend Charles Web Debugging Proxy as the simplest tool.

However, if you run a test and the SAML Request send by the SP is being rejected because the generated string does not match the expected URL, then something is going wrong and it could be with any of the other intermediate devices or it could be a Tomcat configuration problem. So you have to understand something about how this stuff works in order to debug it.

The good news is that Tomcat fully expects that one or more proxies will exist between the Browser and the server VM, so it is quite flexible in its configuration. Shibboleth is not as flexible, but the fact that it can work at all means that if you get the proxies configured right then everything will be OK.

When the Browser generates any HTTP request, it creates a Host header. This header contains the protocol, hostname, and port number from the URL that the Browser is using to get to the application. For a request that thinks it is going to production Shibboleth, this is

Host: https://auth.yale.edu

This header can go through the network and the F5 unmodified, or it can be changed and there are other network standards for other headers that might be generated by the F5 or any other device that receives, modifies, and forwards the request. The people who configure these network devices know all these conventions, and production Shibboleth receives data through a similar path with a similar set of header modifications.

So if you trap a Service Provider generated SAML Request and decide to forward it to a test Shibboleth server, things will work correctly provided that the Shibboleth server is configured with the same EntityID as production and that when the SAML Request arrives at the test server, passes through Tomcat, and is presented to Shibboleth that the Host header (and any other headers commonly generated by machines like the F5) convince Shibboleth that this request was originally addressed to "https://auth.yale.edu/idp".

Although there will be at any time one or more specific network configurations for accomplishing this result, there are lots of rules and tricks and software that can accomplish the same thing in various ways. The thing to remember is that if you try to login to a Service Provider with a test Shibboleth, and instead of getting a CAS login you get a message saying that the request did not meet security restrictions, and the Shibboleth log contains an error message saying that the message was addressed to "https://auth.yale.edu/idp" but Shibboleth has decided that its own network address is "http://auth.yale.edu/idp" (http not https), or "https://auth.yale.edu:8080/idp" (a port number got added), or "http://localhost:8080/idp" (you didn't fake anything out at all), then you have not set the HTTP headers and the network path right and you have to map out all the intermediate boxes and tunnels through which the request is passing to figure out what is wrong or missing.

The problem is that some of this path may not be under the control of the developer or tester, and if you try and create a path that is completely under your control you now have to learn more about the configuration of Tomcat and HTTP proxy tools, which are normally a problem for someone else.

Some successful recipes will be provided in the Testing Setup document, but if you decide to deviate from them you need to understand the problem described above and figure out new solutions.

  • No labels