Versions Compared

Version Old Version 6 New Version 7
Changes made by

Howard Gilbert

Howard Gilbert

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The most important part of any Shibboleth login to an application occurs as it generates a "SAML Response" XML message and sends it to the Service Provider. The message contains a large amount of XML boiler plate and a very small amount of important content.

In order for Shibboleth to generate and for the Service Provider to accept the SAML Response, it has to pass a long list of checks:

  • The EntityID of the Service Provider has to be matched in one of the Metadata files configure to Shibboleth
  • The Metadata file has to provide an AssertionConsumerService (ACS) URL to which the Response can be sent.
  • The Digital Signature generated by Shibboleth has to be validated by the Public Key or Certificate configured for Yale in the Service Provider application and the EntityID name that Shibboleth uses to identify itself also has to match. In practice, this means that a Service Provider configure with PROD Shib has to get a message from a server configured with the private key credentials and EntityID of PROD Shib, while a SP that expects a message from TEST needs a message signed with the TEST key and sent with the TEST EntityID.
  • The message will contain the EntityID and ACS URL of the Service Provider. SAML messages have an implied "eyes only" rule, so these two strings have to exactly match the EntityID and ACS URL that the Service Provider expects. Since these two values came from a Metadata file the SP sent us, this check is usually never a problem.
  • The message does not contain any information about the network address or server name of the Shibboleth server. So it is not important that Shibboleth run on any particular machine.
  • The SAML Response must contain any attributes that the Service Provider has told us are required. Generally speaking, across any change to Shibboleth the value of each attribute should not change. If the user changes their name or email address, then Shibboleth will start sending the new attributes, but the attributes should not change if Shibboleth switches to a new database or we put in a new version of Shibboleth.
  • Each SAML message contains a Subject. In most cases the Subject is ignored, but there are a few special cases that also turn out to be really important cases (Google for EliApps, Service-Now) where the Subject is the only field and it is special and must have the same value each time.

The most common Shibboleth testing occurs when we have a new Service Provider to support. They will provide us with Metadata and a list of attributes they need us to supply.

If the attributes are already defined for someone else, all we need to do is to add the Metadata and to add or modify an entry in the attribute-filter.xml to release the attributes to this EntityID. Generally we can set this up in Sandbox or DEV testing, use the IdP Initiated Login URL with this EntityID, and then capture the Response. Correct behavior can be determined by an inspection of the Response. Then the code can move to PREPROD and a real login to the application becomes possible.

Occasionally someone asks for something new, or for a new version of something we are already generating. The Law School asks, "send us the list of groups in the Law School OU that contain this user as a member." We may have previously generated a list of all group membership, but now we need to filter it for just Law School groups and that is new. So now you create the new attribute, release it to this partner, and again manually check the generated SAML message to see if the right values were generated.

Attributes have a formal unique name, a friendlyName (short nickname), and a single value or occasionally a list of values. A simple example (removing all the XML boilerplate) would be:

FirstName: John
LastName: Doe
Email: John.Doe@yale.edu
eduPersonPrincipalName: jd345@yale.edu

Different partners can ask for the same attribute with different names. So "John" can be a FirstName, First_Name, givenName, and so on. So you have to generate both the right name and the right value for each item.

When a new Service Provider appears, it is convenient to initially ask them to configure the EntityID and Public Key Certificate of ourTEST Shibboleth. Then we can verify the attribute list, attribute names, and values and verify a login to the application. When it is all working, then we promote the changes to Production Shibboleth and ask the Service Provider to change EntityID and Certificate configuration to point to PROD. After a few days, we lose our contacts with the person at Yale coordinating the installation and the contact at the Service Provider who is making any changes.

Very few Service Providers bother to have a permanent configuration for our TEST Shibboleth. InCommon doesn't even have a concept of configuring TEST Identity Providers to anyone. So if after a few years we need to upgrade Shibboleth and test the new release, we have to do it with all of our Service Providers configured to use only auth.yale.edu and the production Shibboleth credentials. This explains the need for a PREPROD Shibboleth.

PREPROD is a Shibboleth IdP server on a VM that has the same /credentials subdirectory (and therefore the same private key) as PROD Shibboleth. If it generates a SAML Response, all of our Service Providers will accept it as genuine. It uses the production databases. It has been installed passing a property with the production EntityID. The only difference is that it is not actually at the IP address that the DNS server reports to be "auth.yale.edu" and if a message goes through the F5 to get to it, then it will use a separate (but hopefully equivalent) configuration so it is handled the same as production.

If we can arrange that on one computer in one Browser, all traffic addressed to production Shibboleth goes instead to PREPROD, and all messages back from PREPROD are sent on to the Service Provider, then we can verify that the new code running on PREPROD works before the same code is installed on PROD.

PREPROD is necessary for final testing where it is not good enough to simply "eyeball" the SAML XML, but you really need to go all the way to login to the Service Provider and verify that you get the right account contents. This isn't necessary for ordinary day to day testing of the configuration of Service Providers, but it is necessary if someone makes a change to Shibboleth itself or do a database like IDR that provides a lot of Shibboleth data.

IdP Initiated Login

An "IdP Initiated" test occurs when you click a URL that points to our Shibboleth server and provides the EntityID of one of our Service Provider partners:

http://localhost:8080/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu

The hostname part ("localhost:8080") could reference a Sandbox test Shibboleth running on Tomcat on your desktop, or it could be a test version of Shibboleth running on a VM in the machine room where you have created an SSH tunnel connecting port 8080 on your desktop to the real Server port on the real Server VM.

At the end, after "providerId=" you add the EntityID of a partner. It is a Yale testing convention that the ID "nobody.yale.edu" is configured to release a very large number of attributes including most of the interesting ones you might want to check. It is the first test to make because is does so much it will point out most errors.

When a browser sends this "Unsolicited/SSO" URL to Shibboleth, the code that receives it does not inspect the request. It does not care that the URL says "localhost:8080". Internally Shibboleth generates a request that might have come from the Service Provider, then it processes it normally, finds the attribute values, generates the SAML Response, and sends it to the Service Provider.

This can only work if the Service Provider will accept a SAML Response that does not actually respond to a real SAML Request it previously generated.

Shibboleth will generate the SAML Response from the databases it is configured to use. It will put its own configured EntityID in the Response, and it will digitally sign it with the shibboleth-idp/credentials/idp.key file it has been given. It will then send the Response using Browser POST protocol through a Form it writes to the Browser to the AssertionConsumerService URL configured in the Metadata for the EntityID identified in the URL.

For initial basic testing, it is not necessary for this Response to actually get to anyone. If it does, it isn't necessarily a problem if it arrives with the wrong credentials and the SP rejects the digital signature. If you have installed the SAML Trace plugin in Firefox, it will capture a copy of the Response and you can review it to make sure all the content seems reasonable.

However, for final 100% coverage testing, then you should use production credentials to generate a SAML Response the Service Provider will actually accept and log you on. Of course, for that to work the URL has to point to PREPROD Shibboleth, either directly through the F5 or through an SSH tunnel from localhost.

Change the EntityID at the end of the URL and you can verify that the right attributes are sent to any particular Service Provider.

SP Initiated

The hard part occurs when you have to go to the Service Provider first, and it has to send you back to Shibboleth. The generation of the Response will be the same as with IdP Initiated login above. The problem is that before the Response is generated, in fact before Shibboleth even sends the Browser to CAS to login, it first validates the SAML Request it got from the Service Provider. Shibboleth rigorously follows the SAML rules, and there is no good way to tell it to relax the checks. So for Shibboleth to even begin processing, it has to pass a subset of the same sort of tests the SP preformed on the Request:

...

A lot of Shibboleth testing involves checking the SAML Response to make sure it has the right data for the Subject and for each supplied Attribute. This is determined by the configuration files for Shibboleth. It must find the EntityID of the Service Provider in a configured Metadata file, find the necessarily attributes released in the attribute-filter xml file, generate the SAML Response, and send it to the configured "ACS" URL in the Metadata. The Service Provider will accept the message if the EntityID and digital signature of the Shibboleth server match what it has been configured to expect.

However, Shibboleth passes data back to the Browser in response to something the Browser set to it, and the subsequent transmission of the Response from the Browser to the Service Provider is unrelated to the network configuration of the Shibboleth servers. There is almost nothing in this part of the transmission to configure or test.

IdP Initiated Login

An "IdP Initiated" test occurs when you click a URL that points to our Shibboleth server and provides the EntityID of one of our Service Provider partners:

http://localhost:8080/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu

The hostname part ("localhost:8080") could reference a Sandbox test Shibboleth running on Tomcat on your desktop, or it could be a test version of Shibboleth running on a VM in the machine room where you have created an SSH tunnel connecting port 8080 on your desktop to the real Server port on the real Server VM.

At the end, after "providerId=" you add the EntityID of a partner. It is a Yale testing convention that the ID "nobody.yale.edu" is configured to release a very large number of attributes including most of the interesting ones you might want to check. It is the first test to make because is does so much it will point out most errors.

The Shibboleth code that processes an "Unsolicited/SSO" does not do any network validity checking. It doesn't care what the protocol (http or https), hostname, or port number are. However, during the processing it does Redirect the Browser to a different URL on the same Shibboleth server, and it order to do that it needs to figure out what its own URL actually is from the point of view of the Browser.

Shibboleth is running on some VM in the machine room. You could be accessing it through the public production URL ("https://auth.yale.edu/idp") but you could also have set up an SSH tunnel from your local computer port 8080 to the VM port 8080 (the port Tomcat uses). The problem is that Tomcat and Shibboleth cannot tell from the network protocol just how the network traffic is routed between the Browser and the VM. The only hint it has is the HTTP Host Header.

Initially the Browser will have generated a Host header based on the network address in the URL it started with. In these cases, the header will either be "http://localhost:8080" or else "https://auth.yale.edu". The Host header may pass through a number of intermediate devices, and some of them may edit some of the HTTP content, but the meaning of the Host header as "the network address of the server from the point of view of the Browser" should be preserved. Then when Shibboleth needs to Redirect the Browser from one of its URLs to another URL, it can use the Host header as the start of the Redirect URL and know that when the Browser can use it to get back to the same Shibboleth server.

If you submit an IdP Initiated Login to Shibboleth and immediately get a Browser redirect to a completely different URL location, then someone has messed up the Host header between the Browser and Shib.

There is a different Redirect problem when Shibboleth uses CAS to log you in. The convention with CAS is that Shib, like any other CAS client, provides a service= string that contains a URL in the Tomcat server to which CAS can Redirect the Browser after it has added the Service Ticket ID. For some reason, although the Unicon CAS-Shib itegration code has access to the Host header, it only uses that header to set the port number. The rest of the Service URL is constructed from a property that in the Yale Install Project is called "cas.target.url" and is supplied at install time.

The problem is that this property is not dynamic and it is not going to adjust between "localhost:8080" for an SSH tunnel and "auth.yale.edu" through the F5. Since it seems like a very bad practice to configure the VM with properties that will not work in production, if you discover that Shibboleth is Redirecting the Browser to CAS with a Service URL that cannot possibly get back to the Shib VM in the way you are currently accessing that VM, then you have to use one of the Client configuration tools described in the Client Configuration part of this set of documents to rewrite or reroute the Service URL (the Browser Redirector plugin can do it for example).

The IdP Initated Login provides the simplest set of tests. About 95% or more of Service Providers will accept IdP Initiated, and you should do all that testing first. It is also the simplest network setup because you only have to fix the two Redirect problems (Shib to Shib and CAS to Shib) if they happen to occur.

In fact, for initial testing it is not actually necessary for the SAML Response to be delivered to the Service Provider (or for it to be trusted if it does arrive there). If you are running the SAML Tracer in Firefox you can capture the SAML Response and usually determine if the Subject and Attributes are correct just by inspecting it.

However, for final 100% coverage testing, then you should use production credentials to generate a SAML Response the Service Provider will actually accept and log you on. Of course, for that to work the URL has to point to PREPROD Shibboleth, either directly through the F5 or through an SSH tunnel from localhost.

Change the EntityID at the end of the "Unsolicited/SSO" URL and you can verify that the right attributes are sent to any particular Service Provider.

SP Initiated

The IdP Initiated test was simplified because the code that handles the "Unsolicited/SSO" URL does not do any checks on the network address. Technically, the "Unsolicited/SSO" is a Shibboleth programming convention and not part of the SAML standard, so checks are not required.

However, if you go to the Service Provider and press a "Login from Yale" button or select Yale from a pulldown list of universities, then the SAML Request object generated by the Service Provider and the protocol by which it is sent to Shibboleth are part of the SAML standard, and that standard requires Shibboleth to do some extra checks that make the network configuration a bit more complicated.

Furthermore, while the network path from the Browser to the Service Provider (to send the Shibboleth Response) is just normal Yale networking and is not changed by the Shibboleth testing configuration, the SAML message that the SP sends to the Browser and then expects the Browser to deliver to Shib obviously is going to be affected by which Shib VM you want to use and how the Browser sends data to it.

Shibboleth processes the SAML Request first, before it even sends you to CAS in order to login. The Request has to go through a series of checks:

  • The EntityID to which the Request is addressed has to match the EntityID configured to this Shibboleth server through the "idp.entityid" property on the Jenkins Install. In the IdP Initiated testing you can use any Shibboleth (PROD, TEST, DEV, or Sandbox) and the Response will be generated and traced so you can look at its contents. It is only when the Response finally gets to the SP that it will be rejected if it comes from the wrong Shibboleth. But with an SP Initiated login you have to send the Request to the Shibboleth selected by the SP or you don't even start to do any processing. Because SPs are almost always configured to talk only to PROD, that means the Shibboleth has to be installed with you send the Request to has to think it has the production Shibboleth EntityID (that is, it must have been installed with the "idp.entityID= https://auth.yale.edu/idp/shibboleth" property).
  • The Service Provider will tell the Browser to send the Request (either by POSTing a form or by Redirect on a GET) to the SSO URL that it the SP has in the Metadata we provided it to use for its copy of the Yale IdP Metadata. That will be a production URL, and it will begin with so the Browser will be instructed to send the Request to "https://auth.yale.edu/idp/..." This means Since that the Browser (or the test computer on which the Browser is running) has to trap requests sent to "auth.yale.edu" and send them somewhere else. How this is done will be described belowwill not be the actual URL of the test machine, you will have to use one of the tools described in the Client Configuration document to rewrite or reroute the request to the test VM.
  • Shibboleth will validate the digital signature on the Request, but that will always work because the SP is genuine.In addition to the EntityID,  the XML in the Request contains the SSO URL for Shibboleth configured to the SP in Yale's Metadata that will always work because the SP is genuine and is not itself being tested.

Now we get to the last test, and that is the hardest and requires a lot more description than the previous simple tests. The SAML protocol requires a Request to have internal XML fields containing both the EntityID of the IdP to which the message is sent and also the Single Sign On URL to which it is being sent. Assuming the SP uses our production Metadata, it will include in the XML one of two URLs, "https://auth.yale.edu/idp/profile/SAML2/POST/SSO" or "https://auth.yale.edu/idp/profile/SAML2/Redirect/SSO"

...

The hard part is the last test. , depending on which of two mechanisms it uses to send the Request.

Whatever mechanism you used in the Browser to rewrite or reroute this URL is not going to change the second copy of the same URL in the XML, and the problem is that SAML requires and Shibboleth implements a check that the URL in the XML is the same as the URL that Shibboleth believes was used to deliver the Request.

You can download the Shibboleth source and find the code that generates the URL and the other code that compares the two strings. Some of the data comes from the HTTP Host header sent to Shibboleth, and some comes from Tomcat. If the two strings don't match, Shibboleth writes an error message to the log containing the two strings it is comparing. You cannot change the string the SP sends in the XML, which will always be "https://auth.yale.edu/idp/profile/SAML2/POST/SSO", so you have to change the way the test service is configured and the network path between the browser and the test server to fix the problem if the non matching string in the log is:

...