Background
The SAML Single SignOn protocols exchange data between the Application and Shibboleth through the Browser. We do not use the protocols where the Application talks to Shibboleth directly.
The simplest Shibboleth SSO solution is what is called "IdP Initiated" (that is, Shibboleth sends an unsolicited login SAML message to the application). The user triggers this by sending clicking on a link or pasting a URL into the Browser of the form:
https://auth.yale.edu/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu
This URL points directly to a Shibboleth instance, which could be auth, auth-test, auth-dev, or localhost:8080 in the Sandbox. The string after the providerId= parameter is the entityId string for the application. Shibboleth searches the attribute-filter.xml for attributes released to that entityId, and it searches the Metadata for a definition of that entityId. It then generates a SAML login and sends it to the configured default AssertionConsumerService URL in the Metadata. The Metadata must exist and the hostname in the AssertionConsumerService URL must exist on the network in order for Shibboleth to generate and attempt to send the SAML.
The other approach is that the user can go to an application that has been configured with a URL of Yale Shibboleth. Typically the user selects from a pulldown list or presses a button, although some applications simply redirect back to Shibboleth immediately. The application sends a SAML Request, and there are two ways that Request can be transmitted. Both involve sending data to the Browser, and then having the Browser forward that data to the configured Shibboleth URL:
- (SAML Browser POST Protocol) writes a Form to the screen containing a pre-loaded text box containing the SAML Request, then uses JavaScript to Submit the data to https://auth.yale.edu/idp/profile/SAML2/POST/SSO
- (SAML Browser Redirect Procol) redirects the Brower to "https://auth.yale.edu/idp/profile/SAML2/Redirect/SSO?SAMLRequest=" followed by the character encoded SAML Request
So the three options are:
- The user click on a link sending the Browser to the Shibboleth URL
- The Application sends a Redirect to the Browser, and the Browser does a GET to the Shibboleth URL
- The Application writes a form and JavaScript Submits it to Shibboleth URL.
Shibboleth always uses the Browser POST Protocol to send the Response back to the Application (it writes a Form to the screen containing a pre-loaded text box containing the SAML Request, then uses JavaScript to Submit the data to the AssertionConsumerService URL configured for that EntityId in the Metadata).
So both Shibboleth and the Application talk only to the Browser and the Browser forwards data between them. Therefore, the network address of the Application and the Shibboleth server are always determined by the Browser and the environment of the desktop system on which the Browser is located. There are many mechanisms you can use privately on your computer (or on a VM running on your computer if you don't want to mess up your real OS):
- In 90% of the cases you can use an IdP Initiated Response, where all you need to do is to change the URL of the link (or bookmark in the Browser) to point to http://localhost:8080/idp or any other URL.
- You can temporarily change the network address of "auth.yale.edu" by putting an entry in the "hosts" table on your computer (or you could point to a dummy DNS server, but that is much harder). This captures the traffic, but it does not change the protocol or port number so generally you have to use a test Shibboleth that is running SSL on port 443.
- You can install a Browser plugin ("Redirector") that matches URL patterns and then rewrites them. It can match "https://auth.yale.edu/idp/*" (note the wildcard at the end) and replace it with "http://localhost:8080/idp/$1" (where $1 is a variable that inserts the rest of the original URL after the matched string.
- You can reroute the URL outside the Browser using a Proxy. Charles Web Debugging Proxy is easy to use and setup, while nginx is widely used and very powerful, but requires more reading because there is so much it can do.
The four options have been listed in increasing order of complexity, but they do not include the minute by minute difficulty during testing of turning the test environment on and off. Because changing the hosts table is usually inconvenient and provides limited function, we typically do not use that approach but include it here for completeness.
Generally you use the simplest option that can handle your test case, which means that most of the time you click on a link to a IdP Initiated operation, and you only use the other options when you have to go to the application first and get a real Request.
Preparation
Get Firefox. You can can certainly use other Browsers in production, but Firefox is needed to test.
Firefox Add-Ons
SAML messages are typically BIN64 encoded and appear to be the contents of a Text Box in a form or else part of the query string. You can line up a set of tools to trace, cut, paste, decode, and format the XML, or you can install the Firefox SAML Trace add on, which does all of this work for you.
A lot of debugging can be done by using Shibboleth on your desktop (Sandbox) and setting a URL starting with http://localhost:8080/idp/..." in the Browser address bar. However, final testing may require access to the actual application using a normal login sequence. This may require the use of the PREPROD Shibboleth VM, because it has credentials (the signing key) identical to production Shibboleth and can produce a Response that the application will accept. PREPROD may have a public URL address through the F5, or it may have an internal URL that can be accessed from a desktop, or it may require that you SSH login to the host and tunnel port 8080 to your desktop. However that works, you need the Redirector Firefox plugin which watches for a particular URL pattern "https://auth.yale.edu/idp/*" and then substitutes a replacement for the original URL "http://localhost:8080/idp/$1" where $1 is replaced by the rest of the original URL (after the matched prefix).
If you click the Firefox Menu icon (three horizontal lines in the upper right corner of the toolbar) then Add-On is an option (which looks like a puzzle piece). Click it.
Go to the Add-Ons, Search for a new Add-On with the word "SAML". Install the SAML Tracer. Look for "Redirector" and install the Redirector Add On (it has a logo of a capital R ending in an arrow).
Now SAML Trace appears in the same menu, and Redirector installs an icon on the toolbar itself.
SAML Tracer requires no configuration. When you turn it on it traces Web activity in a new Window and will highlight, decode, and display the XML in a SAML Request or Response on demand. You turn it off by closing the Trace Window.
Server Access
If you run Shibboleth under Tomcat on your Sandbox desktop, then it is http://localhost:8080/idp. No setup is required.
If you run Shibboleth on the DEV/TEST/PREPROD VMs in the machine room and they have a public URL through the F5, then you can use that URL. If they are new VMs not defined to the F5, then the firewall will not allow access from your desktop directly to the VM. You must do an SSH login and defined a "tunnel" that forwards traffic to a localhost port to a port in the VM. You can forward http and 8080 to port 8080 on the VM, or you can forward https and port 443 to 443 on the VM.
First, you need to use the standard Cisco AnyConnect client to establish a VPN session to an area of the network from which SSH traffic is permitted to the VMs. You need to be granted permission to use this type of VPN, you have to download the Profile files for these special VPN targets, and you have to MFA authenticate when setting up the VPN.
After you have made the VPN connection, the next step is to use your preferred SSH Client to login to the VM. Operations must have created a login for your Netid on the VM and installed your SSH public key. In addition to the terminal session on the VM, the SSH client can be configured to "tunnel" one or more port number from your desktop computer to the VM. In the general case, tunnel ports 8080, 8443, and 443 to the same port numbers on the VM.
Only one program can use port 8080 on your computer at a time. When you test Shibboleth on your local Sandbox, it also uses 8080. Using the same local port number for both the Sandbox and the SSH tunnel will generate an error message if you accidentally run both at the same time. SSH will generate error messages that it cannot create the tunnel if you have forgotten to shut down the Sandbox Tomcat, and Tomcat will generate error messages that it cannot bind to the port if you forget forget to shut down SSH before starting the Sandbox. This is a feature, because you really don't want to spend hours trying to figure out what is wrong only to discover that you are debugging the wrong Shibboleth server.
Browser Point of View
The Browser tries to follow a URL to "https://auth.yale.edu/idp". DNS resolves this address to a Virtual IP address on the F5 front-end. The F5 presents the SSL Certificate identifying itself as "auth.yale.edu", decrypts the data, and forwards it to the VM. Shibboleth is actually running on some host named "example.its.yale.internal" in a part of the network that only the F5 (and SSH through the VPN) can access.
Shibboleth has to Redirect the Browser. This may happen several times where Shibboleth Redirects the Browser to a different one of its own URLs, but it also happens when Shibboleth redirects the Browser to CAS and supplies a URL used by CAS to return the Browser to Shibboleth. In all these cases, Shibboleth has to somehow generate the original "https://auth.yale.edu/idp" URL because the browser cannot talk to the VM directly. The problem is that the F5 acts as auth.yale.edu and Shibboleth is running on a machine with a different name. There are two ways a Shibboleth component can know about the "auth.yale.edu" name. It can be configured with that name in a property file used during the Jenkins Install of Shibboleth onto the VM. Alternately, it can received that name in the HTTP "Host: https://auth.yale.edu" header generated originally by the Browser and then optionally modified by the F5.
During testing you may do everything on your desktop, or you may bypass the F5 and go directly to the VM over the SSH tunnel. Either way you have to be aware of the hostname configured as a property to the Shibboleth to which you are talking, and you have to consider what Host header the Browser is going to generate and whether it will be modified before it gets to Shibboleth. Otherwise, you get halfway through the test and suddenly the Browser gets redirected to an unexpected network address that is not your Shibboleth test machine.
Configure Redirector
Redirector runs inside your Browser. Every time the Browser is about to go to a network URL, Redirector inspects it. If the destination URL matches a pattern, Redirector replaces the string with a different string. This is essentially the same as the Find and Replace function of every Text editor. You can match text with either a Wildcard or Regular Expression. Wildcard is simpler and is perfectly adequate for Shibboleth testing.
The basic remapping is from "https://auth.yale.edu/idp/*" to "http://localhost:8080/idp/$1". Note that the Match string ends in the "*" wildcard character, so it matches all URLs that begin with the string. The Replace string ends in "$1" which is a variable that represents the data that matched the "*" wildcard. In English, this says, "Match all URLs that begin with https://auth.yale.edu/idp/ and replace those characters with http://localhost:8080/idp/, but leave the end of the URL alone."
You need a second remapping because of a problem with how the Unicon CAS-Shibboleth Integration works. When an application uses CAS, it creates a Service string that identifies the application to CAS and also provides a URL to which CAS returns the Service Ticket by Redirecting the Browser to the Service string.
CAS clients (including the Unicon CAS-Shib integration) are smart enough to realize that the host name of the computer on which Shibboleth is running and the port number the Tomcat service listens on may not be some internal values used only in the machine room. The Public URL for Shibboleth may be something the F5 knows, but Shibboleth cannot find this URL on its own, so you have to configure the Service URL as a property.
There is an another solution. The Browser sends an HTTP Host header containing the protocol, hostname, and port number from the point of view of the Browser before it goes through any network front end. That is exactly what we need for the Service string, although someone needs to determine the context path ("/idp") and add it on the end of the Host header information.
Unfortunately, the Unicon integration has a very strange algorithm that combines all three sources:
- It gets the protocol and hostname from a configured property. In the Yale Jenkins Install project, the property that sets this value is cas.target.url.
- It gets the port number from the HTTP Host header.
- It gets the context path from Tomcat.
Shibboleth PROD is configured with the correct official hostname:
cas.target.url=https://auth.yale.edu
If you are testing on your desktop Sandbox, nobody else cares how your private Shibboleth instance is configured, and you have to manually edit an install.properties file that overrides all other property configurations. So in this case you put in your install.properties file:
cas.target.url=http://localhost
Then if you enter "http://localhost:8080/idp" in your Browser, that also sets the Host header to "http://localhost:8080". Then the Unicon Integration takes the property value "http://localhost" appends the port number from the Host header ":8080" and the path from Tomcat "/idp" and sends the correct service=http://localhost:8080/idp to CAS.
But suppose you are going to use an SSH tunnel to a VM in the machine room. Now the cas.target.url property will be the official value for the dev/test/prod hostname through the F5. So you configure Redirector to change that URL to point to the Tunnel as described above.
The problem is that the Host HTTP header is generated by the Browser after Redirector rewrites the URL. If you convert "https://auth.yale.edu/idp/*" to "http://localhost:8080/idp/$1" then the Host header becomes "http://localhost:8080".
Now you run through the three step process that the Unicon code uses that were documented above:
- It gets the protocol and hostname from cas.target.url (and gets "https://auth.yale.edu" in PROD)
- It gets the port number from the HTTP Host header. (and gets ":8080" because that is how Redirector rewrote the URL).
- It gets the context path "/idp" from Tomcat.
So this produces service=https://auth.yale.edu:8080/idp.
Of course, the port number of 8080 is incompatible with the https protocol, but that is not really the problem. The problem is that with only one entry, Redirector is prepared to rewrite "https://auth.yale.edu/idp" but it is not prepared to rewrite "https://auth.yale.edu:8080/idp". So you need to create a second Redirector remapping entry to also map "https://auth.yale.edu:8080/idp/*" to "http://localhost:8080/idp/$1".
Now when CAS redirects the Browser back using the URL from the service= string, the Browser will also send the ticket string through the SSH port to the Shibboleth VM.
I could have just given you the two Redirector remap entries and told you to enter both of them without explaining why. Then at some point something will go wrong and you will be unable to return from CAS to Shib and there will be a strange URL in the address bar and you have no way to figure out what is wrong. This happens frequently enough that the explanation is helpful.
Also, if you try to login to another application and you cannot do it, and you end up with a "http://localhost:8080/idp" address in your Browser address bar and a "Page Not Found" error, that means that you forgot to disable the Redirector mappings in your Browser and there is no Sandbox Shibboleth and no SSH tunnel currently active.
A harder problem is when you leave the Redirector on and the tunnel active to DEV or TEST and then cannot logon to other applications like Service Now (because they do not accept DEV or TEST signatures). Best practice is to always turn Redirector off after testing.
Charles Web Debugging Proxy
A Web Proxy is a program that sits between the Browser and the Web Server. In the old days with a slower Internet, the Yale Proxy server cached frequently used Web pages from other locations to speed up browsing for Yale users. Today the F5 acts as what is called a "reverse proxy", where it appears to the network to be all the important Yale Web servers (including "auth.yale.edu") and then it forwards the request to other computers or VMs in the machine room that do the real work.
You can configure the Apache Web Server to be a proxy, and there is a very useful tool called nginx that specializes in acting as a proxy. However, these are larger solutions used by system administrators in production, and you could have to read a book to learn how to use them. A simpler solution is the Charles Web Debugging Proxy that can run on your desktop and modify the URL of Browser requests that pass through it.
The primary function of Charles is to intercept Browser traffic and display a log of data passing between the browser and the servers. This would be extremely valuable if we did not already have the SAML Tracer built into the Browser providing a more easily read summary of the important (SAML) data.
Without tracing, the Charles is simply an external alternative to the URL rewriting function of Redirector. There are some advantages to an external solution. Without Redirector, the Browser generates exactly the same data and headers that it would use to talk to the real Shibboleth. Because Redirector rewrites the URLs before they are logged and before the Host header is generated, you have to take its actions into consideration when you are debugging.
Charles only intercepts data while you run it. Close Charles and everything behaves normally. Charles only intercepts the traffic you configure, and it runs locally on your desktop and only intercepts traffic from your Browser. Generally you only run it during testing, and when you are running it you only generate test related traffic.
Charles has a lot more functions than we will use, but unlike Apache or nginx, it has a simple to use GUI configuration that is easy to learn. There are a sequence of steps you need to perform, and since it is easy to forget something, I will explain each step and what it does.
Download and Install Charles. Install the Charles Plug-In for Firefox.
Start Charles. Start Firefox. When Charles is running, the Charles Plug-in for Firefox configures Charles as the Firefox Web proxy. All traffic from Firefox goes to Charles, and Charles forwards it to the network. Stop Charles and the plug-in removes the proxy configuration and now Firefox runs normally.
Now suppose Charles is running and you tell the Browser to go to production Shibboleth ("https://auth.yale.edu/idp/...").
Because the URL starts with "https", Firefox is going to establish an SSL/TLS session and encrypt all the data it sends. It expects to be talking to a host named "auth.yale.edu" and part of the SSL session setup requires that host to prove its identity with an X.509 Certificate that says it is "auth.yale.edu". If Charles has no specific configuration, it will pass data between Firefox and the real auth.yale.edu server. The data will be encrypted and Charles will be unable to read it, but Firefox will run normally as if Charles was not present.
In order to decrypt the data, Charles has to provide a certificate and claim to be the "auth.yale.edu" server. This is exactly what the F5 does in the machine room, but in this case it will be private between your Firefox browser and your Charles Proxy, both running on your desktop. For security, they will use a private Certificate that you create and only your Firefox trusts.
When Charles is installed, it generates a local self-signed Certificate for itself and uses it to create a mini Certificate Authority (CA). In the Charles menu, you select "Proxy" and then "SSL Proxying" from the pulldown list. Click the "Enable SSL Proxying" box and then add an SSL hostname of "auth.yale.edu" to the list box. Charles internally generates a Certificate for"auth.yale.edu" created by its internal Certificate Authority.
In addition to creating the certificate, the SSL Proxy configuration just told Charles to intercept any HTTP traffic issued by your Firefox browser for a URL that begins with "https://auth.yale.edu/..." and to send back to Firefox the dummy Certificate issued by the internal Charles CA for hostname "auth.yale.edu". The Charles CA will not be in the list of real commercial Certificate Authorities that Firefox is distributed by Mozilla to automatically trust. So when Firefox gets the dummy Certificate from Charles, it displays a Warning page saying that the Web server certificate is not from a recognized Authority. You can click on the message page and tell Firefox to configure an Exception and trust this Certificate. It is convenient to tell Firefox to trust it from now on, and then you only get the Warning page the first time.
Now there is an SSL session inside your desktop computer between Firefox and Charles (acting as its Web proxy). Firefox encrypts data and Charles decrypts it. If this was all you configured, Charles would establish a second SSL connection between it and the real "auth.yale.edu" endpoint (the F5) and simply forward messages between Firefox and the F5, although now that it can decrypt the data it can log readable information flowing in both directions. The SAML generated by Shibboleth contains no sensitive information and can flow over an http unencrypted session, so this part is nothing special.
However, we want to do something different. We want to take the data that was originally going to production Shibboleth and reroute it so it goes the the PREPROD VM with new code or new configuration. This is a second step where we now tell Charles to send the data for "auth.yale.edu" to a substitute URL address.
As with Redirector, if the VM in the machine room has a public URL provided through the F5 then you can simply use that address. If not, then establish an SSH tunnel and use "localhost:8080".
The Charles version of the Redirector function is configured by selecting Tools from the Charles menu, then "Map Remote" from the pulldown list. Click the "Enable Map Remote" box and then Add a mapping.
The Map From part of the mapping provides data that must match a URL sent from your Firefox browser to Charles. In this case the Protocol is "https" and the Host is "auth.yale.edu". Generally you leave the other fields (port, path, query) blank and they default to matching anything.
The Map To part of the mapping specifies the changes you want to make to the incoming URL. In this case, you want to change the Protocol to "http", the host to "localhost", and the port to "8080". Leaving the other fields blank means that the path (/idp/...) will be copied from the incoming URL to the outgoing connection. It is assumed that you can figure out what target URL you want for the Map To address in other situations. If the F5 has a public URL for the VM you are trying to access, then just configure the Map To with the virtual host name on the F5 for the VM you want to access and Charles will send the data to the F5 instead of localhost:8080.
There is one last step. Click the "Preserve Host Header" box. When Firefox generated its request, it sent a Host header with the "https://auth.yale.edu" value. Without Redirector, Firefox does not know about the URL mapping so the Host header is the same as it would send to real production Shibboleth. This turns out to be exactly what we want to get the Unicon CAS-Shibboleth integration to generate the correct Service string without any fudging.
Charles is a larger tool and it has a license fee. Redirector is simpler and is free. Because Redirector operates inside the Browser there are changes in the URL and the Host header that are visible to the Browser, to SAML Tracer, and to the Shibboleth server (at least the Unicon CAS-Shib integration). Because Charles operates outside of the Browser and performs the same function that in production is performed by the F5, when we use Charles then everything is exactly the same as it will be in production. However, the differences created by Redirector are generally not important and do not interfere with any normal Shib testing.
The techniques used by Charles are similar to exploits used by some malware. The difference is that Charles only functions when you explicitly run it and it only decodes traffic for hosts you configure it to proxy. If you accidentally leave it running and do some banking, then since bankofamerica.com is not in any of its configuration lists the SSL encrypted data remains secure and no sensitive information is exposed, even to other windows on your desktop. If you use it to debug CAS, then close it when you are done and don't save files that contain your Netid password.