Background
The SAML Single SignOn protocols exchange data between the Application and Shibboleth through the Browser. The Application does not talk to Shibboleth directly. In most cases the user can go directly to Shibboleth to get a SAML message that is sent on to the application (an "IdP Initiated Login"). When the user goes to the Application first, then there are two ways the Application transfers the Browser to the Shibboleth URL:
- (SAML Browser POST Protocol) writes a Form to the screen containing a pre-loaded text box containing the SAML Request, then uses JavaScript to Submit the data to the Shibboleth URL that was configured to the application
- (SAML Browser Redirect Procol) redirects the Brower to the configured Shibboleth URL appending "?SAMLRequest=" and the encoded Request content (so Shibboleth received a GET instead of a POST).
So the three options are:
- The user click on a link that points to the Shibboleth URL
- The Application sends a Redirect to the Shibboleth URL
- The Application Submits a form where the action on Submit is a Shibboleth URL.
Even though the Application may have a URL for Shibboleth, the actual navigation to that URL occurs in your Browser. Therefore, we can test Shibboleth by changing the Browser or your desktop computer to go to substitute a different network address or URL for the production Shibboleth URL ("https://auth.yale.edu/idp/...").
It is trivial to replace the link. Just create another link that goes to another address.
There are three ways to handle the case where the Application sends the URL.
- You can temporarily change the network address of "auth.yale.edu" by putting an entry in the "hosts" table on your computer. This will work, but you may also need to change the protocol from https to http, and you may need to change the port number, and the hosts file won't do either of these.
- You can install a Browser plugin (Redirector) that matches URL patterns and then rewrites them. It can match "https://auth.yale.edu/idp/*" (note the wildcard at the end) and replace it with "http://localhost:8080/idp/$1" (where $1 is a variable that inserts the rest of the original URL after the matched string.
- You can reroute the URL outside the Browser using a Proxy. Charles Web Debugging Proxy is easy to use and setup, while nginx is widely used and very powerful, but requires more reading because there is so much it can do.
Currently Redirector (Option 2) seems like the simplest workable approach, although this document will also describe the Proxy solution.
Preparation
Get Firefox. You can can certainly use other Browsers in production, but Firefox is needed to test.
Firefox Add-Ons
SAML messages are typically BIN64 encoded and appear to be the contents of a Text Box in a form or else part of the query string. You can line up a set of tools to trace, cut, paste, decode, and format the XML, or you can install the Firefox SAML Trace add on, which does all of this work for you.
A lot of debugging can be done by using Shibboleth on your desktop (Sandbox) and setting a URL starting with http://localhost:8080/idp/..." in the Browser address bar. However, final testing may require access to the actual application using a normal login sequence. This may require the use of the PREPROD Shibboleth VM, because it has credentials (the signing key) identical to production Shibboleth and can produce a Response that the application will accept. PREPROD may have a public URL address through the F5, or it may have an internal URL that can be accessed from a desktop, or it may require that you SSH login to the host and tunnel port 8080 to your desktop. However that works, you need the Redirector Firefox plugin which watches for a particular URL pattern "https://auth.yale.edu/idp/*" and then substitutes a replacement for the original URL "http://localhost:8080/idp/$1" where $1 is replaced by the rest of the original URL (after the matched prefix).
If you click the Firefox Menu icon (three horizontal lines in the upper right corner of the toolbar) then Add-On is an option (which looks like a puzzle piece). Click it.
Go to the Add-Ons, Search for a new Add-On with the word "SAML". Install the SAML Tracer. Look for "Redirector" and install the Redirector Add On (it has a logo of a capital R ending in an arrow).
Now SAML Trace appears in the same menu, and Redirector installs an icon on the toolbar itself.
SAML Tracer requires no configuration. When you turn it on it traces Web activity in a new Window and will highlight, decode, and display the XML in a SAML Request or Response on demand. You turn it off by closing the Trace Window.
Server Access
If you run Shibboleth under Tomcat on your Sandbox desktop, then it is http://localhost:8080/idp. No setup is required.
This is also the URL if you create an SSH tunnel from local port 8080 to vm-shibxxx-01.its.yale.internal port 8080, but to use it you need to use VPN and SSH.
The PREPROD VM is in a machine room behind a firewall that does allow direct access from desktop computers. In normal production, only HTTP traffic gets to VMs through the F5. If the F5 has a mapping to the PREPROD VM, then you can configure Redirector to just use that public URL and you are done. However, in the new DevOps world the common practice is to put new code on new VMs that may not yet have been configured to the F5. In this case, you need to access the machine room indirectly through a VPN.
First, you need to use the standard Cisco AnyConnect client to establish a VPN session, but instead of going to the public "access.yale.edu" target that normal Yale people use from off campus, you need to use one of the special VPN targets reserved for ITS staff on campus to access a part of the network from which they can connect to VMs in the machine room. Someone will tell you the VPN target name, explain how to download the AnyConnect profile for that VPN, and put you in the AD group that has access to that VPN target name.
After you have made the VPN connection, the next step is to use your preferred SSH Client to login to the VM. Operations must have created a login for your Netid on the VM and installed your SSH public key. In addition to the terminal session on the VM, the SSH client can be configured to "tunnel" one or more port number from your desktop computer to the VM. Since Shibboleth runs on Tomcat and the default Tomcat port number is 8080, you configure the SSH client to tunnel 8080 on your computer to 8080 on the VM. Now when you browse to "http://localhost:8080" the SSH Client forwards the traffic through the SSH session (and through the VPN) to Tomcat running on the VM in the machine room.
Only one program can use port 8080 on your computer at a time. When you test Shibboleth on your local Sandbox, it also uses 8080. Using the same local port number for both the Sandbox and the SSH tunnel will generate an error message if you accidentally run both at the same time. SSH will generate error messages that it cannot create the tunnel if you have forgotten to shut down the Sandbox Tomcat, and Tomcat will generate error messages that it cannot bind to the port if you forget forget to shut down SSH before starting the Sandbox. This is a feature, because you really don't want to spend hours trying to figure out what is wrong only to discover that you are debugging the wrong Shibboleth server.
Configure Redirector
Redirector requires you to create URL Remappings. What you are really doing is a Match and Replace, just like you would do in a Text editor. You can match text with either a Wildcard or Regular Expression. Wildcard is simpler and is perfectly adequate for Shibboleth testing.
The basic remapping is from "https://auth.yale.edu/idp/*" to "http://localhost:8080/idp/$1". Note that the Match string ends in the "*" wildcard character, so it matches all URLs that begin with the string. The Replace string ends in "$1" which is a variable that represents the data that matched the "*" wildcard. In English, this says, "Match all URLs that begin with https://auth.yale.edu/idp/ and replace those characters with http://localhost:8080/idp/, but leave the end of the URL alone."
You need a second remapping because of a problem with how the Unicon CAS-Shibboleth Integration works. When an application uses CAS, it creates a Service string that identifies the application to CAS and also provides a URL to which CAS returns the Service Ticket by Redirecting the Browser to the Service string.
CAS clients (including the Unicon CAS-Shib integration) are smart enough to realize that the host name of the computer on which Shibboleth is running and the port number the Tomcat service listens on may not be some internal values used only in the machine room. The Public URL for Shibboleth may be something the F5 knows, but Shibboleth cannot find this URL on its own, so you have to configure the Service URL as a property.
There is an another solution. The Browser sends an HTTP Host header containing the protocol, hostname, and port number from the point of view of the Browser before it goes through any network front end. That is exactly what we need for the Service string, although someone needs to determine the context path ("/idp") and add it on the end of the Host header information.
Unfortunately, the Unicon integration has a very strange algorithm that combines all three sources:
- It gets the protocol and hostname from a configured property. In the Yale Jenkins Install project, the property that sets this value is cas.target.url.
- It gets the port number from the HTTP Host header.
- It gets the context path from Tomcat.
Shibboleth PROD is configured with the correct official hostname:
cas.target.url=https://auth.yale.edu
If you are testing on your desktop Sandbox, nobody else cares how your private Shibboleth instance is configured, and you have to manually edit an install.properties file that overrides all other property configurations. So in this case you put in your install.properties file:
cas.target.url=http://localhost
Then if you enter "http://localhost:8080/idp" in your Browser, that also sets the Host header to "http://localhost:8080". Then the Unicon Integration takes the property value "http://localhost" appends the port number from the Host header ":8080" and the path from Tomcat "/idp" and sends the correct service=http://localhost:8080/idp to CAS.
But suppose you are going to use an SSH tunnel to a VM in the machine room. Now the cas.target.url property will be the official value for the dev/test/prod hostname through the F5. So you configure Redirector to change that URL to point to the Tunnel as described above.
The problem is that the Host HTTP header is generated by the Browser after Redirector rewrites the URL. If you convert "https://auth.yale.edu/idp/*" to "http://localhost:8080/idp/$1" then the Host header becomes "http://localhost:8080".
Now you run through the three step process that the Unicon code uses that were documented above:
- It gets the protocol and hostname from cas.target.url (and gets "https://auth.yale.edu" in PROD)
- It gets the port number from the HTTP Host header. (and gets ":8080" because that is how Redirector rewrote the URL).
- It gets the context path "/idp" from Tomcat.
So this produces service=https://auth.yale.edu:8080/idp.
Of course, the port number of 8080 is incompatible with the https protocol, but that is not really the problem. The problem is that with only one entry, Redirector is prepared to rewrite "https://auth.yale.edu/idp" but it is not prepared to rewrite "https://auth.yale.edu:8080/idp". So you need to create a second Redirector remapping entry to also map "https://auth.yale.edu:8080/idp/*" to "http://localhost:8080/idp/$1".
Now when CAS redirects the Browser back using the URL from the service= string, the Browser will also send the ticket string through the SSH port to the Shibboleth VM.
I could have just given you the two Redirector remap entries and told you to enter both of them without explaining why. Then at some point something will go wrong and you will be unable to return from CAS to Shib and there will be a strange URL in the address bar and you have no way to figure out what is wrong. This happens frequently enough that the explanation is helpful.
Also, if you try to login to another application and you cannot do it, and you end up with a "http://localhost:8080/idp" address in your Browser address bar and a "Page Not Found" error, that means that you forgot to disable the Redirector mappings in your Browser and there is no Sandbox Shibboleth and no SSH tunnel currently active.
A harder problem is when you leave the Redirector on and the tunnel active to DEV or TEST and then cannot logon to other applications like Service Now (because they do not accept DEV or TEST signatures). Best practice is to always turn Redirector off after testing.
Charles Web Debugging Proxy
Things get more complicated when you want to do final testing with real applications that do not support IdP Initiated logon. You have to go to the application first, and it is configured with the URL of the real Shibboleth production server (https://auth.yale.edu/idp). It will generate a Request and send it to that URL using your Browser to transport the message. So somehow you need to intercept the Request and send it "PREPROD", the test Shibboleth with the new configuration or code but also with the production Shibboleth credentials so it can generate a Response that the application will accept.
All communication between the application and Shibboleth go through the Browser on your computer. It might be possible to intercept the message inside the Browser, or to create a test program that does HTTP and replaces the Browser. Both would be a major coding project.
Instead, this problem can be solved with Charles Web Debugging Proxy, a light weight debugging tool written in Java that runs on Windows, Mac, and Linux. We may have a some licenses available, or an extra license is not expensive and can be provided by the department before your 30 day free trial expires. Go to charlesproxy.com, download and install it. You also want to install the Charles plug-in for Firefox in the same way you installed the SAML Tracer.
A Web Proxy is a program that sits between the Browser and the Web Server. In the old days with a slower Internet, proxies cached files to speed up browsing. Today the F5 acts as what is called a "reverse proxy", where it appears to the network to be the named Web Server ("auth.yale.edu" in this case) but then it forwards the request to other computers in the machine room that do the real work. For our purposes, Charles acts as a private F5 that we can configure to pretend to be the "auth.yale.edu" server (from the point of view of the Firefox browser on the desktop test machine). Like the F5 it forwards the request to a configured VM, but in this case instead of sending the message to the production VM it sends it to the PREPROD test machine.
Charles only intercepts data while you run it. Close Charles and everything behaves normally. Charles only intercepts the traffic you configure, and it runs locally on your desktop and only intercepts traffic from your Browser. Generally you only run it during testing, and when you are running it you only generate test related traffic.
For this test, you need to start specific services and make specific configuration entries. Each performs a specific function, and you have to do all of them or the test won't work. This is sufficiently complicated that I will explain what each step does, how to do it, and maybe what you will see if you do it wrong.
Start Charles. Start Firefox. When Charles is running, Firefox regards it as its Web Proxy and sends all Internet traffic to Charles.
Suppose you tell the Browser to go to production Shibboleth ("https://auth.yale.edu/idp/...").
Because the URL starts with "https", Firefox is going to establish an SSL/TLS session and encrypt all the data it sends. It expects to be talking to a host named "auth.yale.edu" and part of the SSL session setup requires that host to prove its identity with an X.509 Certificate that says it is "auth.yale.edu". If Charles has no configuration at all, it will allow the data to pass through it to the real host with that name, and although it will be in the middle of the data transfer, all the data will be encrypted and any data logged by Charles will be unreadable.
Charles needs to decrypt the data. The only way it can do that is (like the F5) it has to convince Firefox that it is the host named "auth.yale.edu", and to do that it needs to provide its own Certificate. It would be a violation of security practices to give Charles the real X.509 Certificate that the F5 has (because if that file is compromised then any bad guy can pretend to be Shibboleth). In this case, we do not need a Certificate that everyone trusts. We only need a Certificate that the Firefox Browser on our local desktop computer trusts as "auth.yale.edu". Since the Browser is under our control, that is easy to accomplish.
When Charles is installed, it generates a local self-signed Certificate for itself and uses it to create a mini Certificate Authority (CA). In the Charles menu, you select "Proxy" and then "SSL Proxying" from the pulldown list. Click the "Enable SSL Proxying" box and then add an SSL hostname of "auth.yale.edu" to the list box. Charles internally generates a Certificate for"auth.yale.edu" created by its internal Certificate Authority.
You have also just told Charles to intercept any HTTP traffic issued by your Firefox browser for a URL that begins with "https://auth.yale.edu/..." and to send back to Firefox the dummy Certificate issued by the internal Charles CA for hostname "auth.yale.edu". The Charles CA will not be in the list of real commercial Certificate Authorities that Firefox is distributed by Mozilla to automatically trust. So when it gets the dummy Certificate from Charles, Firefox displays a Warning page saying that the Web server certificate is not from a recognized Authority. You can click on the message page and tell Firefox to configure an Exception and trust this Certificate. It is convenient to tell Firefox to trust it from now on, and then you only get the Warning page the first time.
Now there is an SSL session inside your desktop computer between Firefox and Charles (acting as its Web proxy). Firefox encrypts data and Charles decrypts it. If this was all you configured, Charles would establish a second SSL connection between it and the real "auth.yale.edu" endpoint (the F5) and simply forward messages between Firefox and the F5, although now that it can decrypt the data it can log the information flowing in both directions. This is no big deal because SAML contains mostly biographical information, and you already know your own first name and last name, and really sensitive information like passwords do not go to Shibboleth.
However, we want to do something different. We want to take the data that was originally going to production Shibboleth and reroute it so it goes the the PREPROD VM with new code or new configuration. This is a second step where we now tell Charles to send the data for "auth.yale.edu" to a substitute URL address.
Before we configure the rerouting, we have to get access to the PREPROD VM. It is in a machine room behind a firewall that does allow direct access from desktop computers. In normal production, traffic gets to VMs through the F5. If the F5 has a mapping to the PREPROD VM, then you can configure Charles to just use that public URL and you are done. However, in the new DevOps world the common practice is to put new code on new VMs that may not yet have been configured to the F5. In this case, you need to access the machine room indirectly through a VPN.
First, you need to use the standard Cisco AnyConnect client to establish a VPN session, but instead of going to the public "access.yale.edu" target that normal Yale people use from off campus, you need to use one of the special VPN targets reserved for ITS staff on campus to access a part of the network from which they can connect to VMs in the machine room. Someone will tell you the VPN target name, explain how to download the AnyConnect profile for that VPN, and put you in the AD group that has access to that VPN target name.
After you have made the VPN connection, the next step is to use your preferred SSH Client to login to the VM. Operations must have created a login for your Netid on the VM and installed your SSH public key. In addition to the terminal session on the VM, the SSH client can be configured to "tunnel" one or more port number from your desktop computer to the VM. Since Shibboleth runs on Tomcat and the default Tomcat port number is 8080, you configure the SSH client to tunnel 8080 on your computer to 8080 on the VM. Now when you browse to "http://localhost:8080" the SSH Client forwards the traffic through the SSH session (and through the VPN) to Tomcat running on the VM in the machine room.
Only one program can use port 8080 on your computer at a time. When you test Shibboleth on your local Sandbox, it also uses 8080. Using the same local port number for both the Sandbox and the SSH tunnel will generate an error message if you accidentally run both at the same time. SSH will generate error messages that it cannot create the tunnel if you have forgotten to shut down the Sandbox Tomcat, and Tomcat will generate error messages that it cannot bind to the port if you forget forget to shut down SSH before starting the Sandbox. This is a feature, because you really don't want to spend hours trying to figure out what is wrong only to discover that you are debugging the wrong Shibboleth server.
Now we need to configure Charles to map the production Shibboleth URL (https://auth.yale.edu/idp) through that tunnel to the VM.
In the Charles menu, select "Tools", and then "Map Remote" from the pulldown list. Click the "Enable Map Remote" box and then Add a mapping.
The Map From part of the mapping provides data that must match a URL sent from your Firefox browser to Charles. In this case the Protocol is "https" and the Host is "auth.yale.edu". Generally you leave the other fields (port, path, query) blank and they default to matching anything.
The Map To part of the mapping specifies the changes you want to make to the incoming URL. In this case, you want to change the Protocol to "http" and the host to "localhost". You also want to change the Port to "8080". Leaving the other fields blank means that the path (/idp/...) will be copied from the incoming URL to the outgoing connection.
In certain unusual situations the "context" name of Shibboleth on the VM might not be "idp". Then you will want to configure the path field of the Map To to be the odd context name, but this is not the normal case.
If the F5 has a public URL for the VM you are trying to access, then you can forget the VPN and the SSH part. Just configure the Map To with the virtual host name on the F5 for the VM you want to access and Charles will send the data to the F5 instead of localhost:8080.
There is one last step. Click the "Preserve Host Header" box. When Firefox generated its request, it sent a Host header with the "auth.yale.edu" hostname and the original port. Then it went to Charles, who sent it through "localhost:8080" to a VM with some odd vm-shibxxx-01.yale.internal name. Unless the Host header is preserved, the VM has no way to know what the original host and port were. It turns out that this is important for the Unicon CAS Shibboleth integration, because it uses the Host header to get the port number to send in the service= string. The CAS Service has to be a URL that makes sense from the Browser's point of view, but when the CAS-Shibboleth Integration is building it, it is running on the VM in the machine room. The only way to know what URL the Browser used is to examine original Host header the Browser sent, which means that Charles has to send it on to the server unchanged. If you don't do this, then CAS gets a service string of the form "https://auth.yale.edu:8080/idp" which is not going to work.
Now I want to walk through the process end to end. For this example, suppose you start by pointing Firefox to the "yale.box.com" URL. Charles is still the proxy, but since it has no configuration for this URL, it sends the HTTP Request on to the real Box server on the internet. You press the "I am from Yale" button and Box sends back a SAML Request and tells Firefox to forward the request to production Shibboleth, which Box knows to be "https://auth.yale.edu/idp".
The Firefox Browser sends this URL and data to Charles (its Web proxy). Because Charles has "auth.yale.edu" in its list of SSL Proxy hosts, Charles acts as the server end sending its dummy Certificate and decrypts the data. Because "https://auth.yale.edu" is also in its Map Remote list, Charles substitutes "http" for "https" and "localhost:8080" for "auth.yale.edu" and sends the data to local port 8080 on your machine. Because that is the local end of an SSH tunnel, SSH now sends the data to the VM in the machine room where it is sent to that port 8080, which is the Tomcat running PREPROD Shibboleth. The Shibboleth response goes back through the same path in the opposite direction and comes back to Firefox.
If you have to CAS login, then the Unicon CAS Shibboleth integration running on the VM in the machine room gets the "https://auth.yale.edu" hostname part from its properties file, and it gets the port (which is empty/default) from the HTTP Host header, and it Redirects Firefox to "https://secure.its.yale.edu/cas/login?service=https://auth.yale.edu/idp..." CAS returns the ticket to the URL in the service string, and since that URL is also https://auth.yale.edu Charles does the whole SSL Proxy + Map To thing all over again and sends the ticket back to the PREPROD Shib VM, which logs you on to Shibboleth. Shibboleth builds a SAML Response and sends it back to Firefox along with code to forward the SAML to the login URL of box.com.
Because PREPROD Shibboleth has the same credentials/idp.key file as PROD Shibboleth (and the same configured entityID), the SAML message sent to box.com appears to be from production Shibboleth and the digital signature matches one from production Shibboleth. So box accepts the SAML and logs you on. Of course the actual attributes and Subject came from the PREPROD configuration, so if you screw up and don't generate the right data then box will not log you in and you fix the PREPROD configuration and rerun the test.
The techniques used by Charles are similar to exploits used by some malware. The difference is that Charles only functions when you explicitly run it and it only decodes traffic for hosts you configure it to proxy. If you accidentally leave it running and do some banking, then since bankofamerica.com is not in any of its configuration lists the SSL encrypted data remains secure and no sensitive information is exposed, even to other windows on your desktop. If you use it to debug CAS, then close it when you are done and don't save files that contain your Netid password.