Versions Compared

Version Old Version 1 New Version Current
Changes made by

Howard Gilbert

Howard Gilbert

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Background

Every Yale partner uses the Browser based SAML Protocols. The application does not talk to Shibboleth directly, but instead uses your Browser as the transport mechanism. When the application (Archer, Google, Box, ...) wants you to login, it either

  • (SAML Browser POST Protocol) writes a Form to the screen containing a pre-loaded text box containing the SAML Request, then uses JavaScript to Submit the data to the Shibboleth URL that was configured to the application
  • (SAML Browser Redirect Procol) redirects the Brower to the configured Shibboleth URL appending "?SAMLRequest=" and the encoded Request content (so Shibboleth received a GET instead of a POST).

Either way, Shibboleth gets an HTTP request (GET or POST) containing a Request XML message identifying the application and providing more specific parameters that may override some of the Metadata information. Shibboleth saves the Request info, then Redirects the Browser to the CAS login function. CAS Redirects the Browser back to Shibboleth with a ticket= string. Shibboleth validates the ticket and obtains the Netid. It uses the Netid in various database and LDAP queries to get data, and prepares a Response object containing a Subject and various Attribute fields. It digitally signs the Response with its private key, so the application can use the distributed Yale Shibboleth Public Key to validate the signature and therefore the data in the Response.The Response is sent back to the application using the SAML Browser POST Protocol (it is put in a text box in a form and POST-ed to the application by JavaScript).

SAML has other protocols, and Shibboleth supports them, but they are not currently used at Yale and therefore do not need to be tested.

As this summary points out, the Application and Shibboleth never talk to each other directly. In all cases, they put data in the Browser (in a Form or in a QueryString at the end of the URL) and using POST or Redirect they get the Browser to send the data from the Application to Shibboleth or from Shibboleth to the Application. Since the Browser mediates between them, the Browser is the place where you can configure all your testing.

There are three plausible techniques.

  1. If you have a testing tool, you could use the tool to instrument HTTP requests and responses to Shibboleth and to the Application. That is a future objective, but developers do not currently have a reasonable tool for doing this type of test automation.
  2. The simplest option is to install add-ons to the Browser that allow you to trace the SAML traffic going back and forth, and to replace production Shibboleth in any URL ("https://auth.yale.edu/idp/...") with some new version of Shibboleth that has new code or configuration ("http://localhost:8080/idp...").
  3. A slightly more powerful option, but a bit more complex, is to insert the Charles Web Debugging Proxy between the Browser and the various network addresses to which it communicates (CAS, the test Shibboleth, and the Application).

Although in the long run the first option would be best, here we will describe how to install and configure options 2 and 3.

Preparation

Get Firefox. You can can certainly use other Browsers in production, but Firefox is needed to test.

Firefox Add-Ons

SAML messages are typically BIN64 encoded and appear to be the contents of a Text Box in a form or else part of the query string. You can line up a set of tools to trace, cut, paste, decode, and format the XML, or you can install the Firefox SAML Trace add on, which does all of this work for you.

A lot of debugging can be done by using Shibboleth on your desktop (Sandbox) and setting a URL starting with http://localhost:8080/idp/..." in the Browser address bar. However, final testing may require access to the actual application using a normal login sequence. This may require the use of the PREPROD Shibboleth VM, because it has credentials (the signing key) identical to production Shibboleth and can produce a Response that the application will accept. PREPROD may have a public URL address through the F5, or it may have an internal URL that can be accessed from a desktop, or it may require that you SSH login to the host and tunnel port 8080 to your desktop. However that works, you need the Redirector Firefox plugin which watches for a particular URL pattern "https://auth.yale.edu/idp/*" and then substitutes a replacement for the original URL "http://localhost:8080/idp/$1" where $1 is replaced by the rest of the original URL (after the matched prefix).

If you click the Firefox Menu icon (three horizontal lines in the upper right corner of the toolbar) then Add-On is an option (which looks like a puzzle piece). Click it.

Go to the Add-Ons, Search for a new Add-On with the word "SAML". Install the SAML Tracer. Look for "Redirector" and install the Redirector Add On (it has a logo of a capital R ending in an arrow).

Now SAML Trace appears in the same menu, and Redirector installs an icon on the toolbar itself.

SAML Tracer requires no configuration. When you turn it on it traces Web activity in a new Window and will highlight, decode, and display the XML in a SAML Request or Response on demand. You turn it off by closing the Trace Window.

Redirector requires you to create URL Remappings. What you are really doing is a Match and Replace, just like you would do in a Text editor. You can match text with either a Wildcard or Regular Expression. Wildcard is simpler and is perfectly adequate.

The first remapping is from "https://auth.yale.edu/idp/*" to "http://localhost:8080/idp/$1". Note that the Match string ends in the "*" wildcard character, so it matches all URLs that begin with the string. The Replace string ends in "$1" which is a variable that represents the data that matched the "*" wildcard. In English, this says, "Match all URLs that begin with https://auth.yale.edu/idp/ and replace those characters with http://localhost:8080/idp/, but leave the end of the URL alone.

You need a second remapping because of a problem with how the Unicon CAS-Shibboleth Integration works. It creates a service= string for CAS using (if it exists) the port number that appears in the Host header sent by the Browser. The original URL for "auth.yale.edu" had no port number, and if that URL was sent to production Shibboleth through the F5 then the Integration code sends CAS a service=https://auth.yale.edu/idp string. However, Redirector replaces that URL with a "localhost:8080" and sends a Host header to Shibboleth that contains the :8080 port. The Integration gets the hostname from a property and the port number from the Host Header. The Yale property is cas.target.url, and the PROD configuration for it is:

 cas.target.url=https://auth.yale.edu

So if you are testing in the Sandbox and change this property to say:

cas.target.url=http://localhost

then the CAS-Shib integration takes this hostname and adds the port number from the Host Header and gets the right thing "localhost:8080". However, if you are using an SSH tunnel to a VM in the machine room, and the VM has the production value for the property, now CAS-Shib takes the "https://auth.yale.edu" piece from the property and appends the port number from Host header and gets the strange value "https://auth.yale.edu:8080", which is neither the right URL for the VM nor the right URL for the desktop. Because the previous Redirector configuration only matched "auth.yale.edu/idp", once the :8080 got inserted it will not match and will not get rewitten. So you need a second URL Remapping to handle this case. This one maps "https://auth.yale.edu:8080/idp/*" to "http://localhost:8080/idp/$1"

Now when CAS redirects the Browser back using the URL from the service= string, the Browser will also send the ticket string to the test Shibboleth.

Server Access

If you run Shibboleth under Tomcat on your Sandbox desktop, then it is http://localhost:8080/idp. No setup is required.

This is also the URL if you create an SSH tunnel from local port 8080 to vm-shibxxx-01.its.yale.internal port 8080, but to use it you need to use VPN and SSH.

The PREPROD VM is in a machine room behind a firewall that does allow direct access from desktop computers. In normal production, only HTTP traffic gets to VMs through the F5. If the F5 has a mapping to the PREPROD VM, then you can configure Redirector to just use that public URL and you are done. However, in the new DevOps world the common practice is to put new code on new VMs that may not yet have been configured to the F5. In this case, you need to access the machine room indirectly through a VPN.

First, you need to use the standard Cisco AnyConnect client to establish a VPN session, but instead of going to the public "access.yale.edu" target that normal Yale people use from off campus, you need to use one of the special VPN targets reserved for ITS staff on campus to access a part of the network from which they can connect to VMs in the machine room. Someone will tell you the VPN target name, explain how to download the AnyConnect profile for that VPN, and put you in the AD group that has access to that VPN target name.

After you have made the VPN connection, the next step is to use your preferred SSH Client to login to the VM. Operations must have created a login for your Netid on the VM and installed your SSH public key. In addition to the terminal session on the VM, the SSH client can be configured to "tunnel" one or more port number from your desktop computer to the VM. Since Shibboleth runs on Tomcat and the default Tomcat port number is 8080, you configure the SSH client to tunnel 8080 on your computer to 8080 on the VM. Now when you browse to "http://localhost:8080" the SSH Client forwards the traffic through the SSH session (and through the VPN) to Tomcat running on the VM in the machine room.

Only one program can use port 8080 on your computer at a time. When you test Shibboleth on your local Sandbox, it also uses 8080. Using the same local port number for both the Sandbox and the SSH tunnel will generate an error message if you accidentally run both at the same time. SSH will generate error messages that it cannot create the tunnel if you have forgotten to shut down the Sandbox Tomcat, and Tomcat will generate error messages that it cannot bind to the port if you forget forget to shut down SSH before starting the Sandbox. This is a feature, because you really don't want to spend hours trying to figure out what is wrong only to discover that you are debugging the wrong Shibboleth server.

Charles Web Debugging Proxy

 

Things get more complicated when you want to do final testing with real applications that do not support IdP Initiated logon. You have to go to the application first, and it is configured with the URL of the real Shibboleth production server (https://auth.yale.edu/idp).  It will generate a Request and send it to that URL using your Browser to transport the message. So somehow you need to intercept the Request and send it "PREPROD", the test Shibboleth with the new configuration or code but also with the production Shibboleth credentials so it can generate a Response that the application will accept.

All communication between the application and Shibboleth go through the Browser on your computer. It might be possible to intercept the message inside the Browser, or to create a test program that does HTTP and replaces the Browser. Both would be a major coding project.

Instead, this problem can be solved with Charles Web Debugging Proxy, a light weight debugging tool written in Java that runs on Windows, Mac, and Linux. We may have a some licenses available, or an extra license is not expensive and can be provided by the department before your 30 day free trial expires. Go to charlesproxy.com, download and install it. You also want to install the Charles plug-in for Firefox in the same way you installed the SAML Tracer.

A Web Proxy is a program that sits between the Browser and the Web Server. In the old days with a slower Internet, proxies cached files to speed up browsing. Today the F5 acts as what is called a "reverse proxy", where it appears to the network to be the named Web Server ("auth.yale.edu" in this case) but then it forwards the request to other computers in the machine room that do the real work. For our purposes, Charles acts as a private F5 that we can configure to pretend to be the "auth.yale.edu" server (from the point of view of the Firefox browser on the desktop test machine). Like the F5 it forwards the request to a configured VM, but in this case instead of sending the message to the production VM it sends it to the PREPROD test machine.

Charles only intercepts data while you run it. Close Charles and everything behaves normally. Charles only intercepts the traffic you configure, and it runs locally on your desktop and only intercepts traffic from your Browser. Generally you only run it during testing, and when you are running it you only generate test related traffic.

For this test, you need to start specific services and make specific configuration entries. Each performs a specific function, and you have to do all of them or the test won't work. This is sufficiently complicated that I will explain what each step does, how to do it, and maybe what you will see if you do it wrong.

Start Charles. Start Firefox. When Charles is running, Firefox regards it as its Web Proxy and sends all Internet traffic to Charles.

Suppose you tell the Browser to go to production Shibboleth ("https://auth.yale.edu/idp/...").

Because the URL starts with "https", Firefox is going to establish an SSL/TLS session and encrypt all the data it sends. It expects to be talking to a host named "auth.yale.edu" and part of the SSL session setup requires that host to prove its identity with an X.509 Certificate that says it is "auth.yale.edu". If Charles has no configuration at all, it will allow the data to pass through it to the real host with that name, and although it will be in the middle of the data transfer, all the data will be encrypted and any data logged by Charles will be unreadable.

Charles needs to decrypt the data. The only way it can do that is (like the F5) it has to convince Firefox that it is the host named "auth.yale.edu", and to do that it needs to provide its own Certificate. It would be a violation of security practices to give Charles the real X.509 Certificate that the F5 has (because if that file is compromised then any bad guy can pretend to be Shibboleth). In this case, we do not need a Certificate that everyone trusts. We only need a Certificate that the Firefox Browser on our local desktop computer trusts as "auth.yale.edu". Since the Browser is under our control, that is easy to accomplish.

When Charles is installed, it generates a local self-signed Certificate for itself and uses it to create a mini Certificate Authority (CA). In the Charles menu, you select "Proxy" and then "SSL Proxying" from the pulldown list. Click the "Enable SSL Proxying" box and then add an SSL hostname of "auth.yale.edu" to the list box. Charles internally generates a Certificate for"auth.yale.edu" created by its internal Certificate Authority.

...

Audience

If you are an application owner testing a new version of Shibboleth before it goes into production, or if you are testing a new application that has not yet been configured to the production Shibboleth server, then you want to look at the instructions for configuring a desktop system to use Pre-Production Shibboleth. See Testing Your Application before a Shibboleth Change.

This document provides instructions for a Shibboleth developer who needs to test a Sandbox Shibboleth running on his desktop under Tomcat, or to test new VMs in the machine room for a future release of Shibboleth that has not yet even replaced the DEV or TEST environments.

Background

Generally speaking, you can test proposed changed to Shibboleth using https://auth-dev.yale.edu/idp and https://auth-dev.yale.edu/idp, then do final testing on the Pre-Production machine by changing your "hosts" file.

However, a Shibboleth developer does 90% of the testing using a Sandbox Shibboleth running on the local desktop computer.

When you are making a major Version upgrade, as we did moving from Shibboleth 2 to Shibboleth 3, then it is necessary to create an entirely new set of VMs in the machine room, with new Install jobs. A certain amount of testing has to be done with these VMs to make sure that the Installation process is working before you can even change the F5 front end to point to them in DEV or TEST. During this period, the only way to access the new VMs is to use an SSH tunnel that exposes the Tomcat running on the VM as if it were a local port on your desktop computer.

In both cases, you end up effectively going to "http://localhost:8080/idp", but even if that is the real address of Shibboleth, that doesn't mean that this is the exact URL you are always going to want to use. Sometimes you need to create an alias name for "localhost" (a.k.a. IP address 127.0.0.1). Sometimes you need to hide or translate the port number. There are several tools that can help.

Why isn't it good enough to simply get to Tomcat and the Shibboleth server application? Sometimes there is a problem created by the default HTTP Host header, the fact that the Sandbox uses http instead of https, and the default Tomcat configuration. Sometimes the problem comes from SAML.

SAML Checks

First, the usual SAML definitions of terms we have to use:

  • The applications you use Shibboleth to login to (Box, Eliapps mail, IPTV) are called Service Providers (SP) and are also called Relying Parties (RP). We will use the term SP here.
  • Shibboleth and each SP has a globally unique ID string called an EntityID. To make it globally unique, it is often a Web URL. The EntityID of production Shibboleth at Yale is "https://auth.yale.edu/idp/shibboleth", but the EntityID of Google Apps is just "google.com".
  • Shibboleth at Yale and each SP exchange a file called Metadata. A Metadata file contains the EntityID, an encoded Certificate that can be used to validate digital signatures, and a URL. The Shibboleth "SSO" URL is where the SP redirects the Browser to login to Shibboleth, and the SP "ACS" URL is the network address to which Shibboleth tells the Browser to send the SAML "Response" message so the user can login.
  • The SAML Response message from Shibboleth to the SP also contains Attributes of the user (like firstname, lastname, email address, etc.) that Shibboleth is configured to send to that particular SP.

Box and Google do not know or care that you are testing a new version of Shibboleth on your desktop. They are configured to talk to production (auth.yale.edu) and their Metadata, Certificates, login URL, and other parameters are not going to change. They will not accept a SAML Response generated by your Sandbox, so either you stop your test server from sending the Response or you accept that you will get an error message from the SP saying that the login failed to work. The trick is to configure your Browser to trace and capture the SAML Response your test Shibboleth generated, so even if the SP won't accept it,  you can go back and examine the XML yourself to verify that the Subject and Attributes are correct and should be accepted eventually when you run this configuration on a real Shibboleth server.

So if a problem occurs, it is triggered by an internal Shibboleth check that occurs at the very beginning of processing before Shibboleth even fetches the attributes.

You have to provide Shibboleth with the EntityID of a Service Provider. Then

  • The EntityID of the Service Provider has to be matched in one of the Metadata files configured to Shibboleth
  • The Metadata file has to provide a default AssertionConsumerService (ACS) URL to which the Response can be sent.
  • A lookup for the hostname in the ACS URL must return an address.

For example, suppose you supply the EntityID of "google.com". Shibboleth will search the Metadata files and finds in google-metadata.xml a line:

<EntityDescriptor entityID="google.com"

Having matched the EntityID, it now looks at the ACS statement in that Metadata file:

        <AssertionConsumerService index="1" isDefault="true"
            Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
            Location="https://www.google.com/a/yale.edu/acs"/>
        <AssertionConsumerService index="1" isDefault="false"
            Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
            Location="https://www.google.com/a/gdev.yale.edu/acs"/>
        <AssertionConsumerService index="1" isDefault="false"
            Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
            Location="https://www.google.com/a/gtst.yale.edu/acs"/>

There are three ACS locations that turn out to be for the production, dev, and test instances of Eliapps. The one with isDefault="true" is production, so that will be the URL to which the login message is sent. Shibboleth will verify that there is a machine on the network with name "www.google.com".

Of course, Shibboleth has to also look up the EntityID in the attribute-filter.xml file to know what attributes to send, and it has to find the attribute definitions. Those are all steps where your configuration can make mistakes, and errors will show up when the generated SAML Response message does not contain required data or contains the wrong value.

However, if the EntityID is not found in a Metadata file, or the Metadata file does not have a single or default ACS URL, then the request is incomplete and Shibboleth will not even start to process it.

If you actually want to login to the SP, then you need to be running the request on a Shibboleth that the SP is configured to respect. That will be Production or Pre-Production Shibboleth for most SPs, although during early testing with a new SP, it may be configured to use TEST Shibboleth. Now things are a bit more complicated because there are a long list of checks the SP will make on the SAML Response that you send it:

  • The EntityID of the IdP (auth-test or auth) that sent the Response has to match the EntityID configured in the SP.
  • The Digital Signature or the Response has to validate using the Public Certificate that the SP has configured for the Yale Shibboleth Identity Provider.
  • The SP EntityID and AssertionConsumerService URL in the Response XML have to match what the Service Provider believes to be its own EntityID and URL.
  • The first time the SP provides service to Yale users it is typically ready to accept any format of identifier. Therefore, we can initially provide either a simple Netid or a "scoped" value of netid@yale.edu. Once we start sending one format of Subject or Attribute, changing from one release of Shibboleth to another should not change the format of the attribute, or the SP will think that "joe@yale.edu" is a different person than "joe", forget any history and start with a new user blank slate.

Generally speaking, this means that final testing that goes all the way to a real login can only be done with the Pre-Production Shibboleth. However, any Shibboleth including the one on your desktop can generate a SAML Response with the XML names, values, and formats of Subject and Attributes, and if you compare the test and production XML you can see if everything is right.

IdP Initiated Logon

About 95% of the applications that use Shibboleth support "IdP Initiated" logon, where the Browser starts with a URL that points to Shibboleth and Shibboleth sends a SAML Response unsolicited to the Application. The starting URL is of the form:

https://auth.yale.edu/idp/profile/SAML2/Unsolicited/SSO?providerId=nobody.yale.edu

The last thing in the URL, the "providerid" parameter, is the EntityID of the SP. It has to match some configured EntityID in some Metadata file. Then Shibboleth logs you in, generates the SAML Response, and sends it to the configured AssertionConsumerServer (ACS) URL in the Metadata file.

You can use an IdP Initiated Login to generate a response for every SP, unless the SP Metadata demands signed requests:

<SPSSODescriptor AuthnRequestsSigned="true"

For testing purposes, you can set this parameter temporarily to "false", do your testing with the IdP Initiated login URL above, but then deploy the original Metadata to production.

If you do an IdP Initiated login to an SP that doesn't allow them, it will reject the SAML Response and you won't actually login. However, the digital signature was also probably invalid (not from production) and there may be a lot of other defects, and you probably were not planning to go all the way to a successful login at this stage anyway. You can still capture and check the Response for content.

It is not possible to test SP Initiated Login in this kind of preliminary testing configuration. Use IdP Initiated login to verify that the right attributes are being generated, and hold off SP Initiated testing until you get your code to Pre-Production.

Network Routing

When you are using IdP Initiated login, the only network routing problem is to make sure that the URL you use to send a request to the Shibboleth server matches the URL configured in the Shibboleth server for the CAS Service.

The CAS configuration is the "cas.target.url" parameter. When you run Shibboleth in the desktop Sandbox, it is convenient to set this property in the install.properties file you edit in your Install Project. However, if you are building new DEV and TEST server VMs in the machine room, this parameter will have been set by the install-DEV.properties or install-TEST.properties file.

So on the server you want to test, you might have:

Sandbox - cas.target.url=http://localhost:8080
TEST - cas.target.url=https://auth-test.yale.edu

What you need to do is to figure out how to run the test from your client machine using the same URL for the IdP Initiated login that was configured as the cas.target.url. In the Sandbox, all you have to do is use "http://localhost:8080" in both cases and you are done.

If you have a TEST configured VM that you are accessing through an SSH tunnel from your machine to the VM in the machine room, you have a problem. The tunnel makes the machine look like it is "localhost:8080" but the cas.target.url claims that it is https://auth-test.yale.edu.

So this document explains several tricks you can use to fix that.

Preparation

Get Firefox. You can can certainly use other Browsers in production, but Firefox is needed to test.

Firefox Add-Ons

SAML messages are typically BIN64 encoded and appear to be the contents of a Text Box in a form or else part of the query string. You can learn to manually locate and decode these strings, but the SAML Trace addon to Firefox does all the work for you.

If you click the Firefox Menu icon (three horizontal lines in the upper right corner of the toolbar) then Add-On is an option (which looks like a puzzle piece). Click it.

Go to the Add-Ons, Search for a new Add-On with the word "SAML". Install the SAML Tracer. Look for "Redirector" and install the Redirector Add On (it has a logo of a capital R ending in an arrow).

Now SAML Trace appears in the same menu, and Redirector installs an icon on the toolbar itself.

SAML Tracer requires no configuration. When you turn it on it traces Web activity in a new Window and will highlight, decode, and display the XML in a SAML Request or Response on demand. You turn it off by closing the Trace Window.

VM Access

If you run Shibboleth under Tomcat on your Sandbox desktop, then it is http://localhost:8080/idp. No setup is required.

If you run Shibboleth on the DEV/TEST/PREPROD VMs in the machine room and they have a public URL through the F5, then you can use that URL.

If they are new VMs not defined to the F5, then the firewall will not allow access from your desktop directly to the VM. First, you need to use the standard Cisco AnyConnect client to establish a VPN session to an area of the network from which SSH traffic is permitted to the VMs. You need to be put in an AD Group to use this type of VPN, and you have to download the Profile files for these special VPN targets. Get help from another developer if this is not set up on your machine or for your Netid. Access to the VPN requires MultiFactor Authentication even though your desktop is on campus.

When the VPN is enabled, you may have access to http port 80 and https port 443 using the native VM hostname (vm-something-01.its.yale.internal). That may be good enough for testing.

In other cases, you may need use an SSH tunnel for ports 8080 or 8443, but it is almost impossible to test or debug anything without SSH access to VMs and log files anyway. SSH requires that Operations create a login for your Netid on the VM and installed your SSH public key. In addition to the terminal session on the VM, the SSH client can be configured to "tunnel" one or more port number from your desktop computer to the VM. In the general case, tunnel ports 8080, 8443, and 443 to the same port numbers on the VM.

Only one program can use port 8080 on your computer at a time. When you test Shibboleth on your local Sandbox, it also uses 8080. Using the same local port number for both the Sandbox and the SSH tunnel will generate an error message if you accidentally run both at the same time. SSH will generate error messages that it cannot create the tunnel if you have forgotten to shut down the Sandbox Tomcat, and Tomcat will generate error messages that it cannot bind to the port if you forget forget to shut down SSH before starting the Sandbox. This is a feature, because you really don't want to spend hours trying to figure out what is wrong only to discover that you are debugging the wrong Shibboleth server.

Configure Redirector

Redirector runs inside your Browser. Every time the Browser is about to go to a network URL, Redirector inspects it. If the destination URL matches a pattern, Redirector replaces the string with a different string. This is essentially the same as the Find and Replace function of every Text editor. You can match text with either a Wildcard or Regular Expression. Wildcard is simpler and is perfectly adequate for Shibboleth testing.

The basic remapping is from "https://auth.yale.edu/idp/*" to "http://localhost:8080/idp/$1". Note that the Match string ends in the "*" wildcard character, so it matches all URLs that begin with the string. The Replace string ends in "$1" which is a variable that represents the data that matched the "*" wildcard. In English, this says, "Match all URLs that begin with https://auth.yale.edu/idp/ and replace those characters with http://localhost:8080/idp/, but leave the end of the URL alone."

There is a problem because Redirector replaces the hostname before the Host header is generated. Therefore, Shibboleth may have properties that tell it to generate a hostname of "https://auth.yale.edu", but it gets an HTTP header of "Host: http://localhost:8080".

Initially, this will cause trouble when the Unicon CAS-Shibboleth integration generates a Service string to send to CAS. After login, CAS redirects the Browser back to the URL in the Service string, which you will discover has been set to "service=https://auth.yale.edu:8080/idp". You can get around this in one of two ways.

If you are testing on your desktop Sandbox, nobody else cares how your private Shibboleth instance is configured, and you have to manually edit an install.properties file that overrides all other property configurations. So in this case you put in your install.properties file:

cas.target.url=http://localhost

Now the property value in the properties file and the rewritten Host header are the same, and the Service string becomes "service=http://localhost:8080/idp" and it works. However, this is generally not a viable choice for a VM in the machine room which you probably do not want to configure with a "localhost:8080" property.

So you need to create a second Redirector remapping entry to map "https://auth.yale.edu:8080/idp/*" to "http://localhost:8080/idp/$1".

This second remapping entry rewrites the URL that the Unicon integration generated and sent to CAS to the working network address of the test server you are using.

Unfortunately, this will not solve a mismatch problem between the "destination" string generated by an application when you go the the application first and it generates a Request and the URL Shibboleth generates to validate the destination. The application will generate the official production URL for production Shibboleth ("https://auth.yale.edu/..." and to send back to Firefox the dummy Certificate issued by the internal Charles CA for hostname "auth.yale.edu". The Charles CA will not be in the list of real commercial Certificate Authorities that Firefox is distributed by Mozilla to automatically trust. So when it gets the dummy Certificate from Charles, Firefox displays a Warning page saying that the Web server certificate is not from a recognized Authority. You can click on the message page and tell Firefox to configure an Exception and trust this Certificate. It is convenient to tell Firefox to trust it from now on, and then you only get the Warning page the first time.

Now there is an SSL session inside your desktop computer between Firefox and Charles (acting as its Web proxy). Firefox encrypts data and Charles decrypts it. If this was all you configured, Charles would establish a second SSL connection between it and the real "auth.yale.edu" endpoint (the F5) and simply forward messages between Firefox and the F5, although now that it can decrypt the data it can log the information flowing in both directions. This is no big deal because SAML contains mostly biographical information, and you already know your own first name and last name, and really sensitive information like passwords do not go to Shibboleth.

However, we want to do something different. We want to take the data that was originally going to production Shibboleth and reroute it so it goes the the PREPROD VM with new code or new configuration. This is a second step where we now tell Charles to send the data for "auth.yale.edu" to a substitute URL address.

Before we configure the rerouting, we have to get access to the PREPROD VM. It is in a machine room behind a firewall that does allow direct access from desktop computers. In normal production, traffic gets to VMs through the F5. If the F5 has a mapping to the PREPROD VM, then you can configure Charles to just use that public URL and you are done. However, in the new DevOps world the common practice is to put new code on new VMs that may not yet have been configured to the F5. In this case, you need to access the machine room indirectly through a VPN.

First, you need to use the standard Cisco AnyConnect client to establish a VPN session, but instead of going to the public "access.yale.edu" target that normal Yale people use from off campus, you need to use one of the special VPN targets reserved for ITS staff on campus to access a part of the network from which they can connect to VMs in the machine room. Someone will tell you the VPN target name, explain how to download the AnyConnect profile for that VPN, and put you in the AD group that has access to that VPN target name.

After you have made the VPN connection, the next step is to use your preferred SSH Client to login to the VM. Operations must have created a login for your Netid on the VM and installed your SSH public key. In addition to the terminal session on the VM, the SSH client can be configured to "tunnel" one or more port number from your desktop computer to the VM. Since Shibboleth runs on Tomcat and the default Tomcat port number is 8080, you configure the SSH client to tunnel 8080 on your computer to 8080 on the VM. Now when you browse to "http://localhost:8080" the SSH Client forwards the traffic through the SSH session (and through the VPN) to Tomcat running on the VM in the machine room.

Only one program can use port 8080 on your computer at a time. When you test Shibboleth on your local Sandbox, it also uses 8080. Using the same local port number for both the Sandbox and the SSH tunnel will generate an error message if you accidentally run both at the same time. SSH will generate error messages that it cannot create the tunnel if you have forgotten to shut down the Sandbox Tomcat, and Tomcat will generate error messages that it cannot bind to the port if you forget forget to shut down SSH before starting the Sandbox. This is a feature, because you really don't want to spend hours trying to figure out what is wrong only to discover that you are debugging the wrong Shibboleth server.

Now we need to configure Charles to map the production Shibboleth URL (https://auth.yale.edu/idp) through that tunnel to the VM.

In the Charles menu, select "Tools", and then "Map Remote" from the pulldown list. Click the "Enable Map Remote" box and then Add a mapping.

The Map From part of the mapping provides data that must match a URL sent from your Firefox browser to Charles. In this case the Protocol is "https" and the Host is "auth.yale.edu". Generally you leave the other fields (port, path, query) blank and they default to matching anything.

The Map To part of the mapping specifies the changes you want to make to the incoming URL. In this case, you want to change the Protocol to "http" and the host to "localhost". You also want to change the Port to "8080". Leaving the other fields blank means that the path (/idp/...) will be copied from the incoming URL to the outgoing connection.

In certain unusual situations the "context" name of Shibboleth on the VM might not be "idp". Then you will want to configure the path field of the Map To to be the odd context name, but this is not the normal case.

If the F5 has a public URL for the VM you are trying to access, then you can forget the VPN and the SSH part. Just configure the Map To with the virtual host name on the F5 for the VM you want to access and Charles will send the data to the F5 instead of localhost:8080.

There is one last step. Click the "Preserve Host Header" box. When Firefox generated its request, it sent a Host header with the "auth.yale.edu" hostname and the original port. Then it went to Charles, who sent it through "localhost:8080" to a VM with some odd vm-shibxxx-01.yale.internal name. Unless the Host header is preserved, the VM has no way to know what the original host and port were. It turns out that this is important for the Unicon CAS Shibboleth integration, because it uses the Host header to get the port number to send in the service= string. The CAS Service has to be a URL that makes sense from the Browser's point of view, but when the CAS-Shibboleth Integration is building it, it is running on the VM in the machine room. The only way to know what URL the Browser used is to examine original Host header the Browser sent, which means that Charles has to send it on to the server unchanged. If you don't do this, then CAS gets a service string of the form "https://auth.yale.edu:8080/idp" which is not going to work.

Now I want to walk through the process end to end. For this example, suppose you start by pointing Firefox to the "yale.box.com" URL. Charles is still the proxy, but since it has no configuration for this URL, it sends the HTTP Request on to the real Box server on the internet. You press the "I am from Yale" button and Box sends back a SAML Request and tells Firefox to forward the request to production Shibboleth, which Box knows to be "https://auth.yale.edu/idp".

The Firefox Browser sends this URL and data to Charles (its Web proxy). Because Charles has "auth.yale.edu" in its list of SSL Proxy hosts, Charles acts as the server end sending its dummy Certificate and decrypts the data. Because "https://auth.yale.edu" is also in its Map Remote list, Charles substitutes "http" for "https" and "localhost:8080" for "auth.yale.edu" and sends the data to local port 8080 on your machine. Because that is the local end of an SSH tunnel, SSH now sends the data to the VM in the machine room where it is sent to that port 8080, which is the Tomcat running PREPROD Shibboleth. The Shibboleth response goes back through the same path in the opposite direction and comes back to Firefox.

If you have to CAS login, then the Unicon CAS Shibboleth integration running on the VM in the machine room gets the "https://auth.yale.edu" hostname part from its properties file, and it gets the port (which is empty/default) from the HTTP Host header, and it Redirects Firefox to  "https://secure.its.yale.edu/cas/login?service=https://auth.yale.edu/idp..." CAS returns the ticket to the URL in the service string, and since that URL is also https://auth.yale.edu Charles does the whole SSL Proxy + Map To thing all over again and sends the ticket back to the PREPROD Shib VM, which logs you on to Shibboleth. Shibboleth builds a SAML Response and sends it back to Firefox along with code to forward the SAML to the login URL of box.com.

Because PREPROD Shibboleth has the same credentials/idp.key file as PROD Shibboleth (and the same configured entityID), the SAML message sent to box.com appears to be from production Shibboleth and the digital signature matches one from production Shibboleth. So box accepts the SAML and logs you on. Of course the actual attributes and Subject came from the PREPROD configuration, so if you screw up and don't generate the right data then box will not log you in and you fix the PREPROD configuration and rerun the testidp") but unfortunately the Spring Beans configured by default in the Shibboleth /system directory generate the protocol that they use to compare against this string from the actual protocol (http or https) through which the Request arrived from the Browser. No matter how many tricks you use to rewrite or modify stuff, if the Request arrives as http over 8080, then Shibboleth will generate "http://..." and that will fail to match the "https:" in the Request XML.

So if you are doing a final end-to-end login test of one of the applications that require you to go to the application first and get a Request object, then Shibboleth has to be running under a Tomcat configured to use SSL, and the Host header generated should have no port number.

Charles Web Debugging Proxy

A Web Proxy is a program that sits between the Browser and the Web Server. In the old days with a slower Internet, the Yale Proxy server cached frequently used Web pages from other locations to speed up browsing for Yale users. Today the F5 acts as what is called a "reverse proxy", where it appears to the network to be all the important Yale Web servers (including "auth.yale.edu") and then it forwards the request to other computers or VMs in the machine room that do the real work.

You can configure the Apache Web Server to be a proxy, and there is a very useful tool called nginx that specializes in acting as a proxy. However, these are larger solutions used by system administrators in production, and you could have to read a book to learn how to use them. A simpler solution is the Charles Web Debugging Proxy that can run on your desktop and modify the URL of Browser requests that pass through it.

The primary function of Charles is to intercept Browser traffic and display a log of data passing between the browser and the servers. This would be extremely valuable if we did not already have the SAML Tracer built into the Browser providing a more easily read summary of the important (SAML) data.

Without tracing, the Charles is simply an external alternative to the URL rewriting function of Redirector. There are some advantages to an external solution. Without Redirector, the Browser generates exactly the same data and headers that it would use to talk to the real Shibboleth. Because Redirector rewrites the URLs before they are logged and before the Host header is generated, you have to take its actions into consideration when you are debugging.

Charles only intercepts data while you run it. Close Charles and everything behaves normally. Charles only intercepts the traffic you configure, and it runs locally on your desktop and only intercepts traffic from your Browser. Generally you only run it during testing, and when you are running it you only generate test related traffic.

Charles has a lot more functions than we will use, but unlike Apache or nginx, it has a simple to use GUI configuration that is easy to learn. There are a sequence of steps you need to perform, and since it is easy to forget something, I will explain each step and what it does.

Download and Install Charles. Install the Charles Plug-In for Firefox.

Start Charles. Start Firefox. When Charles is running, the Charles Plug-in for Firefox configures Charles as the Firefox Web proxy. All traffic from Firefox goes to Charles, and Charles forwards it to the network. Stop Charles and the plug-in removes the proxy configuration and now Firefox runs normally.

Now suppose Charles is running and you tell the Browser to go to production Shibboleth ("https://auth.yale.edu/idp/...").

Because the URL starts with "https", Firefox is going to establish an SSL/TLS session and encrypt all the data it sends. It expects to be talking to a host named "auth.yale.edu" and part of the SSL session setup requires that host to prove its identity with an X.509 Certificate that says it is "auth.yale.edu". If Charles has no specific configuration, it will pass data between Firefox and the real auth.yale.edu server. The data will be encrypted and Charles will be unable to read it, but Firefox will run normally as if Charles was not present.

In order to decrypt the data, Charles has to provide a certificate and claim to be the "auth.yale.edu" server. This is exactly what the F5 does in the machine room, but in this case it will be private between your Firefox browser and your Charles Proxy, both running on your desktop. For security, they will use a private Certificate that you create and only your Firefox trusts.

When Charles is installed, it generates a local self-signed Certificate for itself and uses it to create a mini Certificate Authority (CA). In the Charles menu, you select "Proxy" and then "SSL Proxying" from the pulldown list. Click the "Enable SSL Proxying" box and then add an SSL hostname of "auth.yale.edu" to the list box. Charles internally generates a Certificate for"auth.yale.edu" created by its internal Certificate Authority.

In addition to creating the certificate, the SSL Proxy configuration just told Charles to intercept any HTTP traffic issued by your Firefox browser for a URL that begins with "https://auth.yale.edu/..." and to send back to Firefox the dummy Certificate issued by the internal Charles CA for hostname "auth.yale.edu". The Charles CA will not be in the list of real commercial Certificate Authorities that Firefox is distributed by Mozilla to automatically trust. So when Firefox gets the dummy Certificate from Charles, it displays a Warning page saying that the Web server certificate is not from a recognized Authority. You can click on the message page and tell Firefox to configure an Exception and trust this Certificate. It is convenient to tell Firefox to trust it from now on, and then you only get the Warning page the first time.

Now there is an SSL session inside your desktop computer between Firefox and Charles (acting as its Web proxy). Firefox encrypts data and Charles decrypts it. If this was all you configured, Charles would establish a second SSL connection between it and the real "auth.yale.edu" endpoint (the F5) and simply forward messages between Firefox and the F5, although now that it can decrypt the data it can log readable information flowing in both directions. The SAML generated by Shibboleth contains no sensitive information and can flow over an http unencrypted session, so this part is nothing special.

However, we want to do something different. We want to take the data that was originally going to production Shibboleth and reroute it so it goes the the PREPROD VM with new code or new configuration. This is a second step where we now tell Charles to send the data for "auth.yale.edu" to a substitute URL address.

As with Redirector, if the VM in the machine room has a public URL provided through the F5 then you can simply use that address. If not, then establish an SSH tunnel and use "localhost:8080".

The Charles version of the Redirector function is configured by selecting Tools from the Charles menu, then "Map Remote" from the pulldown list. Click the "Enable Map Remote" box and then Add a mapping.

The Map From part of the mapping provides data that must match a URL sent from your Firefox browser to Charles. In this case the Protocol is "https" and the Host is "auth.yale.edu". Generally you leave the other fields (port, path, query) blank and they default to matching anything.

The Map To part of the mapping specifies the changes you want to make to the incoming URL. In this case, you want to change the Protocol to "http", the host to "localhost", and the port to "8080". Leaving the other fields blank means that the path (/idp/...) will be copied from the incoming URL to the outgoing connection. It is assumed that you can figure out what target URL you want for the Map To address in other situations. If the F5 has a public URL for the VM you are trying to access, then just configure the Map To with the virtual host name on the F5 for the VM you want to access and Charles will send the data to the F5 instead of localhost:8080.

There is one last step. Click the "Preserve Host Header" box. When Firefox generated its request, it sent a Host header with the "https://auth.yale.edu" value. Without Redirector, Firefox does not know about the URL mapping so the Host header is the same as it would send to real production Shibboleth. This turns out to be exactly what we want to get the Unicon CAS-Shibboleth integration to generate the correct Service string without any fudging.

Charles is a larger tool and it has a license fee. Redirector is simpler and is free. Because Redirector operates inside the Browser there are changes in the URL and the Host header that are visible to the Browser, to SAML Tracer, and to the Shibboleth server (at least the Unicon CAS-Shib integration). Because Charles operates outside of the Browser and performs the same function that in production is performed by the F5, when we use Charles then everything is exactly the same as it will be in production. However, the differences created by Redirector are generally not important and do not interfere with any normal Shib testing.

The techniques used by Charles are similar to exploits used by some malware. The difference is that Charles only functions when you explicitly run it and it only decodes traffic for hosts you configure it to proxy. If you accidentally leave it running and do some banking, then since bankofamerica.com is not in any of its configuration lists the SSL encrypted data remains secure and no sensitive information is exposed, even to other windows on your desktop. If you use it to debug CAS, then close it when you are done and don't save files that contain your Netid password.