IIQ AD Updater Technical Design

The purpose of this project is to replace the old AD Daily Updater with newer better code residing in an IIQ instance. However, the future plan to support Exchange Online routing of Email (Mail Relay Replacement) adds and additional set of requirements.

Issues

This project replaces

1. The AD Daily Updater, a Java program driven by the Oracle ACS1 database, which populated a select set of "biographical" fields in the AD for non-private people. None of these fields were important from a systems point of view, except for UPN which is an identifier to login to services.
2. The Mail Relay configuration, which maps Email Aliases (ending in "@yale.edu") to the native email addresses of Yale systems. Originally this meant "Netid@connect.yale.edu" for Exchange accounts and "Netid@pantheon.yale.edu" for IMAP. Current Exchange has moved to O365 and IMAP has been replaced by Eliapps.
3. The Exchange mail routing fields in AD (MailNickName and ProxyAddresses). When mail was sent to Exchange, it delivered mail addressed to various Email Aliases using data stored in two fields of the AD. A separate Exchange field management function in the AD Daily Updater maintained these two Exchange Mail Routing fields.

This new project continues to support the biographical fields of AD, but now gets the data from IIQ. It updates the logic of function 3) to reflect differences between the original on premise Exchange and the O365 Azure AD requirements, and merges the function 2) code in by populating the MailNickname, ProxyAddresses, and now Target Address field of an AD object for Eliapps and legacy departmental Email Aliases.

We have to plan ahead for expected changes that do not currently have a schedule:

• The on premise Exchange system serves no useful purpose. It should be decommissioned. This would involve removing certain Exchange related fields from the on premise AD (fields that begin with the "msExch" prefix. The old AD Daily Updater code depends on these fields, but the new replacement code will be written to work if they are null. The one exception is that Private users are excluded from the Outlook directory with two of these fields and that function still needs to be supported.
• The Mail Relay machines will at some point be decommissioned and the Exchanged Online function in the Azure Cloud will take over routing all mail addressed to an "@yale.edu" Alias. Since Excahnge Online is configured by Azure AD in essentially the same way that on premise Exchange was configured by on premise AD, we have to add AD Email Aliases to the ProxyAddresses list and the native Email address (MAILBOX) of the aliases to the TargetAddress field for existing User objects or new Contact objects to be created for the purpose.

Currently mail addressed to an "ALIAS_NAME@yale.edu" Email address goes to the Mail Relay machines. They look up the ALIAS_NAME in configuration data and generate a native Email address that in practice sends the mail to O365 or Eliapps (although there are a few legacy entries that may go to deparmental standalone mail servers). If a Primary Email Alias points to a native Email Address ending in "@bulldogs.yale.edu" then the mail goes to Eliapps.

However, when the Mail Relays go away and are replaced by Exchange Online, then what was once a two tiered processing becomes a single step, and Exchange gets first dibs on the mail. Because Exchange Online puts its configuration data in the Azure/Local AD, some rules about the way AD has to work can conflict with specific settings in the Email Alias table. Fortunately, the current configurations that will become invalid are also in violation of Yale policy. This will not, however, avoid the requirement to communicate the change to affected users.

There are a set of Yale and Exchange rules:

• It is a Yale rule that the Primary Email Alias (first.last@yale.edu) also be the UserPrincipalName and its prefix is the MailNickName (first.last). We could have used Netid instead of Primary AliasName, but we didn't.
• It is an Exchange Online rule that the UPN has to be in the ProxyAddress list of the User object (that is, the UPN has to be a deliverable mail address to anyone with an O365 mail account).
• It is an Exchange Online rule that the same Email address cannot be in two ProxyAddress lists of two Azure AD objects.

When you combine the rules, no Netid can have a Primary Email Alias that points to Eliapps if they also have an O365 account. It has been Yale policy that ordinary users must have one or the other, O365 or Eliapps, but not both. This has been relaxed for special users, but there the policy remained that if a user had both types of mail, the O365 account should be Primary and the Eliapps account should be Secondary. It was previously possible to break the rule, because the Mail Relays allows the Primary Alias to point to either system. Exchange Online requires the Primary Alias to effectively point to O365 for any user who has an O365 mail account, and that will happen no matter what the Email Alias table says. Fortunately, only a small number of users are this far out of Yale policy compliance. Unfortunately, they generally had to be important people to get the exception in the first place.

It will not be possible to exactly match the accidental quirks of current two tier mail routing. However, a person can continue to own both an O365 and Eliapps mail account and can continue to route mail addressed to his Primary Email address to the Eliapps account. You can even retain your Email address in Yale directories. However, to do this you need to create a Dependent Netid to own your O365 mailbox, and then someone has to swap your current AD User object (that is connected to your mailbox) for the Dependent Netid User object renaming the CN and SAMAccountName that connect the User object to the Netids. We may do this for a handful of important people, but that is all.

Both the current AD Daily Updater and the current Mail Relay configuration program get the same mail routing data from the Email Alias system (the SMART schema in ACS1, and in particular the SMART.DIR_ONLINE_INFO table). AD Daily Updater only uses the Exchange subset of the data, but this will change in the new code where the non-Exchange data will also be added to AD. The TargetAddress field is not used for Exchange mail accounts, but will point to the native email account address (the MAILBOX field of the Email Alias table) for non-Exchange AD objects.

The Mail Relay configuration data was just a table that translated every Email Alias address ("@yale.edu") to a native address ("Netid@connect.yale.edu" for Exchange, "AccountName@bulldogs.yale.edu" for Eliapps). The AD Updater had to take the same data but assign Exchange users to an AD User object. You might think that this assignment would be done using the NETID column (the owner of the alias) but a few Netids directly own Aliases that point to Dependent Netids (including group accounts) where the mailbox is actually under a different AD User object. So the only reliable way of correlating Email Alias data to IIQ Identities (and AD User objects) is to group aliases that have the same MAILBOX value and look for the Netid as the value in front of the suffix "@connect.yale.edu". Mailboxes with other suffixes are Eliapps or legacy aliases, and for them the Netid that owns the Alias is information, but the correlation rule is more complicated and will be described later.

If an alias has a MAILBOX column that ends in "@connect.yale.edu" but where the part in front of the "@" does not match any Netid, then it is an Exchange resource of some sort. Exchange Online automatically knows about its own internally configured non-Identity resources (meeting rooms, distribution lists, etc.) and it will automatically route mail that has the address of a native resource without any additional configuration. There is some code to configure such resources in the Mail Relays, but when they are replaced by Exchange Online we can simply ignore them. IIQ does not need to create Identity objects for these Exchange resources (at least not at this time).

Every active Netid generates a Local AD User object. Every Local AD User object generates an Azure AD object within the next hour through the Azure AD Connector synchronization process. However, while the Netid value is important in the Local AD ( it is the first CN=Netid element in the Distinguished Name, and the legacy login ID (the SAMAccountName), and then if it got an Exchange mailbox it also added a "Netid@connect.yale.edu" entry to the ProxyAddresses list.

However, while the Local AD User object and Azure AD User object are connected by UUIDs, the Netid is not actually a direct field in Azure. There is no DN, no SAMAccountName, and not everyone has a mail account. For sanity and to ensure correct management of mail, there will be at most one "Netid@connect.yale.edu" entry in the ProxyAddresses list of an Azure AD User object, and if it exists it will have the Netid in the Local AD user object connected by UUID to that Azure AD object. For this to work, there can be only one "Netid@connect.yale.edu" entry in the ProxyAddresses list of the Local AD object, and it must be attached to the User object whose CN and SAMAccountName have the matching Netid.

Non O365 mail account routing information can be configured in either a User or Contact object. That is determined by Yale policy. The handling of objects with O365 mail accounts is determined by Microsoft system restrictions.

There are 1800 Email Aliases that do not resolve to O365 or Eliapps. They are spread across mailman, elilists, panlists, cs, som, aya, invest, med, physics, chem, geology, cmp, math, astro (all .yale.edu). Some of this is junk, but cleaning it up is a different project and some of it is not junk and has to be managed by Foreign Mail Aliases (described below).

There are existing Contacts created in AD to make specific Eliapps users visible to Outlook. These "non-routing" Contacts have a Mail attribute and not much else. When we create Routing Contacts (with a targetAddress, MailNickName, and proxyAddresses list) we may duplicate information. We need to decide if we are going to try to identify duplicates and delete old Contacts, and whether routing Contacts and Foreign Contacts we create are going to be put in different OUs and are these old Legacy Contacts going to be moved to yet another OU.

Assumptions

IIQ will be at 7.2p2 or later.

The DEV AD (yale.org) does not work and may be used for a different project (NGN). We will develop and unit test this code in a desktop VM Sandbox environment. However, we may use the yaleorg.onmicrosoft.com Azure AD.

Assignment of Functions to Instances

The update of biographical fields (firstname, lastname) will occur in the Identity Management instance of IIQ and will be driven by changes to Identity variables. The update of mail routing fields will be done in the Email Provisioning instance of IIQ.

The two instances have a common set of Identity Variable names. In this application a common unit of code may run in both applications and set the same Identity Variable name to the same value in both instance, but the variable will be used in one instance and ignored in the other.

The UPN of the Azure AD object should be updated if it gets set to the wrong value. Although this could be regarded as a biographical field (the LocalAD UPN is), AzureAD is currently not an Application in the Identity Management space and there is no reason to add it. So Azure UPN is regarded as a honorary Email Routing field and will be provisioned from the Email Provisioning instance.

Identity Management Instance

Additional Applications (data sources) required:

• LocalAD Users (copy existing application from Email Provisioning instance)

Email Provisioning Instance

Additional Applications (data sources) required:

• Dependent Netids (copy existing application from Identity Management instance)
• Full Email Alias view (there is an existing Application named Email but it only aggregates Primary Aliases)
• Local AD Contacts (provisioned, may become a source of identities)

Provisioning Strategy (Biographical data)

As is currently done in Email Provisioning, Identity Management will have a Role with an Entitlement to membership in the Provisioned Users group of LocalAD. Assigning an Identity to the Role will create a new AD User object for that Netid if one does not already exist. The User object is initially created with minimal information, and is then filled in with the full set of biographical data in the next cycle.

This application adds a set of Identity Variables named ADXXXX (example: ADgivenName) are filled in by a Rule run after an Identity Refresh task. Normally Identity Variables are filled in after new data is read in from a source system, but the ADXXXX variables are assigned values from other Identity Variables that were themselves populated from source systems. For example, your Job Title comes in from Workday Worker and is used to populate a previously configured Identity Variable which then used the raw title for various Yale systems. However, the ADtitle variable is constrained to be a maximum of 128 characters because that is the maximum field length in AD. So at Yale there are currently 51 people with longer Job Titles in Workday Worker and other Yale systems, but we truncate that to 128 when we create the ADtitle variable that provisions the Title property in AD.

There are also restrictions due to Privacy rules for certain people, and in a few cases there are multiple possible source fields for the same AD variable, and the RefreshRule will select the right source based on Affiliation and other attributes.

The Netid System is the source for Identities created for Dependent Netids. It does not have any biographical data, so the source fields are all null, and the ADXXXX variables are all null, and the AD gets no biographical information.

Biographical ADXXXX variables have a LocalAD Target field only in the Identity Management instance. These Identity Variables may be filled in by the RefreshRule running in the Email Provisioning instance of IIQ, but they will not provision AD User objects from that instance.

Provisioning Strategy (Email Routing fields)

The Email Routing ADXXXX variables are calculated from the Account generated by aggregating the Full Email Alias table, an application that is only defined in the Email Provisioning instance. They also only have Target definitions and only update the LocalAD in that instance. The same variables have neither a source nor a target in Identity Management.

Because ADProxyAddresses is a multivalued field, we want to avoid updating it if the only difference between the current value in AD and the calculated correct value from the Email Alias table is to list the same values in a different order.

Dependent Netids and Contacts can own or describe Email accounts, so they do have ADMailNickName, ADProxyAddresses, and ADTargetAddress fields that must be calculated and, if necessary, provisioned.

However, correlating the Email Alias data to the right Identity cannot be done on the fly in IIQ. We rely on a database view to present the data in the right order. So most of the work is done in the database, and a little work may be done in the correlation rule.

Unlike the previous AD Daily Updater Java program, IIQ will create AD User objects for Identities that need them.

Mail Relay Replacement

Currently all mail with an address ending in "@yale.edu" is processed by the Mail Relay machines at Yale. They are configured (once an hour) from the data in the Mail Alias system, where is unique MailAliasName+"@yale.edu" is resolved to a native Mail system address (the Mailbox). If the Mailbox is "netid@connect.yale.edu" then this is an O365 address to be processed by Exchange Online. If it is "@bulldogs.yale.edu" then it is an Eliapps account to be processed by Google. In all other cases, the Mailbox address substitutes for the original "@yale.edu" and is processed through SMTP.

The Mail Relay function is to be retired and all mail will now pass through Exchange online. Essentially, we don't have to do anything differently for O365 accounts because that mail already passes through Exchange and is delivered correctly. So the change is for Eliapps and other addresses.

Exchange Online is configured by data in the Azure AD. So the problem is to use the current contents of the Email Alias table (DIR_ONLINE_INFO) to configure (Azure) AD objects so the mail will be delivered the correct way. Exchange has a concept called MailNickName which is not standard and can be ignored at this point. Otherwise, Exchange Online looks at any "xxxx@yale.edu" address and tries to match it to exactly one "smtp:xxxx@yale.edu" entry in the ProxyAddresses list of an AD User or Contact object. If it matches and this is not an O365 account, then it delivers mail based on the TargetAddress field of the matched object.

So the requirement is simple:

Every AliasName in the Alias table (prefixed with "smtp:" and suffixed by "@yale.edu") must be in exactly one AD User of Contact ProxyAddress list and

The TargetAddress of the matched User or Contact must be set to the Mailbox value of that entry in the Alias table.

Today we populate the ProxyAddress list of the User object of everyone who has an O365 account with an entry for every Alias Table row that has a Mailbox value of "netid@yale.edu".

So the new step is to do essentially the same thing for Eliapps mail users. If they do not have an O365 account, then we can set the TargetAddress of their AD User object to "EliappsAccount@bulldogs.yale.edu" and then populate their ProxyAddresses with every Email Alias with a Mailbox that points to that account (exactly what we do now for O365 users). If someone has both an O365 and an Eliapps account, then the O365 account has to go in the User object, so we have to create a second Contact object for the Eliapps ProxyAddresses list and TargetAddress.

Then for any other Alias (not O365 or Eliapps) the one Contact object per row in the Alias table works fine.

We accomplish this by aggregating Alias entries by Netid, and then for each Netid

• If there is any Alias with an O365 MAILBOX, then all the Alias names with that MAILBOX value generate an smtp:AliasName@yale.edu entry in the User object (the PrimaryAlias is capitalized SMTP:), the TargetAddress of the User object is set to some O365 value, and now the User object is full and any other aliases have to go into Contacts.
• Starting with the Primary, if there is any Alias with a MAILBOX ending in "@bulldogs.yale.edu" then that Mailbox value becomes a TargetAddress and all aliases with the same MAILBOX value populate a ProxyAddresses list in the same way. If there was no O365 account, these go in the User object. If not, they populate Identity Variables that will generate a Contact object.
• Create one Contact object for EACH remaining Alias (that are not O365 or Eliapps) with one ProxyAddress equal to the AliasName and a TargetAddress equal to the MAILBOX. We do not attempt to combine entries that resolve to the same MAILBOX because these types of Aliases are almost always unique.

Unfortunately, the existing Alias table has some Netids that have a mixture of personal and departmental Eliapps accounts. We can looks for Google Accounts with a name like PrimaryAliasName, or PrimaryAliasName.eliapps, or Netid. After that, it is difficult to write a program that can reliably distinguish an alias name of "Alfred.E.Neuman" as personal but "Mad.Magazine" as departmental (and therefore an Alias entry that should have been moved to a Dependent Netid). For now we will follow the Shibboleth login rule that regards a PrimaryAlias as personal, but if the Primary is O365 then the first Secondary Eliapps alias is Personal (where it becomes the "special" Eliapps Contact where multiple Aliases are aggreated) and any subsequent Eliapps alias with a different MAILBOX is treated as Departmental and generates Contacts where we do not attempt to aggregate more than one Alias into the ProxyAddresses list. The Mail Routing will be done correctly no matter what. If we confuse personal and departmental then the number of Contacts can increase and the Outlook directory may not be optimal. Users can fix this by moving their damn departmental Eliapps aliases to Dependent Netids like they were supposed to.

Deprovisioning Departmental Contacts is not optional. Any Contact not supported by a current Email Alias entry has to be deleted. In fact, because the Azure AD will complain if any entries violate the rule that the same alias name cannot appear in the ProxyAddresses list of two objects at the same time, deprovisioning should occur before provisioning to handle the cases where Aliases are transferred from one Netid to another. Unfortunately, this is exactly the sort of thing IIQ does not handle well. So while we could continue the current model of building tools that update the Alias Table and then expect routing information to be generated in the background as is done for the current Mail Relays, future tooling should be designed to change the AD at the same time that the Alias table is changed.

It is proposed that Users are created as they are currently (CN=Netid, CN=Users, DC=yu, DC=yale, DC=edu), that Eliapps personal Contacts be created as (CN=Netid, OU=EliappsUsers,...) and that the remaining individual Aliases generate Contacts in (CN=AliasName, OU=Aliases, ...).

Inactive Accounts

There are 300K Netids, but only ~50K are active.

There is a big difference in performance between frequently aggregating all the users, or just the active users. An inactive user with no mail account does not change information frequently and there is no SLA on updating the last name in AD if an ALUM or retiree changes it. Therefore, it would be vastly better if we can agree to aggregate only the active users most of the time, and to update the inactive users (SOI=IDR and no Email Address) less frequently. Currently the TEST AD has such a scheme, but the PROD AD does not.

If inactive users can be deprovisioned from AD (which means they will need a PIN to reactivate), this would solve the problem.

It is a requirement that the code not create AD objects for inactive accounts (SOI=IDR) with no mail.

Bean Shell or SQL

The current AD Updater is driven by values generated by the YUONLINEDIR.PUBLISHED_AD_ACCOUNTS_V view in ACS1. It has logic of the form:

          DECODE (r.flag,
                  'G', ph.grad_student_phone,
                  'U', ph.campus_phone,
                  ph.office_phone
                 ) telephonenumber

Which basically says that the variable 'telephonenumber' will be taken from the grad_student_phone if the identity is a Grad Student, the campus_phone if the identity is an Undergrad, or else the office_phone for everyone else (Employees).

There are two ways this can be translated. We can create a comparable Database View only in IDR instead of ACS1 (which means translating Oracle SQL into SQL Server SQL) or we can create a Global Rule in IIQ that populates an Identity Variable named 'telephonenumber' from one of three IDR source fields (which means translating Oracle SQL into Java).

The primary difference is staffing (Java or SQL programmer) and maintenance (Change to the application or to the database).

We have to decide this before coding can begin. 

Out of Scope

For the moment, the management of the Google Directory (GADS) is out of scope. We expect this might change in a Phase 2 of the development rather than being added as initial work. Once Eliapps users have to have Contacts in the AD in order to be Email routed by Exchange Online, we have all the information needed to provision the Google Directory.

Service Accounts will not be managed. They have no interesting attributes, no source of identity, and we have nothing extra of offer.

Computer objects are currently handled by the Microsoft Domain.

Groups may be managed by Grouper.

We are interested in the Users container and the SOM OU. Anything outside those two areas will be ignored if it is well behaved. Generally speaking, if an object has a SAMAccountName outside the range of valid Netids and if it does not own ITS managed Email accounts, then we can leave it alone.

User and Contact objects not related to a Netid are out of scope. We may choose to declare that they are illegal in the Users container, but migrating them elsewhere is not a separate cleanup.

The Email Alias system will remain in its current form in ACS1. In some sense the information in AD duplicates the content of DIR_ONLINE_INFO, but AD is not able to provide the required function for generating unique Email Aliases.

Identities and Accounts

There will be an Identity object for every Netid (personal and dependent).

An Account object is created when aggregation correlates data from any of the six sources to an Identity. Correlation is by Netid, except for Azure AD where we will use the existing Email Provisioning correlation logic based on ImmutableID (because Azure doesn't store Netid).

Personal identities will have an IDR Account object.

Dependent netids will have an Netid System Account object.

At this time we believe that Contacts can be maintained as an Account under the Netid that owns the Alias that created the Contact. If that proves unworkable, then Contacts become a separate Identity.

We may choose to ignore personal Netid Accounts because they provide no useful information (the Netid is in IDR).

Each Email Alias is an Account that correlates to the Netid (personal or dependent) that owns it. Email Aliases are used to generate the proxyAddresses lists and drive Contact creation for a Netid with both O365 and Eliapps accounts.

Identity Variables

Although it is possible to provision fields from the IDR Account record directly into the AD object, the simplest approach is to create identity variables for each item that must be provisioned into the AD and Azure AD.

If we do not create Contact Identities, then when a Netid has both an O365 mail account and an Eliapps mail account, requiring the generation of a Contact to contain the routing information for the Eliapps account, then the Contact name, mail, mailNickName, proxyAddresses, and targetAddress  will be separate Identity variables from the corresponding variables that generate the same fields for the User object.

Indexed

• Netid
• UPN
• PrimaryAliasName
• ImmutableID (from AD, correlates Azure AD)
• SID (from AD, correlates Azure AD)

Needed for code selections

• YaleAffiliation (change EPA to distinguish Undergraduate, GradStudent, ...)
• O365Mail indicator
• EliappsUsername
• PrivacyIndicator

AD Fields (unindexed, from IDR)

• UPI
• Mail
• Surname
• GivenName
• DisplayName
• Title
• Description
• Department
• Company
• PhysicalDeliveryOfficeName
• Street Address
• PostOfficeBox
• Location
• ST
• PostalCode
• TelephoneNumber

Calculated

• O365Addresses (from AliasNames)
• EliappsAddresses (from AliasNames)

Mail Related Fields

Any non-mail entry (X500 or sip) in the ProxyAddresses list will be ignored by processing and will "pass through" from aggregation to provisioning.

The "SMTP/smtp" entries of the proxyAdddresses list will be generated by us and will replace all entries in the current AD. This will fix reported problems where the same value is duplicated in more than one proxyAddresses list belonging to different objects (which cannot be allowed if this information is used to do Mail Routing).

Because it is Yale Policy that an O365 account is Primary, and because the Primary Email Alias sets the UPN which is a key field of Azure AD, we do not allow an Eliapps account to be Primary if an O365 mailbox is assigned to the AD User object. Generally speaking, for an O365 user we will ignore the Mailbox (should it point to bulldogs) and associate the Primary Alias with O365.

Therefore, there are four AD configurations:

1. Primary is O365 and there is no Eliapps
2. Primary is Eliapps and there is no O365
3. Primary is O365 and there is an Eliapps secondary
4. Alias unrelated to ITS mail systems

There are some variables that need to be defined, and in a few cases the exact definition is technical:

• Netid
• PrimaryAlias is the type='person' Alias and has a PrimaryAliasName and a PrimaryMailbox.
• GoogleAccount is the name of the Eliapps account in the G Suite User Directory. If the PrimaryMailbox is of type 'bulldogs' then the Account is the part in front of "@". If the Primary is O365, then the Account is the part in front of "@" in the first "bulldogs" Mailbox we encounter in the Alias table, and if there is more than one bulldogs Mailbox the result is undefined.
• GoogleAlias is the AliasName from the Alias entry that set the GoogleAccount.
• O365Aliases are all the Secondary AliasNames that have a Mailbox of 'netid@connect.yale.edu'.
• EliappsAliases are all the Secondary AliasNames that have a Mailbox value of "GoogleAccount@bulldogs.yale.edu" except for the GoogleAlias.

The "Mail" attribute in AD is not a technical field. It is the Email Address to be "published" in the directory. At Yale it is the PrimaryAliasName@yale.edu and that happens to be the O365 account of a two-mailbox user because we require O365 to be Primary if it exists.

Primary O365

User Object

Primary Eliapps

User Object

Primary O365 with Eliapps Secondary

User Object (Same as Primary O365 above)

Contact Object

Foreign Mail Alias

Contact Object