Table of Contents

Background

There has been a lot of confusion, and a lot of us talking past each other regarding update set management, and how the platform limitations with regard to update set management apply to a 'best practices' SDLC process. Many team members have worked with other projects on platforms where rolling back an environment to a previous known state is trivial and quick.

...

To Preview an Update Set, choose the Update Set from the Retrieved Update Sets view, and then click "Preview Update Set". A series of screens will indicate progress, and there will then be a screen to review the status of the Update. This feature is called "View Update Set Preview list" Often times a report will indicate that the Update Set contained change elements that modify an existing element, and you will be asked to confirm the change. 

Confirm any conflicts

On the View Update Set Preview list view, there is a Proposed Action field. The ideal scenario is 'Commit'. Sometimes during a conflict with a particular element, the platform will suggest 'Ignore'. Ordinarily this is a mistake, and we should review the conflict, confirm we wish to move forward, and modify the Proposed Action for that conflict to 'Commit'. We should never advance an update set partially by leaving some conflicts to 'Ignore'. If the Admin tries to do so, the system will stop the advance, and warn the Admin. The conflicts should be regarded as a "Yellow light" rather than a "Red light", even though they are colored red. The system tries to stop you from shooting yourself in the foot, but this feature is more annoyance than helpful.

Commit the update set

After fixing up the conflicts, the  Preview list Proposed Action should be entirely set to Commit. Go back to the Retrieved Update Sets view, choose the same Update Set, which is now in state Previewed. Commit the Update Set with the "Commit Update Set" feature. The Admin will see a series of progress screens. If the Admin returns to the Retrieved Update Sets view, that Update Set will now be in state Committed.

Backing out the update set

If we return to the Retrieved Update Sets view, we will notice the Update Set will now be in state Committed. Then we realize we applied the wrong update set. We wish to Back Out.

To Back Out, we need to change to the Local Update Sets view, and sort by the field Created, ordered by most recent. 

Only the most recently-created update set has the Back Out feature. This is a platform limitation. We do not have the ability to Back Out just one particular thing that was applied thirteen or an arbitrary number of update sets ago.

It is possible to chain the Back Out feature, and back out multiple update sets, but doing so gets very tricky very fast. To do so, you most follow a last in-first out process, where to back out the thirteenth-ago update set, you first need to back out the twelve that were applied more recently, than you need to reapply those twelve in the appropriate order. Recall that apply order is essential to NOT reintroducing defects that you have already solved with later update sets. 

In the process of backing out several Update Sets, you are likely to break other requirements and features that are wholly unrelated to the issue you are trying to solve. If you tried to chain Back Outs in Production, you would be taking certain features offline, or reintroducing bugs. If you tried to chain Back Outs in TEST, you disrupt the testing process, and testing can not resume until the disruption is complete. And again, chaining Back Outs is a very delicate operation, and things that get disrupted need to get reapplied in exactly the same order in which they were originally applied, or we may introduce a new defect. In short, trying to chain Back Outs should be an item of absolute last resort, as there is almost aways a less disruptive way to accomplish your goal.

Alternate ways to disable code than Back Out

If Back Out should be a process of last resort, what should we do instead? It depends upon our goals. And this is where I think a lot of people were confused by Allen's advice, and thought he was contradicting himself. The best course of action depends on why we don't like the code.

This code is pretty good, but there is a not-very-bad defect

In this case, a previous approach Yale took was that we documented the defect in HPALM, applying a defect number, and generating screenshots or any other steps needed to demonstrate the issue. Then we would leave the defective code in place in TEST, providing the defect was limited and would not interrupt testing or possibly cause issues with other testing.

The root cause would be isolated, the fix would be generated in a new Update Set in DEV. The fix would be unit tested in DEV, and then advanced to TEST using the process illustrated above (set Update Set Complete, Update Source, Preview, Resolve Conflicts, Commit). Then tester(s) would review the fix and either mark the defect resolved in HPALM, file a new defect, or say that the fix does not resolve the existing defect. The process may be repeated a second time if the defect still only affects the feature in question and does not have impact on the remaining system.

Benefits of this approach: testing is not interrupted, testers can continue testing, and developers can focus on fix.

I recommend this approach for R812. R812 has a minor defect, caught in UAT. There is a known defect, with a fix ready in DEV. I recommend we apply the fix to TEST, and if approved, that R812 and the defect fix launch on this release.

Root cause identified, and it's one portion of one requirement. We want to keep fixing the requirement until we get things right.

This code is pretty good, but there is a defect so bad that it affects other testing

In this case, a defect has been found that affects multiple modules, and testing cannot continue. The testers should file a defect and immediately notify the development team of the issue. Development team needs to begin Root Cause Analysis. Testing has stopped anyway, and we are less reticent to cause disruption, and more concerned about fully eradicating the issue. There are multiple approaches to solving the problem, and they depend on the results of the root cause analysis.

Root cause identified, and it's one portion of one requirement. We want to keep fixing the requirement until we get things right.

In this situation, there are two steps. Step one is to disable the portion of the requirement that is causing the defect. This would mean that we would also likely remove the feature from that code, but it ends the disruption to testing, and allows testing to resume. If the root cause was a bad client script, we could create an update set in DEV that disables the bad client script, advance that update set to TEST, and have testers confirm that the TEST system is back to normal and testing can resume on other items.

Step two is to create a new version of the broken feature in DEV, unit test for meeting the original functionality, and test that it does not reintroduce the defect, and then coordinate with testing team about appropriate time to introduce the feature fix update set to TEST. If this happens to PRE-PROD, the fix will always be the most recent code, so when the code fix is ready and approved by testers, it can be tacked on to the end of the release in PRE-PROD, no clone needed, and added to set of updates in the release.

Root cause identified, and requirement is going to get yanked from the release

Sometimes we determine that a requirement is so broken that it cannot be fixed in time for this release, or for other reasons it is being yanked from the release. In TEST, the appropriate action would be to disable the code. If the requirement was a Service Catalog Item, we can toggle the Catalog Item to non-Active, and the Item will disappear from the Service Catalog. If the requirement only included a Catalog Item, this is sufficient for TEST. If there were Business Rules, or database changes, or other more complex changes, it may be necessary to clone over TEST to fully eradicate the code and perform a known-clean basis to reapply the release, minus the items that will no longer be part of the release.

In PREPROD, if we have to actually remove code, there are again two approaches:

There is nothing wrong with the code, we just don't want it available to the customers yet

In this situation, the most appropriate fix is to leave the code in the release, and apply just a small update set to disable the code. In the case of a Service Catalog Item, an Update Set that toggles the Catalog Item to inActive is sufficient. When we are ready, we can have a future release where we have a small Update Set that toggles the Catalog Item back to Active. This is my recommendation for