RFG’s 8 Lessons from Sandy

1. By-and-large, plans worked. Major hosted solutions were up and running and major data centers were resilient for the most part.
2. We got lucky.  As bad as Sandy was, there was lead time; mandatory evacuations moved staff into areas where communication remained possible, mid-town N.Y. had power, and the N.Y. markets were closed for two days.  It’s impossible to know what the situation would have been if things had rolled out differently.
3. BCP/DR needs to be customized. Strategy counts.  PE funds and others with long buy and hold cycles may have the luxury of waiting for infrastructure to return.  HFTs need to consider co-location and multiple hot sites. Of course, for HFTs, latency issues resulting from the use of remote locations may come into play. Commodity advisers need to consider market movements caused by the specific crisis. One size does not fit all.
4. Home is not a remote location. Or, it may not be. Expecting people to work from home becomes a true wild card when work locations and homes are so thoroughly impacted.  Also, home office technology is more susceptible to malware—and, sadly, attacks increase during times of crisis.  It seems that only a few senior executives have dedicated home computers that are adequately-protected on this score.
5. Testing needs to involve all aspects of backup processes. Some firms had staff arrive at backup locations only to find firewalls preventing access to servers or a shortage of remote licenses and “hot seats”.  Expect increased focus on what is required during a major outage. Also, many firms could not rely on the availability of IT or other critical staff.  In cases where these staffers were out of communication, any lack of well-documented procedures made problems particularly difficult to solve. Documented procedures need to be tested for ease of use as part of an overall BCP/DR program.
6. Testing needs to involve different time periods.  Sandy hit at the end of the month, a time with its own unique business needs—for example, investors coming in or adding on new investments.  There was notice and time to prepare.  A disruption at a different time (year-end, for instance), or with less notice, may cause different problems or require different solutions.  Think about timing as much as geography when designing one’s BCP/DR plan.
7. Plan for the possibility of longer outages.  Many advisers came to the conclusion that their existing plans were short-sighted.  This applies to the immediate situation and, also, to the after-effects—in terms of travel time and diversion of key resources.  We heard reports of one firm with a server in its basement and a backup on a higher floor.  The backup worked in the short term, but with elevators out and a problem that lasted longer than anticipated, generator fuel ultimately could not reach the higher floor and the firm lost access.  Different plans are required for one day, one week, one month and longer disruptions.
8. Investors will be focusing on service providers. Several funds had power but found that service providers were down.  Phone and voicemail were impacted at some firms. Also, some custody banks were inaccessible, either because they were actually down or because they were so thinly staffed that business activities slowed.  Other funds relied on third -parties as part of the investment process, but when these providers went down, they needed to obtain market pricing and portfolio analytics elsewhere.  The lesson here is that there is a difference between a written policy and actual practice. Lots of policies require firms to assess third-party vendors’ BCP/DR, but sometimes little is done in reality. Investors and managers will be focusing on service providers; the quality of their backup systems, regional redundancy and plans in place for future outages.

Our clients who spent time and resources developing robust BCP/DR plans over the past few years often felt vindicated by their efforts.

Obviously, if you could predict the exact scope of a future crisis you would take steps to prevent it.  Nevertheless, there are a number of tasks and issues that are common to all crises.  For that reason, RFG Pathfinder® contains a section on crisis management.  Find it here.  Or, purchase our dedicated discussion of these issues in our Crisis Management Workflow, which can be found here.

Our friends at Eze Castle Integration also posted a piece about what hedge funds can learn from Sandy, available here.