[Treasury-News] Follow-Up on U.S. Bank Outage

[Brady Coy]

What happened?
On Tuesday, August 3rd, a power outage occurred at a U.S. Bank operation center in St. Paul, Minnesota. Treasury was informed of the incident early the morning of Wednesday, August 4th and began notifying agencies immediately using our existing Business Continuity Plan and program-specific communications plans. While we made initial contact with all agencies by 10:00 a.m. regarding potential funding and posting delays, we continued to learn more about the scope of the immediate impact throughout the day. Treasury maintained close contact with U.S. Bank throughout the day, and attempted to provide updates and information about transaction processing decisions to agency staff as quickly and efficiently as possible. At times we had to balance communication about an action with taking the action itself.
As you know files from U.S. Bank (U.S. Bank Deposit file, Incoming ACH file, ACH Returned Item file & Redeemed checks file) were not available to Treasury and therefore could not be processed through our systems on Wednesday, August 4th. Additionally, ACH transactions sent by agencies to Treasury on August 3rd with an effective date of August 4th were delayed in posting to customers' bank accounts until late that afternoon or, in some cases, until the following day. ReliaCard funding was also delayed, and was corrected by approximately noon on August 4th.
Resolution
File processing problems related to the incident were resolved by Thursday, August 5th, with the exception of Thursday's U.S. Bank Deposits file.  This file was not posted until Friday due to a backlog of processes running at U.S. Bank.   The effective date of all transactions has been backdated at Treasury to ensure interest calculations were the same as if the delays had never occurred.  Treasury is still in the process of ensuring everything was processed correctly and that the incident did not negatively impact our agencies' customers and clients, e.g., bank account overdrafts due to funding delays on August 4th.  We will continue to work closely with U.S. Bank and our agency partners to resolve such issues. In fact, the following statement was prepared by the bank for release:

Late in the day on August 3rd, U.S. Bank experienced a power outage, which caused a number of delays in the processing of transactions and reporting.  There may be situations where this service disruption caused Agencies, or perhaps your constituents or vendors, to experience adverse financial implications.   U.S. Bank is committed to addressing and resolving these situations. This may include overdrafts, late fees or other costs that would not have occurred with timely posting of transactions. Treasury will be the point of communication with U.S. Bank for requests for resolution.

What next?
While we could not have prevented this incident, we believe our Business Continuity Plan functioned adequately with respect to identifying and resolving specific business problems and disruptions and, in most cases, communicating those issues with our agency partners.  We do believe, however, that some improvements can be made with respect to communications with our agency partners. Notably, we see a potential need to distinguish some of the most technical information requirements from that which may be more useful to our partners on the programmatic frontlines. We plan to meet with our Business Continuity team in the next week to capture and make improvements to our processes based on lessons learned.  We plan to seek your input about how Treasury performed during this incident, and how we could improve. Please look for additional information about how to share that input, as well as details regarding requests for resolution of financial impacts related to untimely transaction posting, in our August 2010 Cash Management Newsletter, which will be published later this week.
In the meantime, we sincerely thank you for your patience last week and look forward to working closely with you to further improve our processes.





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omls.oregon.gov/pipermail/treasury-news/attachments/20100809/689079ad/attachment.html>