Business Continuity & Recovery Plan Summary

Transcription

1 Introduction An organization s ability to survive a significant business interruption is determined by the company s ability to develop, implement, and maintain viable recovery and business continuity plans. Southwest Business Corporation (SWBC) has instituted a comprehensive Business Continuity Program that addresses all business continuity phases following a significant incident: response, recovery, and resumption. These plans are supported by Response Teams (Emergency Conditions Team, Damage Assessment Team, and Incident Command Center Teams), Recovery Teams for voice and data technology, Business Continuity Teams to recover the critical business processes, and Resumption Teams to expedite the return to normal business operations. I) Scope and Objectives The scope of SWBC s Business Continuity Program plan is to centralize the methods and procedures of business continuity and recovery when a significant business interruption affects the business and services of SWBC Headquarters, Service Centers and other offices within the United States. The areas covered in this Plan are: Resources and vendors necessary to recover critical functions Responsibilities for business recovery activities Critical voice, systems, and applications needed to be restored within predetermined timeframes, taking into consideration interdependence of applications and communication systems. Necessary materials that should be stored off-site in case of an emergency The overall Objectives of the Business Continuity Program plan document are: To protect personnel, assets and informational resources from further injury or damage Minimize economic losses resulting from significant interruptions to business functions Provide a plan of action to facilitate an orderly recovery of business operations To meet business continuity and IT recovery timeframes II) Risk Assessment and Business Impact Analysis Business continuity and IT recovery planning continuously evolves during the normal course of business. To begin the planning process, SWBC conducted a Risk Assessment (what may cause an incident) and Business Impact Analysis (what is the impact of an incident) for Headquarters and Service Centers which led to the Business Continuity and IT Recovery Planning process. In addition to these assessments, SWBC performs risk assessments across all business units in the enterprise in conjunction with the SSAE 16 SOC-1, Type II Audit that is conducted on an annual basis.

2 The Risk Assessment process involves identifying, assessing and ranking risks based on probability and severity. Once ranked, mitigation strategies or plans for action are put in place. SWBC Risk Mitigation efforts include but are not limited to redundant systems configurations between HQ and SC1 for voice and data, UPS, fire extinguishing mechanisms (chemical and water based), temperature level controls and flood water sensors in data centers, redundant power supplies, smoke detectors, humidity level controls and multi-zone HVAC, and generators. HQ and SC Risk Assessments will be performed each year to assess any significant changes. The risk of a Pandemic event requires a significantly different strategic approach and therefore SWBC has developed a separate SWBC Pandemic Business Continuity Plan that leverages this plan s Team structure. SWBC s Business Continuity planning is based on results from a detailed Business Impact Analysis (BIA). The initial BIA included various interviews with management. Based on this analysis, Management determined the various recovery time objectives (RTO) and recovery point objectives (RPO) for business continuity planning efforts. A Business Impact Analysis (BIA) is performed each year to ensure SWBC s Business Continuity planning continues to be aligned with changing recovery requirements within SWBC s dynamic business environment. SWBC s plan is designed to restore Critical processes within 24 work hours of an outage and Essential processes within 72 hours of outage, thus reducing the potential impact of a significant event to an acceptable level. III) Business Continuity and Recovery Plans SWBC s Chief Information Officer (CIO), with assistance from key Company support areas, is responsible for developing the SWBC Business Continuity and IT Recovery Plans. Ongoing development and support of departmental plans are the responsibility of the functional area planning for recovery. SWBC s IT department works to ensure that the recovery efforts adequately meet the requirements of the business continuity plans. Business Continuity and Recovery planning employs a three phased approach to respond and recover from a significant incident : Response, Recovery and Resumption phases. SWBC Business Continuity Plans incorporate various strategies to combine people, critical processes, and technology to ensure continued business operations during a significant interruption to normal operations. All departments with critical business processes have Business Continuity Plans that address the following critical requirements: team members, critical business processes, system applications and software, RTOs and RPOs, vendor and client notification, telecommunications, alternate locations, manual processes, reports, documents, supplies, equipment, recoverable items and regulatory requirements. Exercising business continuity plans is a daunting task, but of the greatest importance for plan maintenance and employee training. Exercises serve as the primary training opportunities to prepare employees for plan activation as well as validate the overall plan effectiveness. All plans have a Plan Owner assigned who is responsible for ensuring that the plan is exercised at least annually. Businesses are encouraged to increase scope or complexity with each subsequent exercise of their plan. Each business continuity plan has identified voice and data systems that are required to perform critical businesses processes within specified RTO (recovery time objectives for systems) and RPO (recovery point objectives for data). SWBC s Information Technology departments use this information to prepare Recovery plans to ensure the critical systems are available to support the businesses objectives. The SWBC IT Recovery Plans encompass the processes, policies, procedures, hardware, and software necessary for restoring voice and data systems capabilities that are critical to SWBC business units. IT Testing cycles are predicated by the

3 functionality or process being tested. Although some tests may occur 2 4 times per year, others are required to be tested on a weekly basis. Processes may be subjected to individual tests specific to a department or BCP Team and/or more comprehensive tests that could include testing of numerous aspects of the Plan during a single test. The following provides detail regarding SWBC IT Recovery provisions and related technical support necessary for the successful execution of the BCP: Both San Antonio based facilities are equipped with additional voice and data cabling infrastructures to provide emergency access for personnel from other facilities in the event of an outage at any location. SWBC s primary production system, an IBMi Power 740 Express model 8205-E6D running the IBMi operating system, v7r1, is located at its Corporate Headquarters in San Antonio, Texas. A redundant IBMi Power System resides at the SWBC Service Center, a separate facility. Depending on the time of year, this may be reversed. All disk, memory, and controllers are internal to the system. There are two tape drives on each system that are LTO4 fiber channel libraries, model 3573-TL. These two systems are configured to replicate key stroke changes in a near real-time environment over redundant 1GB Ethernet point to point circuits to ensure ongoing access and availability to business systems in the event of an interruption. Mission critical Windows servers are attached to redundant Storage Area Networks (SAN) with hardware level replication to ensure that customer-facing B-2-B websites and customer data will be available in a failover scenario when our customers need access. SWBC has stand-by generators located at the Corporate Headquarters and the SWBC Service Center to sustain operations in the event of a power outage. UPS capacity is sufficient for an orderly transition to generator power or graceful shutdown of equipment at HQ and SC1. UPS starts within 18 seconds of a power transfer to UPS which allows for a graceful transition to generator power. To mitigate the risk of temporary loss of power during the delay, laptops have battery backup or personal UPS hardware resides at selected workstations. The internal Telecommunications infrastructure at SWBC includes redundant Avaya 8730 servers installed at the SWBC Service Center 1, redundant Avaya 8720 Enterprise Survivable Servers (ESS) installed at the Corporate Headquarters and Avaya Local Survivable Processors (LSP) support Service Center 2 and other SWBC s offices. SWBC also utilizes Avaya hardware and software solutions to enable SWBC representatives to work from pre-configured computers via a secure VPN connection established to the corporate network. This solution, using Avaya IP Agent software and an Avaya VPN Phone, provides the same contact center functionality found within the office environment, including the ability to receive skill-based calls, monitor live call statistics, and record customer voice and data interactions through Witness. The external Telecommunications infrastructure consists of diverse Carriers, Paths, and Services. Long distance and MPLS circuits are installed at SWBC Headquarters and the Customer Service Center. Digital Signal 3 s (DS3) delivered via an Optical Carrier 12 (OC12) are implemented at both locations. Local and long distance services are delivered by multiple Digital Signal 1 (DS1) circuits at SWBC Headquarters and the Service Center.

4 IV) Redundancy and High Availability Web, Network, and Windows Systems At SWBC, disaster recovery and business continuity plans exist as living models and are continually changing and adapting to support business objectives and to provide our customers with timely access to computing and telecommunications services. The growing dependence on information technology that supports these processes requires diligence in planning and the implementation of strategic solutions to ensure disaster preparedness. An intense focus on business continuity, including a concentration in solutions to support the technology infrastructure, has enhanced disaster recovery strategies and resulted in a more robust position in both areas. SWBC has made a commitment to provide high availability solutions in answer to the many scenarios that have the potential to impact operations. FocusNet, SWBC s client portal, and the critical systems that support it are designed in a redundant configuration. Data replication between production and Disaster Recovery (DR) systems reduces recovery time to 8 hours, or less. Other business critical systems in the Windows environment, along with data, are replicated to redundant systems implemented at the DR site. This is accomplished through both redundant physical and virtual servers. IBM iseries (AS/400) SWBC maintains redundant, core business systems in support of our Lender Placed Operations (LPO) services. These redundant IBMi Power 740 mid-range systems run IOS v7r1 and are configured to allow for role-swapping, primarily in the event of an outage at either the production or DR site. In this environment, replication is realtime, with only a slight delay of up to 2 hours to accommodate for the most recent processing that occurred. The swap requires 6 8 hours to complete, and is tested semi-annually. Circumstances that may require a role-swap (outside of semi-annual testing) would include a disaster that doesn t allow for access to the production site, or a hardware failure, such as a multiple disk failure. Maintenance and service contracts are in place with IBM to ensure that in most cases the system can be repaired within 8 hours. V) Incident Management SWBC has adopted the Incident Command Center structure to expedite response and communication. The Incident Command Center (ICC) structure provides the back-bone of support through all Incident Phases: Response, Resumption and Recovery. The ICC structure is closely aligned with local emergency responders to help facilitate communications for a faster recovery. The Incident Command Center Team is comprised of members of SWBC s Senior Management team that fill the role of ICC Team Leads. The Incident Command Center (ICC) Team Leads will be responsible for declaring a significant incident and activating the Incident Command Center and associated Business Continuity and Recovery plans. The ICC Team will manage all activities associated with the event until such time operations have been returned to normal. The SWBC Headquarters building serves as the alternate or backup site for the Service Center 1, and vice versa, depending on the scenario. Alternate site requirements have been identified by each critical business based on

5 Loss of Building Scenarios. Most Business Continuity Plans use a Home Team strategy to provide Day 1 support. Other strategies may include relocation to another field office or alternate site during a Loss of Building situation. Alternate locations and resource requirements have been pre-identified. VI) Client Notification Communication with SWBC s clients and vendors is of utmost importance. The Business Continuity/IT Recovery Team will assist individual business units with initiating client notification campaigns as needed. Most business units will use the Corporate Emergency Notification System, MIR3, while others have contracted a 3rd party hosted software provider to perform Client Notification. Additionally, SWBC websites may be used to provide alerts and status updates to the client. Communication with vendors will be performed over the telephone or by as applicable. Critical vendor contact information has been identified in the plans. VII) Employee Notification SWBC will use emergency notification software to notify employees of an incident or activate the recovery plans. Notifications can be launched by phone, web or and sent to phones, s and SMS text all at once or as preferred. The software can also be used to allow members to hear a message and opt into a teleconference. Pre-determined messages and message groups have been created and stored. Messages can be edited or ad hoc messages can be used. Responses are monitored and reported within the software. Utilizing the employee notification software allows SWBC to initiate the communication process to all employees regardless of their level of BCP involvement. Employees can also monitor the recovery status through the Employee Information line or the special SWBC website. VIII) Program Management and Authority Core Incident Command Center (ICC) Team Leads, comprised of members of SWBC s Senior Management Team, have been established to provide leadership vital to the success of SWBC s Business Continuity Program. SWBC s CIO, with assistance from the Business Continuity Program Manager and key support areas (i.e. IT), is responsible for oversight of SWBC s Business Continuity Program and associated plans. The leadership provided by these individuals is vital to the success of SWBC s ability to recover and continue business with minimal impact to customers/clients. While the responsibility for facilitating and ensuring the success of the plans lies on the Incident Command Team, the final approval authority of this document resides with the Company President and Chairman. Company President & Chairman Gary Dudley, President Charlie Amato, Chairman