Testing Resilience: The Impact of a Building-Wide Blackout Test

The emergency generator system often falls within the commissioning agent’s (CxA’s) scope of work, requiring the CxA to oversee field quality control testing such as load bank tests and equipment startups. Functional testing typically involves transitioning a building from normal power to emergency power and back, validating that all systems reliant on emergency power operate as intended. 

However, what happens to systems that aren’t on emergency power?

This case study explores a recent project where SES was asked to coordinate and execute a building “blackout” test at a secure government facility. The test aimed not only to confirm the operation of the emergency generator system but also to observe and confirm how the entire building, including kitchen equipment, elevators, security cameras, access controls, IT infrastructure, and central plant systems, reacted during a power outage. 

Project Background & Test Preparation

Coordinating a blackout test at a secure site presented unique challenges due to the stringent security protocols in place. This site required thorough background checks and badging processes for all personnel involved, which added time to scheduling and complicated logistics. Additionally, restrictions on phones and cameras in certain areas limited real-time communication and documentation, requiring alternative methods for coordination. The need for escorts in secured zones further complicates the movement of team members, adding layers of complexity to the test execution. These factors make planning and executing a blackout test significantly more demanding than at a standard facility.

To overcome these challenges, SES and the project team began planning the blackout test months in advance. A stakeholder meeting was held to debrief the team on the testing intent and desired outcomes. Representatives from the client’s IT, security, facilities, and public safety departments were also introduced to the project construction team to discuss areas of overlap and potential issues.

One of the primary challenges in preparing for the blackout test was coordinating with multiple parties and companies involved in this secure facility. SES played a crucial role in identifying which equipment in the building was on emergency power and which was not, confirming that everyone involved knew what to expect when power was dropped. Given the facility’s active use, the test also had to be scheduled for a day and time that would minimize disruption to the site’s operations. Representatives from each company and department were required to be present, and SES issued a detailed schedule and test plan were distributed to the team. The schedule & test plan included an outline of when power would be dropped, when it would be returned, and which systems and equipment would be assessed.

Safety was a significant concern due to the secure environment. The client was particularly worried about the possibility of individuals becoming locked in certain areas. To mitigate this risk, the team implemented a “buddy” system, and radios and flashlights were provided to key team members. A designated muster point was also in place for the team to rendezvous at. 

Execution of the Blackout Test 

Before disconnecting power, all systems were reviewed and verified to be operational, ensuring that any issues identified during the emergency power walkthrough could be directly attributed to the power switchover. At the designated time, power to the building was disconnected to the building at the main switchgear. Team members stationed at the generator and automatic transfer switches (ATS) immediately radioed in to confirm that the system reacted within the specified timeframes. The team then dispersed to validate their assigned systems.

The emergency power walkthrough for this project included verifying the following:

• Security Systems: Communication with security cameras and intrusion detection devices was confirmed, with the security department reviewing all interactions.
• Access Controls: Locksmiths checked the operation of keycard access systems and fail-open positioning.
• Fire Alarm System: The functionality of the fire alarm system was verified.
• Kitchen Equipment: The operational status of kitchen freezers, refrigerators, and other critical food storage systems was inspected.
• Emergency Lighting and Power Outlets: The project team walked the building to check each emergency light and power outlet.
• Elevators: The elevator company confirmed the recall and operation of elevators designated for emergency power.
• IT Infrastructure: IT data and communication systems were validated to ensure uninterrupted service.
• Building Automation System (BAS): SES reviewed the BAS to confirm that the main server remained operational and that all HVAC systems on emergency power continued to function correctly.

After completing the emergency power walkthrough, normal power was restored to the building. The team conducted a final walkthrough to confirm all systems had returned to normal operation without issue.

Outcomes & Findings

At the conclusion of the test, the team gathered to discuss their findings. The blackout test revealed several issues that were critical for the facility’s ongoing operation:

• Elevator Malfunction: An elevator designated for emergency power was found to be inoperable due to an internal programming issue.
• Power Discrepancies: A section of emergency power outlets serving vending machines was discovered not to be on emergency power as expected.
• HVAC System Issues: A piece of HVAC equipment was mistakenly not connected to emergency power.
• Security Concerns: Security gates controlling access to the building were not on emergency power and remained locked, presenting a potential life safety risk.
• Manual Resets Required: Some central plant systems, such as steam humidification boilers and water-cooled chillers, required manual resets to return to operation after the switch back to normal power.

SES documented these findings and provided the client with a comprehensive report detailing the results of the blackout test. The report included instructions for the facilities staff on what to expect during a power loss and recommendations for addressing the identified issues.

Conclusion

The building blackout test was a thorough examination of the building’s resilience during a power outage. The test not only confirmed the operation of the emergency generator system and equipment on emergency power but also highlighted the importance of understanding how all building systems react during such events. The insights gained from this test will help the facility’s management prepare for future power outages to sustainably maintain the safety and functionality of the building.

Lessons Learned

Following the completion of the testing, there were some notable task items that directly contributed to the successful execution of the test as outlined below:

• The success of the test hinges on thorough planning and coordination among all stakeholders. Starting the planning process months in advance allowed SES to address potential challenges and align all parties on the test’s objectives and procedures.
• Clearly identifying which systems are on emergency power and which are not is essential. This understanding helps set realistic expectations for the test and ensures that all systems are appropriately monitored during the power outage.
• Verifying that all systems are operational before dropping power validates that any issues identified during the test can be attributed to the power transition, rather than pre-existing conditions. This step is critical in diagnosing problems accurately.
• The secure nature of the facility required specific safety measures, such as the “buddy” system and the distribution of radios and flashlights. Customizing safety protocols to fit the environment is essential for the safety of all participants and allows for easier “buy-in” from the higher ups.
• Involving client representatives from IT, security, facilities, and public safety departments guaranteed that all critical systems were covered during the blackout test. Cross-departmental coordination enhanced the thoroughness of the test and ensured that all relevant perspectives are considered.

If you’re interested in specifying a blackout test on your project, contact SES today. Our team is ready to assist with comprehensive testing and tailored solutions to meet your unique needs. Reach out to SES at [email protected] to discuss how we can support your project.