In today’s connected world, data centers are the backbone of our online infrastructure. Hyperscale data centers are massive facilities that house thousands of servers, providing the computing power and storage capacity needed to run many of the applications that we depend on every day.
To keep these centers running smoothly, it is essential that the electrical and mechanical systems be commissioned by a knowledgeable partner. To put it simply, commissioning is a critical part of the construction process that ensures these complex systems are fully operational and performing as expected before the data center goes live.
In this blog post we’ll explore the commissioning process and its importance, plus some insights from recent commissioning projects.
Understanding the Commissioning Process
Commissioning is a quality assurance process that verifies the performance of critical systems in a data center. It involves several key levels that are performed to ensure that the systems meet the design intent of the facility owner and operate reliably under real-world conditions. The industry has multiple different Commissioning Level definitions with varying level descriptions, however a common description of the levels of commissioning (L1, L2, etc.) are:
- Level 1 (L1): Factory Witness Testing (FWT)
- Verification at the factory that the equipment meets the design specifications before it is shipped to site
- Level 2 (L2): Delivery and Pre-Installation Testing
- Verification that the equipment was received undamaged, placed per the manufacturer and design specifications, and is ready for energization
- Level 3 (L3): Pre-Functional Testing (PFT)
- Verification that the mechanical and electrical systems are individually performing as expected
- Level 4 (L4): Functional Performance Testing (FPT)
- Verification that the systems operate as intended under a variety of scenarios and fault conditions
- Level 5 (L5): Integrated System Testing (IST)
- Verification that all the individual systems operate together per the design specifications; this testing typically includes full load testing and allowable overload/stress testing
Electrical and Mechanical Systems Commissioning
The electrical infrastructure in a hyperscale data center includes critical components like Uninterruptible Power Supplies (UPS), medium and low voltage switchgear, rack and floor mounted Power Distribution Units (PDUs), transformers, generators, and control systems such as the EPMS. Each of these elements must function properly to provide the necessary power and redundancy for the data center to operate properly.
The mechanical infrastructure in a hyperscale data center includes HVAC (Heating, Ventilation, and Air Conditioning) systems such as chillers, pumps, Air Handling Units (AHU), Computer Room Air Conditioners (CRAC), Computer Room Air Handlers (CRAH), exhaust fans, and the mechanical control systems such as the BMS.
The TechSite team develops testing schedules and confirms the commissioning test scripts. Script updates are made as necessary to ensure the equipment is properly and thoroughly tested per the owner’s requirements. We then follow the testing scripts and document the results of each test. As part of the data center commissioning, some of the tests, checks, and verifications that TechSite performs are:
- Confirms the correct equipment is installed and in the correct locations
- Confirms equipment isn’t scratched or damaged
- Checks that Power Quality Meters (PQMs) built into equipment are showing correct voltage, current, power values
- Performs IR scanning to ensure electrical connections are correctly made/torqued
- Confirms breaker and relays settings are set per the Short Circuit and Coordination Study (SCCS)
- Checks that HMI screens display the correct information
- Performs fully loaded/stress tests on the equipment/systems/building
- Works with vendors/equipment providers to confirm Field Service Bulletins (FSBs) have been implemented on the installed equipment
- Reviews L2 and L3 documentation provided by equipment techs/providers
- Confirms adequate cooling and ventilation of the IT equipment spaces
- Checks sensor locations and verifications
- Confirms accuracy of the monitoring, alarm and control systems
- Tests redundant functions in the power distribution and cooling systems
- Confirms proper pressurization in the IT equipment rooms
- Creates observations on failed tests/items, follow up on repairs, and retest when repairs are made
While commissioning, the TechSite team runs the systems through many scenarios to see how they respond. The goal is to confirm that the systems respond in a predictable manner dictated by the owner’s project requirements and the Sequence of Operations (SOO). As an example, for switchgear the first test might be to remove utility, see the backup systems start and transfer to the generator. Then we may return to utility power and see the system transfer back. Multiple iterations of the same test may follow with failure of other items, for example a generator fails to start, a key breaker fails to open or close, or a control panel loses power. All of the different scenarios have an expected outcome that is being tested for and this ensures the system as a whole operates reliable and works as intended. When the new facility passes these tests, the owner can be sures that it works as intended.
Issues Uncovered in Recent Commissioning Events
TechSite’s commissioning agents and engineers regularly uncover issues on site that need correcting, ranging from mislabeled equipment to equipment not operating as intended. Some recent examples from our Commissioning Team include:
- We commonly find that the labeling for equipment is missing or is displaying incorrect information. This could be anything from phenolic labels for lights on a control cabinet that denote redundant power supplies being swapped with each other, to entire pieces of equipment with an incorrect label, or incorrect panel schedules. This can lead to operational issues where incorrect breakers are opened and critical equipment is deenergized when it shouldn’t be.
- A big part of our commissioning is simulating alarms and confirming they are showing up on the Electrical Power Monitoring Software (EPMS) or Building Management System (BMS). For example, we will jumper two terminals in a control panel to simulate a “Low Fuel” alarm on the generator. We should then see a “Low Fuel” alarm on the EPMS. This allows facilities staff or remote monitoring personnel to see when critical alarms come through. We’ve found alarms that show up as incorrect alarms or do not show up at all, commonly due to improperly landed sensor wires. If that happens, the facilities’ staff is not able to respond to issues or properly maintain the system.
- Recently our team was testing an AHUs response to calls for emergency cooling. The emergency cooling mode calls for the AHU to open supply air dampers to 100% and ramp fan speeds up to 100% to maximize cooling in the IT space. However, while the AHU would display it was initiating this mode on its HMI screen, the supply air dampers had shut completely, and the fans would not ramp up at all. If this happened during live operation of the data center, the IT equipment could overheat.
- On another project, the team was testing for sump high water level alarms in the AHUs. This required closing the drain valves and filling the sumps with water to trigger the alarm. On the AHU, there is an additional drain that allows excess water from the unit’s evaporative mesh to drain into the sump. This additional drain is called the stage header drain. However, the stage header drainpipe was too long which caused water to shoot out of the sump and into the AHU. The installation team was aware of the issue but had not scheduled a fix until it was caught during commissioning. Had this not been discovered, the unit could potentially have incurred flooding issues during operation.
- While testing an AHU, our team found that the programming on the unit had to be corrected. After the changes were made the unit was successfully retested.
Post-Commissioning Activities
Once the commissioning process is complete, several post-commissioning activities are necessary. This includes creating the commissioning reports with the test results and all issues documented. Depending on the customer, those can be uploaded into a portal or provided directly back to the end users commissioning team. This documentation is not only essential for ensuring the owner’s design standards are met, but also serves as a valuable resource for future maintenance and troubleshooting during the lifetime of the equipment.
Conclusion
Commissioning electrical and mechanical systems in hyperscale data centers is a critical process to confirm the data center operates as intended. It ensures that these critical systems are fully operational, reliable, and will work under a variety of real-world conditions. By following a thorough commissioning process with a trusted partner such as TechSite, data center operators will be ready and confident in their ability to support today’s digital world.
For more information on TechSite’s commissioning services please contact us, our team of experts is ready to help!