One of the most popular games available right now is a variation on chess, called 5D Chess with Multiverse Time Travel. It claims to be the first-ever variant of chess “with spatial, temporal and parallel dimensions”. Pieces can be moved back in time and branching timelines therefore created. That means you could find yourself checkmated last week in a move that your opponent will make yesterday.

And if that sounds complicated, it’s nothing compared to project managing the commissioning of a modern hyperscale data center campus.

Commissioning is the process of starting up and setting to work all major electrical and mechanical equipment in the data centre. This process verifies the owner has received the equipment they purchased and that it is working as specified in the design.

“It needs very thorough planning,” says Steve Krupka, a senior project manager at Commissioning Agents Inc (CAI). “When I say planning, I don’t just mean the commissioning program or process, but the whole project because the commissioning agent and/or team needs to anticipate issues and bumps as far in advance as possible, whether in the program to schedule equipment, and they need to predetermine and predefine the assessment plan.”

CAI doesn’t project manage the whole data center process from architects’ drawings to completion: acquiring building permits, digging foundations, doing some nice brickwork, landscaping the site, making sure the construction goes according to plan and so on is someone else’s job.

Rather, CAI specializes in the even more complicated, technical parts, focusing on what goes inside the building: consulting on site selection; reviewing the owner’s project requirements and the basis of design; planning the testing of each piece of equipment and how it will be sequenced, overseeing the testing of electrical and mechanical systems before and after installation; and getting the whole facility up-and-running to the satisfaction of the client, including the training of operators.

Retrofitting and upgrading part or all of an existing data center typically carries with it even greater risks as the client can’t afford to experience any unanticipated – and therefore unmitigated – downtime.

Rather than trying to work at full speed during a retrofit, as they would to get a new build up-and-running, retrofitting means working at the client’s pace with a plan that minimizes disruption. “We work very closely with the data center operator and contractor to ensure when the construction can take place, and when the testing and inspections can take place. Little things, like inspections, can become a big task taking many hours or even days,” says Krupka. “There’s a lot of intricacies associated with it.”

Process, process, process

The key to efficient project management is a thorough process, which is why CAI has a number of process and procedure handbooks, on which all data center project staff are thoroughly trained. “Currently, we have five different handbooks, depending on what country or region you’re in,” says Andrew Kochanski, a senior project manager and discipline lead at CAI.

“From the moment we receive an RFP or RFQ, we have processes to formalize responses so we are providing valuable information and clarifying how the work will be executed. Our project-controls procedure details exactly how we manage a project and sets the client’s expectations on how we will communicate from a continuous improvement standpoint, regardless of what their RFP or RFQ is requesting. Once we are awarded a project, our project-specific procedure documents exactly how each type of work will be performed,” explains Kochanski.

These detailed handbooks are typically between 60 and 80 pages long and provide the road-map for how various aspects of data center projects will be traversed and tackled, from RFP to certification. They include templates that every project manager and team member will be following. Furthermore, should a senior project manager be confounded by any unexpected issue, there’s also a formal escalation process to involve senior leader support to achieve a satisfactory resolution.

Sensible, robust and closely followed processes are at the heart of good project management, and at the heart of a good project manager is organization, subject-matter knowledge and training.

In addition to hiring long-standing experts with construction, electrical and mechanical qualifications to provide subject matter expertise, project managers at CAI have multiple tiers of training they are put through, with new hires expected to achieve basic proficiency within their first 30 days. This covers safety training, time reporting, and internal CAI guidelines, processes and procedures.

Higher level technical training is more specialized, says Krupka. “There's multiple different certifications, depending on what your career path or technical path may be, or your role and responsibility with the company.

“So we have data center, project management, building commissioning, electrical, mechanical, and other qualifications – but you need to complete a technical tier within six months of employment. Then, people that want to take the next step beyond that can achieve further qualifications in their chosen subject to become a subject-matter expert and principal,” says Krupka.

It goes without saying that the training also includes mentoring, as well as on-the-job learning under the guidance of experienced senior staff.

“I was hired as a project manager. But we have a couple of people now leading some of our smaller projects who've identified an interest in project management. They are guided and led through that process by some of our subject-matter experts in the CAI methods, helping to guide them along these less complex projects first,” says Krupka.

Training, adds Kochanski, is provided both internally and externally “and we provide support up to, and including, college reimbursement as long as it reflects your career field, or job description. So, for example, if I wanted to go back and upgrade my Bachelor's degree in electrical engineering to a Masters, then CAI would support that.

“Internally, there’s an almost endless number of training courses. There are 700 or more different courses for project managers and technicians of all backgrounds.

“Then, as you progress through your qualification tiers, and want to progress along other training programs, we have one-on-ones, too. So, I have one-on-ones with the team to find out what they want to do, where they want to progress to, and it then falls on us to make sure we work with them to make sure that they can get to both where they want to go, as well as where they and their skills will prove a good fit,” says Kochanski.

For example, Kochanski is currently working with two junior colleagues to help move them from junior project manager roles to become fully fledged project managers. “We don’t just throw them in at the deep end: we put them through qualifications, then on-the-job training and work with them every day, not only to ensure that they are successful, but that CAI and its clients are successful, too,” he says.

Overcoming supply-chain surprises

While the supply of competent project managers isn’t in doubt at CAI, the same could not be said of many products, parts, and materials at the current time due to global supply chain challenges.

Indeed, this has proven to be one of the most vexing issues over the past year.

The surges in Covid cases in China this year led to significant localized lockdowns, which has affected the production of many components and materials. And while much of the fallout from Russia’s invasion of Ukraine has yet to settle, Ukraine is the world’s biggest producer of neon, a gas used in the chip-making process. European Union sanctions against Russia have also affected exports by rail from China to Europe.

“We don’t know what the full impact that the Russia-Ukraine conflict will have yet, but we have had some projects experience brief delays, with some equipment taking longer to become available, or with some things just not available. That can put a project significantly behind schedule.

“There are some things expected a year or more ago, ordered by the client, and the supplier is ‘Oops, we don’t have yard stock now,’” says Krupka.

While CAI doesn't involve itself in the procurement process, Krupka does have some advice for those who are. “When the client, manufacturer or vendor identifies a risk of something being delayed, then you need to start looking at your risk planning and scheduling, addressing it for progress. Because with commissioning, you must startup and build a system. So you test, starting with the major equipment; then you go to the systems and components, you can kind-of build them yourself.

“Something that data center owners, commissioning agents and contractors have learned is that you have to build your building and build your schedule to facilitate the commissioning. That's the root of it all, but that does not always happen.

“So every time I arrive on a site, whether someone else has already started the project, or we're coming into it new, that's the first thing we have to do: examine the schedule to see whether it’s aligned appropriately for the commissioning process,” says Krupka.

On the client side, he adds, one of the keys is simply communication, both with the commissioning agents, as well as contractors and suppliers – especially now with heightened supply chain problems with the potential to cause major delays.

After all, it’s in no-one’s interest for the non-arrival of a key part to check – or even checkmate – a multi-million dollar data center build.

Find out more about CAI's technology, workflow management and services for data center commissioning from the CAgents.com website