Glitchy background image of campus with alert icon

Fictional scenario, real results: Tabletop exercise helps IT community plan for crises

Picture it: Madison, Wisconsin. October 2024…

It’s a crisp fall morning on campus—leaves crunching underfoot as students make their way to their first classes, and faculty and staff begin their workdays.

But as campus begins humming to life on this particular morning, the chill in the air turns even colder: Support calls begin pouring in to Help Desks across the university, as scores of people report being unable to boot up their computers and laptops.

Members of UW–‍Madison's IT community gather at Union South to engage in tabletop exercise
144 members of the university IT community gathered Jan 8 at Union South to engage in a tabletop exercise, simulating a widespread technology outage—and planning for how the university would collectively respond

IT professionals across campus are already on high alert, having woken up to 6am alarms signaling that servers and systems aren’t responding. The situation is unclear and difficult to diagnose. Is it the network? A firewall issue? A cyber attack?

As the morning coffee flows, “breaking news” reports point to a global IT shutdown, with millions of computers taken offline.

By 7am, a “CobraStrike” software bug emerges as the culprit. The cybersecurity software provides protection to all UW–‍Madison Microsoft Windows-based computers, laptops, servers and virtual servers. So the critical bug is causing all university-managed Windows machines to crash upon reboot—rendering thousands of computers and devices used by faculty, staff and students useless.

Most research labs are offline. Computer labs are down. As classes meet, only Mac computers can connect to the internet. Need a breakfast bagel at the Union? Sorry—no one can pay for anything because campus payment systems aren’t functioning.

BUT WAIT! There’s more!

It’s the week of the big EDUCAUSE higher education technology conference in San Antonio—and a large contingent of UW–‍Madison IT leaders are en route to Texas. In the air, the pilot announces that “some kind of IT outage” is also affecting the airline’s computer systems.

Sure enough, when the university IT leaders arrive at the Atlanta airport, it’s mayhem. Their cell phones are blowing up with outage notices and SOS calls from deans and IT staff back in Madison… but the WiFi network is overwhelmed at the airport. No one can use their phones to connect with their teams back on campus.

What would you do?

It’s a fictional scenario—but certainly a realistic one. (Remember the CrowdStrike outage last summer?)

And it was the subject of a lively “tabletop exercise” last week at Union South’s Varsity Hall. 144 information and technology professionals across campus gathered together to simulate this emergency scenario—and brainstorm how UW–‍Madison might collectively respond.

What’s a tabletop exercise?

In a tabletop exercise like this one, key personnel in the university’s IT community who are assigned to emergency management roles gather to discuss simulated emergency situations—in a non-threatening environment in which all ideas are welcome. Ground rules include being willing to challenge yourself and engage in the conversation—and it’s OK to not have an answer.

And yet, during exercises like these, creative answers often emerge. So, too, do creative “plot twists.”

Normally during annual tabletop exercises, each table is assigned a leader—and often that leader is a manager or director of an IT team on campus. By introducing the element of key leaders being away from campus for the EDUCAUSE conference, key players were effectively out of commission for the tabletop exercise.

To simulate their absence, table leaders were sequestered in a separate room down the hall at Union South, otherwise known as the exercise’s “Atlanta airport.” To remain true to the scenario, the leaders’ group couldn’t use their phones or laptops to communicate with their teams “back on campus,” gathered in Varsity Hall for the drill.

This plot twist opened up opportunities for other participants in Varsity Hall to emerge as leaders, brainstorming creative ways to handle the crisis and answer critical questions.

Creative plot twists, creative solutions

Just one example: When a new detail in the scenario called for a hands-on fix for each affected Windows machine in the outage, tabletop participants had to think through how the university IT community would mobilize to carry out the remediation. From an in-person drive-through support approach to a do-it-yourself webpage with videos to walk far-flung remote employees through the fix, participants developed a variety of responses.

Recognizing that communication is vital during a crisis, they also devised plans to keep faculty, staff, students and the IT community informed through the outage. (When a real outage strikes: How to stay informed during IT outages.)

Meanwhile in the “airport” room, the leaders’ group grappled with limited information to figure out how they could work together to lead and contribute from afar. As the drill progressed, the leaders’ group rolled large foam dice to select 4 people whose phones eventually started working, as the WiFi at the simulated airport began to open up.

“Gathering information was a challenge, so this required a lot of coordination of communication resources,” said one of the sequestered leaders, Division of Information (DoIT) Communications Manager Lauren Bruce. “We were pleased to find that our teams were largely enabled to address the outage situation, despite our absence.”

“It speaks to the readiness of DoIT and the university’s IT community in a time of crises,” Bruce reflected.

UW–‍Madison Libraries Technology Manager Jay Ray agreed, describing the tabletop exercise as “a meeting of minds from across campus.” Ray said the exercise brought to light areas where campus colleagues can learn from each other about how we respond when the unexpected occurs—and what each team’s priorities are.

Asking tough questions & identifying gaps

Sometimes these priorities are shared, and sometimes individual groups across the university might have their own priorities. One example: During the exercise, the UW–‍Madison Libraries team focused on the role of the university’s libraries as community and gathering spaces that are open to everyone. So how can we ensure these spaces remain open and available for the entire campus community in a crisis, regardless of what’s happening with technology outages?

At the same time, the Libraries team challenged divisional peers on campus to think about our collective responsibility to students during a technology crisis. While the tabletop scenario primarily affected faculty and staff who use university-managed Windows devices, we need to also remain focused on how students are impacted, in their classrooms and beyond, Ray noted.

“We asked tough questions about how we make sure that students are kept aware of an ongoing situation, regardless of the assumptions of whether the problem pertains to them,” Ray said.

Several teams in the tabletop exercise also noted things they can do post-exercise to be better prepared for emergencies—focusing on continuity of operations, gaps in current approaches, communication and problem assessment.

In future exercises, Ray hopes to see continued momentum from this well-facilitated exercise, challenging participants to work more across reporting lines to negotiate opposing viewpoints and differing priorities.

“I’m excited for the next round,” Ray added.

Tabletop exercise planners are already looking ahead to future simulation trainings, said UW–‍Madison Chief Technology Officer Todd Shechter. He noted that in addition to walking through “what if” scenarios and refining our practices, the exercises are also a great way to strengthen connections with colleagues across campus.

“My thanks to everyone who planned, facilitated and participated in the tabletop exercise, and came together as our UW–‍Madison information and technology community,” Shechter said. “We do these simulations not only to become better planners for real situations, but also to build relationships with each other. To meet people we may have only worked with virtually, and connect as a community.”

Added Shechter: “In real-world situations, we rely on and lean into these relationships.”

Special thanks

Special thanks to:

  • Tabletop exercise facilitator Reed Erickson, UW Police Department emergency management specialist, as well as Ed Lawson and Mclayne Vaughn from UWPD, who helped facilitate
  • The planning team: UW–‍Madison Chief Technology Officer Todd Shechter, School of Education IT Director Lisa Jansen and DoIT Business Continuity Manager Jennifer Sutherland
  • DoIT’s Becky Morgan, who led event logistics
  • Academic Technology Director Tamara Walker, who helped facilitate tabletop participant conversation