Thursday, September 18 - For Immediate Release: Shape Minds, Build Brands, and Deliver Results with Game-Changing Public Relations (2015)

For Immediate Release: Shape Minds, Build Brands, and Deliver Results with Game-Changing Public Relations (2015)

Chapter 16

• Thursday, September 18

I’m at my desk, tying up some loose ends when Ellen runs up to me, holding an e-mail printout. It’s from Dick, raising the alarm with all company executives that something has gone terribly wrong with the company invoicing systems. Earlier today, one of the clerks discovered that no customers had been invoiced for three days. Among other things, this means that customers haven’t been paying on time, which means the company will have less cash in the bank at the end of the quarter than projected, which will raise all sorts of uncomfortable questions when the company earnings are announced.

It’s clear from Dick’s string of e-mails that he’s livid, and apparently, his whole accounts receivable staff and controller have been chain smoking and doing damage control at all levels.

From: Dick Landry

To: Steve Masters

Cc: Bill Palmer

Date: September 18, 3:11 PM

Priority: Highest

Subject: ACTION NEEDED: Potential $50MM cash shortfall due to IT failure

ALL CUSTOMER INVOICES ARE STILL STUCK OR MISSING IN THE SYSTEM. WE CAN’T EVEN RETRIEVE THEM TO MANUALLY SEND INVOICES BY E-MAIL!

We’re trying to figure out how we can resume normal business operations. There’s likely $50MM of receivables stuck in the system, which will be missing from our cash account at end of quarter.

Get your IT guys to fix this. The hole this blows in our quarterly numbers will be impossible to hide, and maybe even impossible to explain away.

Call me, Steve. I’ll be on the window ledge.

Dick

We’re all gathered in the NOC conference room. I’m pleased that when Patty finishes describing the incident, she quickly presented all the relevant changes for the last seventy-two hours.

After she’s done, I say firmly to the entire team, “First and foremost on my mind is the risk of losing transactions. Ladies and gentlemen, I need to be very clear about this: DO NOT TOUCH ANYTHING WITHOUT GETTING APPROVAL FROM ME. This is not an outage we’re dealing with here. We’re in a situation where we could accidentally lose order entry or accounts receivable data. This terrifies me. And that should absolutely terrify you.

“As Patty said, we need timelines and hypotheses for what might have caused the invoicing system to fail,” I say. “This is our Apollo 13 moment, and I’m Gene Kranz in Houston Mission Control. I don’t want guesswork. I want hypotheses backed up with facts. So get back to your screens, assemble timelines and data, and I want to hear your best thinking on cause and effect. Failure is not an option.”

By 6 p.m., Patty’s team has documented over twenty different potential failure causes that have been proposed. After further investigation, eight remain as likely possibilities. An owner has been assigned to look into each.

Realizing that there’s little more we can do as a group until they complete their research, we agree to reconvene at 10 p.m. tonight.

On the one hand, I’m frustrated that once again, we’ve been plunged into a crisis and our day is dominated by unplanned incident work. On the other hand, I feel a deep sense of satisfaction at the orderly nature of our incident investigation and quickly text Paige that I’ll be joining the family for dinner shortly.


“Daddy,” I hear, as I’m sitting in bed with Grant, trying to put him to sleep while keeping the thoughts of outages out of my head. “Why doesn’t Thomas the Tank Engine have a tender car? Why?”

Smiling down at him, I marvel at the questions my three-year-old son comes up with. We’re going through our nighttime ritual of reading books. I’m glad to be doing this again, which I do every night. Or did, that is, until the Phoenix recovery effort.

Most of the lights are off, but one lamp is still dimly lit. There is a pile of books on Grant’s bed, and we’re on the third one of the night.

I’m starting to get a little dry-mouthed from reading. The idea of taking a little break and doing some research on the Internet on train tender cars sounds pretty appealing.

I love how inquisitive my kids are and how much they love books, but there are nights when I’m so exhausted that I’ve actually fallen asleep during our nightly ritual. My wife will walk in, find me asleep with one of Grant’s books lying on top of my face and Grant asleep beside me.

Despite how tired I am, I’m grateful to be at home early enough to resume my nighttime ritual with my older son.

“Yes, we need to find out, Daddy,” Grant demands. I smile at him, and I dig my phone out from my pocket, intending to do a Google search for “tank engine tender car.”

But first, I quickly scan my phone for any new updates on the customer invoicing problem. I’m amazed at the difference two weeks can make.

During the last Sev 1 incident that hit our credit card processing systems, the conference call was full of finger-pointing, denials, and, most importantly, wasted time when our customers couldn’t give us money.

Afterward, we did the first of a series of ongoing blameless postmortems to figure out what really happened and come up with ideas on how to prevent it from happening again. Better yet, Patty led a series of mock incident calls with all hands on deck, to rehearse the new procedures.

It was terrific to watch. Even Wes saw the value.

I’m pleased to see all the e-mails indicating a lot of good information and effective discussion among the teams working the problem. They’ve kept the telephone conference bridge and a chat room open for people working the issue, and I plan on calling in at 10 p.m. to see how it’s going.

That’s forty-five minutes from now. Plenty of time to spend with Grant, who should be falling asleep soon.

He nudges me, obviously expecting more progress on the research front.

“Sorry, Granty. Daddy got distracted,” I say as I open up the browser. I’m surprised by how many of the search results are all about Thomas the Tank Engine. It’s the book series that spawned a multibillion dollar franchise of toy trains, clothing, videos, and coloring books. With two sons, we seem destined to own two of every item soon.

I’m reading a promising Wikipedia entry on trains when my phone starts vibrating and the screen displays “Call from Steve Masters.”

I groan and double-check my watch. It’s 9:15 p.m.

I’ve had way too many meetings and phone calls with Steve lately. In my head, I wonder how many of these meetings I can take.

On the other hand, after the Phoenix debacle, every outage and incident is trivial in comparison, right?

I say gently, “Hang on, Grant. Daddy has to take a phone call. I’ll be right back.” I jump out of his bed and walk into the dark hallway.

I’m glad I had just scanned through all the e-mail traffic on the outage just seconds before. I take a deep breath before I hit the button to answer the call.

I say, “Bill here.”

Steve’s loud voice booms in my ear. “Evening, Bill. I’m glad you’re there. Of course you know about the customer invoicing problems from Dick?”

“Yes, of course,” I reply, surprised at his tone. “My team declared a major incident early this afternoon and we’ve been working this issue ever since. I’ve been sending out status reports every hour. Dick and I spent twenty minutes on the phone earlier this evening. I know the problem is serious, and my team is following the process we’ve created after the payroll failure. I’m completely satisfied that the process is working.”

“Well I just got off the phone with Dick, and he tells me that you’re dragging your feet,” says Steve, clearly very angry. “Obviously, I’m not calling you at night because I want to chitchat. Do you understand how intolerable this is? Yet another IT screwup jeopardizing everything. Cash is the lifeblood of the company, and if we can’t invoice customers, we can’t get paid!”

Falling back on old training on handling frustrated people, I calmly reiterate what I already stated. “As I said, I talked with Dick earlier today. He very much impressed upon me all the implications. We’ve activated our new incident process, and we’re methodically looking into what could have caused the failure. They’re doing exactly what I want them to, because with so many moving pieces, it’s way too easy to make things worse by jumping to conclusions—”

“Are you in the office?” Steve demands, cutting me off before I could finish.

His question genuinely catches me off guard.

“Uh… No, I’m at home,” I answer.

Is he worried that I’ve delegated the problem away? To reinforce my role in handling the crisis and what my expectations from my team are, I say, “I will be calling into the war bridge line at ten o’ clock. As always, we have a duty officer on site, and those on my staff who need to be in the office are there already.”

Finally, I ask bluntly, “Steve, want to tell me what’s on your mind? I’m on top of this situation. What do you need that you aren’t getting right now?”

He responds hotly, “What I need from you is some sense of urgency. Dick and his team are burning the midnight oil trying to figure out how our quarter will end up in six working days. But I think I already know what the answer will be.”

He continues, “We’ll probably miss almost every target that we’ve promised the board: revenue, cash, receivables—everything. In fact, every measure we’ve promised the board is going the wrong way! This screwup may confirm the board’s suspicion that we’ve completely lost control of managing this company!”

Steve is almost snarling now as he says, “So, what I want from you, Bill, is to stay sufficiently on top of things, so that I don’t have my CFO saying that you’re dragging your feet. The house is burning down, and all I hear from you is about drawing pictures and timelines. What in the hell is wrong with you? You afraid to get people out of bed?”

I start again, “Steve, if I thought it would help, I’d have everyone pull all-nighters in the data center tonight. For Phoenix, some people didn’t go home for nearly a week. Trust me, I know the house is on fire, but right now, more than anything, we need situational awareness. Before we send the teams crashing through the front door with fire hoses, we have to have someone at least quickly walk the perimeter of the yard—otherwise, we’ll end up burning down the houses next door!”

I realize that I’ve raised my voice in the relative quiet of our house as we’re trying to get the kids to sleep. I resume, more quietly, “And just in case you forgot, during the payroll outage, we made the outage worse by our own actions. We probably could have completed the payroll run during the business day if someone hadn’t started screwing with the SAN. Because of that, we added another six hours to the outage, and we nearly lost payroll data!”

My hopes that the calm voice of reason is reaching him are dashed when I hear him say, “Oh, yeah? I don’t think your team agrees with you. What was the name of that smart guy who you introduced me to? Bob? No, Brent. I talked with Brent earlier today, and he’s very skeptical of your approach. He thinks what you’re doing is separating people who actually do the work from what needs to get done. What is Brent doing right now?”

Shit.

I like transparency. I always try to make my team totally accessible to my boss and the business. But there’s always risk in doing this.

Like having Brent spout off his crazy theories to the CEO.

“I hope Brent is at home, because that’s exactly where he should be,” I respond. “Until we know for sure exactly what went wrong, that’s where I want him. Look, it’s rocket scientists like him that often cause the problem in the first place. Every time we escalate to Brent, we perpetuate our reliance on him, and make it that much less likely we can fix things without him!”

Suspecting that I may be losing Steve, I start again. “The chaotic way we currently work, Brent is having to fix the punctured hulls almost every day. I’m pretty sure, though, that Brent is one of the main reasons the hull is punctured in the first place. It’s not malicious, of course, but it’s just a side effect of the way we work and fix outages here.”

There is a pause. Then he says slowly and decisively, “I’m glad you’re being so professorial about this, but we’ve got a wildfire that’s out of control. Up until now, we’ve done it your way. And now we’re going to do it my way.

“I want you to call Brent in, and I want him to roll up his sleeves and help fix this outage. And not just Brent. I want all eyeballs on screens and all hands on keyboards. I’m Captain Kirk. You’re Scotty. And I need warp speed, so get your lazy engineers off their asses! Do you understand me?”

Steve is yelling so loudly by now that I’m holding the phone away from my ear.

Suddenly, I’m furious. Steve is going to screw this up again.

Recalling my days in the Marines, I finally say, “Permission to speak freely, sir?”

I hear Steve on the other end of the line snort dismissively in response. “Yes, dammit.”

“You think I’m being overly cautious, and that I’m hesitating to do what needs to be done. But you are wrong. Dead wrong,” I say adamantly. “If you do what I think you’re suggesting, which is basically ‘all hands on deck,’ I predict that we’re going to make things much worse.”

I continue, “I tried to advise you of something very similar before the Phoenix launch. Up until now, we have not been sufficiently disciplined in how we work outages. Given all the complexity and moving pieces, there’s too much likelihood of causing another problem. I may not know exactly what caused the customer invoicing issue, but I know enough to absolutely conclude that what you’re proposing is a very bad idea. I recommend continuing along the lines I am currently prosecuting.”

I hold my breath, waiting to hear how he reacts.

He says slowly, “I’m sorry you feel that way, Bill, but the drawers open on my side of the desk. I’m telling you that it’s now DEFCON 1, so go get the smartest people working on this problem. And I want status updates on this IT failure every two hours until it’s fixed. Understood?”

Before I can think about what to say, I find myself saying, “I don’t know why you need me to do that. You’re talking directly to my people, and you’re calling all shots on the ground. Do it yourself. I can’t be held responsible for the results of this FUBAR situation.”

And before I hang up on him, I say with finality, “And expect my resignation in the morning.”

I wipe the sweat off my forehead, and look up from my phone to see my wife Paige staring at me wide-eyed.

“Are you insane? You just quit? Just like that? How are we going to pay the bills now?” she asks, her voice rising.

I turn the ringer off on my phone and put it back in my pocket, saying, “Honey, I’m not sure how much of that conversation you heard, but let me explain…”