Techie ended blame game by treating managers like toddlers • The Register

On Call Welcome once again to On Call, The Register‘s Friday column in which we share your tech support stories.

This week, meet a reader we’ll Regomize as “Warren,” a network engineer who found himself working on a project team that was trying to clean up a big stack of open tickets that had accumulated during the merger of two hospitals.

Some of the tickets were easy to fix – users would tire of waiting and just buy a new mouse to replace malfunctioning machines – but the one that landed on Warren’s desk was more challenging, technically and politically.

“They ‘airdropped’ me into the middle of an ongoing discussion between the hospital and a vendor regarding the sub-par performance of a 3D scanner used to detect breast cancer,” he told On Call.

“When doctors viewed scans, they would sometimes appear in two seconds, and sometimes appear in 45 minutes,” Warren wrote. When times ballooned to the larger figure, it caused all sorts of nasty scheduling problems as appointments ran past their allotted time.

Despite that dire situation, the argument between the vendor of the storage system that held the scans and the hospital dragged on for months.

Warren started his attempt to sort this out by researching the network connecting the scanner, the storage device in one of the hospital’s datacenters, and the medical suites where doctors tried to view the scans. He found robust 250 Mbps links and a two-hop connection across a small city.

He then joined a conference call to discuss the matter and found it depressingly familiar.

“The storage vendor blamed the client’s network, and the client blamed the vendor’s equipment,” he told On Call.

Warren next went looking for network errors by analyzing logs for each device involved in carrying packets from the datacenter to doctors. He found nothing.

The only thing he could think of was asking staff to report the odd delays as soon as they appeared.

Which is why his phone rang the next day and Warren sprang into action, logged onto the router closest to the scanner, and found… nothing. But when he looked at the next device in the chain – a switch – its logs listed line after line of errors.

“It was spitting out a lot of garbage,” Warren told On Call.

He asked around to find out who ran the switch and learned that the storage vendor managed the machine.

Warren knew this finding meant he would need to make another conference call, so he created a PowerPoint presentation that explained the network topology and displayed the relevant bit of the error logs to prove the source of the problem.

“I realized the managers were the sort of people you need to talk to as if they were toddlers to explain things,” he told On Call.

That approach worked. Warren’s bosses did the sort of things people do when they are impressed by clever people presenting irrefutable evidence. Representatives of the storage vendor could not deny culpability and remained speechless.

Warren decided to take charge and asked the storage vendor if they had ever tried the most obvious fix – replacing the network card on their array.

A week later, with a new NIC in place, the problem disappeared. The old card was clearly faulty, and, for six months, nobody thought of checking it despite the serious problems it created for patient care and hospital operations.

“That was probably one of my proudest moments in IT,” Warren told On Call.

Has PowerPoint helped you to, er, make a point? If so, click here to send On Call a slide email so we can transition to your tale on a future Friday. ®