Believing You Ruined Your Employer's Business
After three years at my first job, several decades ago, I confidently believed I was useful. In particular, I had developed a reputation for disentangling knotty problems. But mid-way through my 4th year, I began to worry that I was actually hurting my employer.
My company ran a heavily modified version of a proprietary system1 that we licensed from IBM. My team's job was to maintain the local modifications to this system and make new changes as needed. But this also meant that each time we upgraded the proprietary system to a new release, we had to port all our changes. Often this was no big deal, but sometimes IBM would heavily refactor code where we had local changes, forcing us to heavily refactor our own code.
And in fact that happened to a subsystem for which I was responsible: the accounting subsystem, which kept track of usage so we could bill users (most of our customers ran applications on top of this proprietary system). For one new release, IBM refactored this code, splitting a couple of modules into about a dozen. My task was to refactor our own local changes to this subsystem, many of which were originally made when I was still in elementary school!
I spent an entire weekend studying both the old code and the refactored code, carefully selecting the places in the new code where the modifications needed to be made (I did this with printouts spread out across a conference room table, making my notes in the margins, which is a much more engrossing way to study code than using a screen, especially when your screen is a 3270).
Testing went well so after a few weeks we threw the switch and upgraded the system. A couple of months went by with no complaints from users and no instability. As far as I was concerned, this was another success for my team.
At some point, a hardcopy of the quarterly financial results ended up in my work mailbox. The company wasn't public, but our VP would circulate a single copy of the results to the employees below him. I typically would pass this on to the next person without reading it, but when it showed up this time, I was reminded that I had changed the accounting subsystem. So I looked at it.
Profit was steady. That was reassuring. I looked more closely. Revenue was down, and by quite a bit. That was less reassuring. I didn't know what to think.
I stewed about this for a while, wondering if I had broken something2. I re-ran all my tests, comparing the old system to the new. The usage numbers came out roughly the same. Still, this was a weird coincidence, so I went to my manager and mentioned it to him. He said, "If it was us, I think I would have heard something". That placated me somewhat, but over the next few days I began to get anxious again. This time, I went to the director of my department. She essentially echoed my manager.
For the most part, I was able to move on. Every couple of weeks I would get a pang of anxiety about the revenue drop, but then I would get absorbed in some system crash or new project and forget about it.
Around the time the next quarterly report was due, I started to get anxious again. I was hoping for good numbers but that didn't come to pass. The next report, as before, showed that profit was fine, but revenue was continuing its downward movement.
And this was a clue that I may have missed at the time: if the accounting system was undercounting usage metrics, we would have seen a steady undercount, not a progressively worsening one. While it is possible to craft a bug like that, it isn't likely.
Whether that had occurred to me at the time or not, it wasn't enough to placate me. My recently inflated ego was pierced. Just a few months earlier, I believed myself useful to the company, but now I wondered if I was damaging it instead.
I am reminded of an episode of the BBC series Yes Minister, entitled "The Skeleton in the Cupboard", where it is revealed that Sir Humphrey Appleby, 30 years earlier as a junior civil servant, made a grievous error that resulted in a 40 million pound loss to the government. The titular minister, Jim Hacker, while pretending not to know that Humphrey was the culprit, says to him: "it's still more, a hundred times more, than this official could ever have earned in his entire career."
And I was thinking along those lines: even if I took no salary for the rest of my life, I could not make up for this revenue shortfall that I potentially brought about. I started to think I might be in trouble.
I was pretty moribund for the next 3 months. I continued to test and re-test and compare the old release to the new one. Sometimes the test results would briefly reassured me, but very quickly I would wonder: did I test everything? Is there something in the production environment that doesn't exist in test? Am I missing something?
I was becoming obsessed, like those unfortunate people who feel compelled to retrace the route they just drove to prove to themselves they hadn't run over anyone.
The third quarterly report since the upgrade was more of the same: profit steady, revenue down. Again, I brought up my concerns with my director. This time she said, "I'll check into it". Eventually the VP of our division stopped by my office and said he didn't think the revenue drop had anything to do with us, but he would talk to the finance department.
Finally, a finance director took pity on me and sat me down at the big round table in his office with documents splayed across it. Apparently our customers, which were mostly banks, were merging like crazy. I remember him showing me a list of accounts, pointing out which pairs had consolidated down to a single account. Our revenue numbers accurately reflected the restructuring of the banking industry. And our good profit numbers indicated that the company anticipated these structural changes.
My world came back into alignment. Life went back to normal. I was still somewhat humbled, having fretted so much over a phantom, but I felt pretty good.
Coincidentally, the next release of the proprietary system did actually cause a slight drop with revenue, and our finance department noticed it immediately. This time, someone else other than me was responsible for porting the accounting subsystem changes, but that person wasn't at fault. The new release included a new memory management subsystem with aggressive optimizations that skewed the memory usage stats. We had a work-around after a few weeks.
This proprietary system was Virtual Machine/System Product (or just VM/SP). My team maintained extensive modifications to the Control Program (CP), which was the hypervisor component of VM/SP. The decision to make changes to CP dates back to when the company ran an early version of this product (at that time called CP/CMS) which lacked some basic features like remote networking. While a lot of these issues were alleviated by later releases of VM/370 and VM/SP, many of the company's products came to rely on the local modifications.↩
I don't remember checking whether revenue had been on a downward trend even before my changes, but I imagine I must have.↩