The familiar blue and gold intro graphic fills the screen every evening at six o’clock on the dot. The jabbing staccato string music conjures up vague secondhand memories of what a teletype machine might have sounded like. A high angle view of the studio floor with the large Lexan-clad desk in the middle, then a cross dissolve to a two shot of the presenters for this newscast. The music fades, each person introduces themselves, then they jump straight into the top story for the evening. It’s been this way for as long as anybody can remember. They’ve never failed to get this show on the air.

They’ve never failed.

Who you gonna call?

Everything fails all the time.

Producing any sort of live television show is a complex ballet. The studio’s cameras and microphones route their signals into video switchers and audio mixers, pre-taped packages come from the video server, field reporters are connected bidirectionally through a satellite link, and with a sprinkling of pizazz from the motion graphics machine, the final product is sent off to master control and ultimately to all the kitchen counters and family rooms across the city.

But there are ancillary systems outside of this direct pipeline. The studio lighting is quite important, as most professional broadcast cameras tend to produce underwhelming images under inadequate light. The teleprompters feeding the anchors their scripts are obviously important. The weather reporting segments use an entirely separate system of graphics rendering equipment that must be linked through a chroma keyer to place the meteorologist in front of the computer-generated forecast images. And quite obviously, this equipment requires a handful of human operators.

Studio-grade equipment is obscenely expensive, but it is also incredibly reliable. It is rare for things to outright fail, but anything can eventually wear out after enough daily use. If a camera fails, perhaps they can wheel the one from the sports desk over to cover this part of the broadcast. If the teleprompters fail, the anchors have a copy of the script at their desk that they can look down at. If one of the anchors calls out sick, they can sub in talent from the morning news team.

Each of these is an example of either a redundant backup system or spare capacity that can be reallocated if needed. The broadcast technically does not need any of these contingencies to function under normal circumstances, but in cases where things go wrong it can mean the difference between success and total failure.

Not everything can be made completely redundant. A failure in the power system for the lights will most likely plunge the entire studio into darkness, and that’s no way to run a news program. Similarly, if the $50,000 video switcher dies, it’s highly unlikely that they’ll have a spare holed up in the supply closet. To insure against every possible thing that could ever go wrong, they would have to build a second studio on a separate part of the city’s electric grid, with redundant copies of all the equipment and broadcast content, along with a full crew of understudies ready to take over at a moment’s notice. This is a degree of redundancy that can’t reasonably be achieved by any budget-conscious station.

There is a hybrid between the two options, allowing the station to only maintain a single instance of anything expensive while having some assurance that the equipment they do have will work when needed: They can find an expert of some sort who is capable of fixing anything that breaks well enough to get the broadcast out. We’ll name this person Alex. If the microphone battery dies, Alex will swap it out. If the video server acts up, Alex knows how to get it working again. If the tire pressure light in the Chevrolet Weather Beast comes on, or the studio’s air conditioning fails, or the technical director breaks both their hands and needs somebody to push the buttons on their behalf, it’s Alex’s time to shine.

Now, naturally, most of the time everything is going fine and Alex has nothing to do. So Alex has some other regular job in the studio—say running the audio mixer. In fact, the audio mixer thing is their official job title and their primary responsibility at the station; they only jump into universal-problem-solving mode when something goes wrong. As soon as the problem is resolved, it’s back to the audio mixer.

The other thing about all this is, well, it’s very difficult to find and train people like Alex. So since they are at the station all evening anyway, why not also have them stick around in case anything goes wrong during the 7:00 news, and 11:00? And if anything happens during the 4:30–7 a.m. news, the station can call Alex at home and have them bop over and fix the problem. Oh, and also the news at noon, and the 4 p.m. block. Apparently this station broadcasts six hours of live news programming most days. At least it’s only four hours on Sunday. In the station’s view, there is no need for anybody to relieve Alex because—most of the time—they never need Alex’s emergency response skills at all. There should be no need to hire and train somebody else to do this stuff because they barely use the services of the person they already have.

There is, of course, another option that the station has never seriously entertained: Don’t hold Alex to any of those responsibilities at all, and if things really go to hell they can just throw on an old The Price Is Right rerun and hope for better luck during the next scheduled newscast.

Grandpa, what’s a beeper?

1-800-759-7243
But if you ain’t got that pin number, dummy, you can’t call me
To hook up with Mix you gotta call that number
Then sit by the phone and wonder
Will he call? If you’re fine I might
If you’re a duck, good night

There was a time—not that long ago, really—where people couldn’t contact you if they didn’t know where you were. Telephones were literally screwed into the walls of houses and businesses. Portable two way radios existed, but they were a massive pain to carry around and operate. If somebody wished to contact you, they would not call you specifically but rather your house or your workplace, places where you might or might not have been at the time. If you were not there, maybe they’d try to call your brother’s house, your favorite bar, the Kiwanis club, or another location that was significant to you. If they still couldn’t find you, eventually they’d give up. People used to be more chill in that way.

In a more structured environment—say a hospital where doctors moved from room to room but stayed inside one building—it was important to be able to get in touch with a specific person without knowing which room they were in. To accomplish this, a phone operator would page This verb form of the word “page” uses the same sense as the noun “page,” an old-timey word meaning roughly “servant boy.” I page you in the sense that I am asking Kenneth, the NBC page from 30 Rock to send for you. the desired person via an announcement over the building’s public address speakers: “Paging Dr. Johnson, Dr. Johnson, please call fourth floor nurse station.” Assuming Dr. Johnson was in the building to hear this, they would find a phone and call the station as instructed.

This worked fine, but it generated a lot of “useless” noise because most of the staff were uninvolved in most of the pages they overheard. Thanks to incremental improvements in technology, the voice announcements were phased out to make way for unidirectional radio broadcasts that covered the entire building. The content of the radio message remained the same as the audible announcement: who the page was for, and who that person needed to contact in response. Each person who needed to receive pages was given a pager, a radio receiver that was pre-programmed to only activate in response to pages specifically addressed to it. Each pager contained a small numeric display where the information about who to call could be shown. These were colloquially called beepers because, well, they made a beeping sound to announce each incoming page.

To send a page, a person would pick up one of the building’s telephones and dial the number for the paging system. They would be prompted to enter the recipient’s PIN or unique identification code along with a callback number. If the sender wanted the recipient to call them directly, the callback number would be a phone that the sender was ready to pick up. It didn’t have to be, though. For example, the sender and recipient could have a prearranged system in which a code like “505” could be interpreted as the distress signal SOS with some mutually understood meaning. These codes were more common from senders that the recipient knew well, representing messages they frequently needed to exchange. To a building maintenance worker, “234” could indicate an emergency at 234 Maple Avenue while “5300” could have been 5300 Elm Street. The codes meant what the sender and recipient agreed they meant.

Technology got better. Things got smaller and faster. The unidirectional pager networks started becoming overshadowed by mobile phone networks which soon gained the ability to send bidirectional SMS messages. Microprocessors advanced to the point where a battery-operated handheld device could serve as a phone that could also send and receive text messages. These advances made it possible to send longer messages using a more expressive character set on a device that also did other things. My very first mobile phone could run a game of Snake that objectively blew. But the capabilities were there. Phones continued to gain capabilities, the networks they ran on continued to get faster with wider coverage, but the central thread of “I need to get this message to that device” is as clear today as it was when Sir Mix-A-Lot was courting his lady friends in the 1980s.

Also, the systems described up to this point had one thing in common: The person sending the page was a human being.

Getting on the same page

Dude: They gave Dude a beeper, so whenever these guys call—

Walter: What if it’s during a game?

Dude: Oh, I told them if it was during league play…

Donny: What’s during league play?

Walter: Life does not stop and start at your convenience, you miserable piece of shit.

Like a disheartening number of things in the tech industry, there are no real standards around what on-call responsibilities look like. Each organization And each team within! is free to set things up in whichever way suits their tastes, and the resulting practices vary widely as a result. In order to ground this article in something concrete, I will describe Alex’s on-call arrangement, which seems to be typical for US companies whose business model is “Have a website and/or mobile app, and either put ads all over it or convince the users to enter their credit card information somewhere to use it.” The prevailing attitude of these organizations is that the product must work at all times, otherwise it results in failure to show an ad or collect a payment. Both of these negatively affect revenue.

Alex’s company uses the SEV system, which might Again, no standards. Somebody copied part of the philosophy from Amazon or Facebook or someplace but never bothered to codify exactly what the abbreviation meant to them. mean “severity,” “site event,” “significant event,” “serious event,” or anything else you might care to contrive that matches the pattern. SEVs are further divided into numbered classes depending on their impact on the product experience; a SEV 1 means that the business is currently failing to be a business because it is unable to perform its core functions and/or collect its revenue. The lesser SEV 3 might represent degraded performance on some non-critical portion of the application. An example of a SEV 3 might be a situation where users can still change their profile pictures, but those changes are not showing up promptly in the app due to some kind of processing delay. This will probably not impact the quarterly financial statement in a measurable way. An instance of a SEV 1, on the other hand, might entail the mobile app showing a perpetual loading spinner on every request to every user at once. That type of thing tends to get noticed.

Below the SEV system, there is a bubbling churn of things that are subtly broken, or are well on the way to someday being definitely broken, but are fine for the time being. A good example of this would be a disk that is 98% full. In its current state, nothing is actually wrong. But once it finally becomes 100% full and cannot accept any more data, something else in the system is going to respond poorly and this can likely cascade into some kind of SEV. Most systems in most organizations have monitoring in place for this sort of thing, and it is common for an on-call engineer to receive pages due to (e.g.) high disk usage to investigate specifically to avoid a potential SEV in the future. Practically all pages of this nature are generated and sent through automated means, and these pages can sometimes resolve themselves without outside intervention if (e.g.) the disk usage abates naturally.

The on-call engineer in Alex’s department is selected out of a rotation of all the team members. The on-call shift is seven consecutive days of 24-hour support, or 168 solid hours. ±1 hour depending on how daylight saving time shakes out. The on-call engineer does not need to stay awake for seven straight days; the idea is that they’re supposed to work on typical tasks during business hours and go about their non-work lives as usual, but be able to jump into handling an issue quickly after receiving any page at any time. The “quickly” part is formally defined as time to acknowledge, and durations from 5 to 30 minutes are fairly typical. Alex’s team expects pages to be acknowledged within 15 minutes.

If a page is not acknowledged by the on-call engineer, a system of escalation begins. The escalation policy usually follows one of these patterns:

• If there is only a single on-call engineer, the page may escalate to them again. This re-raises the original alert in case it was somehow missed the first time.
• In a “primary/secondary” type of arrangement, there are actually two people on-call at any given moment. All pages go to the primary, and only unacknowledged pages escalate to the secondary. If the secondary doesn’t acknowledge the page either, it may escalate further as described by the other bullet points here.
• In a “hunt group” configuration, an unacknowledged page is sent to every member of the team—none of whom are officially on-call at the moment—in the hopes that one of them is free to acknowledge and handle the issue. This arrangement has a strong tendency to break down into one of two degenerate states:
  1. One or a few people naturally become highly responsive to all pages, acknowledging them before most of their teammates have the opportunity to do so. Over time, most of the team members stop paying attention to pages and leave their highly-responsive peers to handle everything that comes in.
  2. Something very close to the bystander effect occurs, where everybody in the group assumes somebody else