← All writing

I built a pager for my own developers

· 6 min read · by Rahul Dhileep Kumar

An expired SSL certificate took us four hours to find. Later, I built a Pager System for Track My Academy developers. Why small teams need alerting, and what I think a good internal pager should do.

A small circuit board with a Motorola chip, held between finger and thumb

An earlier post ended with a promise. Sort of.

In the night I broke production, an expired SSL certificate on our backend domain failed silently. It took us about four hours to find it. I stayed in the office all night.

At the end I mentioned, almost in passing, that I later built a "Pager System" for Track My Academy developers. An internal, real-time error and notification system. The idea: problems should find developers, not wait to be found.

This post is about that idea. Why a tiny team needs a pager at all. And what I think a good one should and shouldn't do.

A very old idea

Pagers were not invented for developers. They were invented for doctors.

According to Wikipedia's history of the pager, Al Gross patented the first telephone pager system in 1949, intended for physicians. The physicians, apparently, did not love the idea of being permanently on call. I relate.

A year later, a paging service for doctors launched around New York City. Doctors paid US$12 a month for a pager that worked within about 40 km of one transmitter.

And pagers never quite died. Wikipedia notes that in 2017 the UK's NHS was thought to use over 10% of the world's remaining pagers. Volunteer firefighters and rural first responders still carry them.

The point of a pager has never changed. Something important is happening. The right person needs to know. Now, not when they next check.

A small circuit board with a Motorola chip in the middle, held between two fingers

Photo: Winston Chen on Unsplash

Why a small team needs one

Big companies have on-call rotations, incident commanders and very serious dashboards. Small teams have a group chat and hope.

Hope is not a monitoring strategy. I found that out the hard way, overnight.

Track My Academy runs on React, FastAPI, MongoDB, Supabase, AWS EC2, Vercel, Razorpay, Firebase and Redis. That's a lot of moving parts for a small team. Any one of them can have a bad day. And real academies run on it, so their bad day becomes our bad day.

The awkward truth is that a small team can't afford to watch everything. Nobody is sitting in front of a dashboard all day. We're shipping features, fixing bugs and, in my case, occasionally attending class.

So the system has to do the watching. And when something breaks, it has to come and find a human.

The rules I'd steal from Google

Google's SRE book has a chapter on monitoring distributed systems. I keep going back to it.

Three lines from it should be on every developer's wall:

If a human gets woken up and the fix is a script, automate the script. If the alert is something they'll ignore, delete the alert.

The same chapter splits monitoring into two questions: "what's broken, and why?" What's broken is the symptom. Why is the cause.

That distinction would have saved us hours. We had a symptom: everything was down. We jumped straight to a cause we'd picked ourselves: our new code. The real cause was a certificate nobody was watching.

It also lists four golden signals: latency, traffic, errors and saturation. If you can only measure four things, the book says, measure those.

The danger: a pager that cries wolf

There's a catch with alerting. Too much of it is almost as bad as none.

The SRE book says that "when pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts". And one line I really like: "I can only react with a sense of urgency a few times a day before I become fatigued."

Hospitals know this problem well. The US agency AHRQ has a primer on alert fatigue. It describes one study where ICU monitors generated "more than 2 million alerts in one month", or "187 warnings per patient per day". It also says clinicians "override the vast majority" of computerised ordering warnings.

Doctors are not careless. They're human, and humans tune out noise. Developers are the same, only with worse posture.

What I think a good internal pager should do

This is opinion, not a spec. But if you're a small team building your own, here's what I'd aim for.

  1. Page on symptoms users feel. Errors, slow responses, failed payments. Not every log line.
  2. Say what's broken in one sentence. The person reading it may have just woken up.
  3. Go to the right person. Not the whole team. One owner per alert.
  4. Stay quiet when things are fine. Every alert that isn't urgent teaches people to ignore the urgent ones.
  5. Watch the boring stuff. Certificates, domains, disk space, expiring keys.
  6. Get pruned regularly. If an alert fired and nobody acted, change it or remove it.

You don't have to build it from scratch either. Tools like Sentry let you send notifications or create tickets when issues match rules you set. Its docs say the goal is alerts that are "valuable and not too noisy". They suggest using filters so you only hear about issues your team needs to know about.

About those certificates

The boring stuff deserves its own section, because it got us.

Certificates are about to expire a lot more often. In 2025 the CA/Browser Forum passed a ballot to cut the maximum TLS certificate lifetime from 398 days to 47 days. The reductions start in March 2026 and finish in March 2029.

Meanwhile, Let's Encrypt stopped sending expiry reminder emails on 4 June 2025. Its reasons included that more people now have reliable renewal automation. It pointed people to third-party monitoring instead.

So: shorter lifetimes, fewer reminders. If your renewal automation ever quietly fails, nobody is going to email you about it. Your users will tell you instead.

Automate renewals. Then monitor the automation anyway.

The point

A pager isn't about being available all the time. Honestly, I'd like to sleep more, not less.

It's about not spending four hours blaming the wrong thing. The problem should arrive with a name attached. Then the human can do the part that needs a human.

FAQ

Does a small team really need alerting?

I think so. Small teams have fewer people watching, so the system has to do the watching for them.

What should trigger a page?

Something urgent, actionable and user-facing. If nobody needs to act right now, it's a ticket or a report, not a page.

How do I avoid alert fatigue?

Fewer, better alerts. Review the ones that fired, and delete or fix any that nobody acted on.

Sources

What next

The full outage story is in the night I broke production. For where the late nights happen, see three monitors, one Monster can.

Running an academy? Track My Academy handles attendance, fees, player reports and parent messages.