I once worked at a place that had an interesting little problem stemming
from the way they managed machines and some of the fun side-effects
processes can have on each other. This is back in the pre-systemd days
of running services, such that things tended to be started or restarted
by someone or something that itself ran as root. That is, if you
started something, it might inherit one or more things from your
environment. Likewise, if you ran something with an unusual
environment, it might bestow those “gifts” onto anything it happened to
touch.
This story involves a lot of weird little things going back a ways.
Anyone who used to run OpenSSH before it was a stock part of their Linux
distribution probably ran into the problem where you’d apply an upgrade
to glibc and then OpenSSH would stop letting people log in. This
ultimately turned out to be a mismatch between the dynamically linked
bits of glibc in the running sshd and the NSS stuff for authentication
on the disk that it would grab on the fly.
After locking themselves out of machines too many times, people
generally learned that after you mess with a glibc upgrade, you restart
the sshd listener on port 22. Of course, they also had to learn to not
whack EVERY sshd at the same time, lest they kneecap the very thing
that was letting them on the box, but that