Every ops team has some manual procedures that they haven’t gotten around to automating yet. Toil can never be totally eliminated.
Very often, the biggest toil center for a team at a growing company will be its procedure for modifying infrastructure or its procedure for provisioning user accounts. Partial instructions for the latter might look like this:
- Create an SSH key pair for the user.
- Commit the public key to Git and push to master.
- Wait for the build job to finish.
- Find the user’s email address in the employee directory.
- Send the user their private key via 1Password.
This is a relatively short example. Sometimes there are 20 steps in the process. Sometimes there are branches and special cases to keep track of as you go. Over time, these procedures can become unmanageably large and complex.
Procedures like this are frustrating because they’re focus-intensive yet require very little thought. They demand our full attention, but our attention isn’t rewarded with interesting problems or satisfying solutions – just another checkbox checked. I have a word for a procedure like this: a slog.
We know that this procedure is ripe for automation. We can easily see how to automate any given step. And we know that a computer could carry out the instructions with far greater speed and accuracy than we can, and with l
42 Comments
IanCal
I'm a big fan of this approach.
In general it's another way of defining interfaces around processes. Those processes may be manually done or automated. But the interface can remain the same – this is quite powerful when it comes to automating steps.
It's the same as you'd do with another system really.
I've used this before with Google sheets being manually filled in -> automated with a script. And with Jira tickets being raised and then automatically picked up and processed. It lets you get started sooner, automated the most annoying parts and you never have to fully do it.
Side benefit of do nothing scripts is that they can remain up to date more often as they can be more likely to be actually used than reading the docs.
MattSayar
I was looking for this article recently since it inspired my most recent script to POSSE to various websites [0]. Thanks for posting it again!
[0] https://github.com/MattSayar/post_to_socials
qwertox
It's great, but it can't be interrupted.
It would also be nice if it would show you all the steps beforehand, and then check each item as you progress. Sometimes it's good to actually prepare from a broader perspective.
And it could log into a file as a summary.
So much that could be improved, which is why the simplest solution might be the best.
mettamage
I should just do this for waking up so that I won't be confused on my morning routine. In a sense, it's a rudimentary slide show
BLanen
[flagged]
captnswing
https://www.pyinvoke.org/
notarobot123
> It lowers the activation energy for automating tasks
Does that mean the "do-nothing script" should eventually have some automated steps that do-something?
As a placeholder for future possible automation, this feels like the right balance between automation and efficiency. It allows you to make the first stab without investing too much and it leaves some low-hanging fruit for another time when the effort might be more obviously worthwhile. Thanks for sharing!
tetha
Yeah, I started to incorporate this into my own work.
Like, sometimes you can automate 99% of a procedure, except like putting a secret into an so-far not automated secret management solution, or running a terraform apply that requires access to something that's not accessible from the automation just yet.
Instead of giving up there, I now just go ahead and insert e.g. an `ansible.builtin.pause` with clear instructions to the user what to do, and when to continue the play. This might not be gloriously fully automated (yet), but it is so much better than having to do the other 12 things manually.
Similar, we have "standard plays", which are just shell scripts to invoke ansible in some specific way. But those have started to accrue some user guidance and some do-nothing instruction steps as well. The database creation script asks you if you've added the database in the three other places upon startup, since it'd fail otherwise. Or a disk resize standard play asks you if you need to resize the disk on a failover cluster as well now?
I would like to have these things fully automated, but having these simple scripts as guidance for admins is surprisingly valuable.
linsomniac
The author needs to watch my buddy Jack's talk: Stop Writing Classes: https://youtu.be/o9pEzgHorH0?si=FgZqFGQNQUU2iREQ
linsomniac
A lot has changed since 2019. That "do nothing" script can be fed to an LLM and it's going to be a pretty good starting point for automating it. For example, feeding the script to ChatGPT o3-mini-high produced this for one of the steps:
niemandhier
The biggest hold up in automation for me usually are things that need to be done via a gui.
I tried automatizing those with some tools intended for gui testing, but those solutions tend to be brittle and I just don’t feel well if I cannot get a real success report.
I’d be fretful for somthing that solves that problem.
xp84
I love this! it encourages you to mentally define the steps. To me, the next step is to define the "inputs" and "outputs" that each step should have. Next after that in many cases would be "what to do or just what to suggest to the user if each of the steps fail." And at that point you have not only a nicely already function-based outline if you want to make it a real automation.
It's a thought technology which I suspect will result in better final scripts than what I have tended to do, which is much more linear and requires a significant refactoring step at the end to avoid having one long function of poor quality.
rpicard
I like this way of thinking. I’m working on a bunch of automations right now and using GitHub actions.
I find the idea of setting up “do nothing workflows” that I can compose and implement more thoroughly over time helpful. I’ll probably put this into use.
dolmen
This is an interesting approach.
However the task used as an example just shows how provisionning SSH keys was insecure at that company. In fact the user should generate his private key by himself and just provide the public key to the sysadmin to inject it in the system to grant access. AT NO POINT THE SYSDAMIN SHOULD HAVE A COPY OF THE PRIVATE KEY, even temporary. So the 1Password step shouldn't even be necessary.
By the way, I'm the author of github-keygen, a tool to automate the creation of SSH keys dedicated to GitHub access, and to setup SSH settings for that context.
https://github.com/dolmen/github-keygen
Jtsummers
Past discussions (lots of comments):
https://news.ycombinator.com/item?id=29083367 – 3 years ago (230 comments)
https://news.ycombinator.com/item?id=20495739 – 6 years ago (124 comments)
mamidon
To be clear; you're supposed to have a separate shell open where you do these steps?
MM, interesting idea.
eyehatewindows
[dead]
starkparker
Ideally this documentation would also document the expected output, both toward identifying when the process has gone off the rails when doing it manually and making the steps testable once automated.
Otherwise, it'd be trivially easy for an unfamiliar user (or the automated script) to ignore unclear errors or exit codes and march blindly to the next wait_for_enter().
deadbabe
Great in theory until some error or problem comes up and you don’t know how to handle it.
__MatrixMan__
I bet you could adapt this for orchestrators besides a script. For instance if the goal is to eventually have it all running in Airflow, you could have a do-nothing DAG, which would be nice because your co-workers could see that you're already half-way through today's task, whereas separate python scripts don't know about each other. You could just use a `sleep infinity` to keep the task "running" until somebody manually sets it to "success".
parentheses
This approach always sounds great upon initial consideration but has a fatal flaw. It doesn't scale because people are lazy. Over time, slogs will always revert to their original form.
avipars
A good way of documenting processes for people that prefer to look at code than long PDfs
tejtm
Haven't had my coffee yet but this strike me as, "reinvent the Makefile".
Which I have certainly used as a place to codify steps with benefits.
parasti
This is actually brilliant. This is a step above every list of instructions that I've ever made without committing to full-blown all-at-once automation.
perpil
I did something like this but it helps you build exact command lines or interact with AWS straight from your GitHub markdown. As a bonus, it very easily lets you prompt the user for inputs and pivot between different accounts. https://speedrun.cc
hinkley
I finally tried this an about a year so after it was first posted.
Due to bugs in our toolchain we had a run book for hot fixes that was about twice as complicated as the normal release process.
I never got the credit it deserved but it went from people only using it for sev 1 issues and “last mile” work in epics that were winding up, say once every ten weeks, to using it on average once a week and a couple times 3 in one week. We were able to dig a lot deeper into tech debt because not every single thing had to be a feature toggle.
If you’re at a small company where cloning prod data into preprod is easy, you won’t see this sort of result. But I counted over 150 endpoints we talked to and I believe that averaged 3 per service. So that’s a lot of datasets, and some of them ingested in a Kafka-before-Kafka-existed sort of manner. We had only one guy who would even try to clone prod data, he only had the time (and really energy) to do it once or twice a year, and that was much slower than our customers and features morphed. So it was down to fiddling with the blue-green deployment process and jmeter to figure out if we were close and how to measure success/failure before we went live.
And in the end it was the fiddly error-prone build process that stymied people until I half-automated it.
Later on as we ramped up its use I hunted down all of the URLs for the manual steps and put them in a lookup table in the tool, and ended up exposing them for the normal validation process we did for release sign off as well. Which made that process a little faster and less stressful for the coordinator (that process was annoying enough that we round robined it through three separate teams to share the load)
throwpoaster
Even though “toil” is a term of art, be careful using it around executives who are not SMEs.
I once saw an SRE get managed out because he kept talking about “reducing toil” and his director eventually said, “he obviously didn’t like toiling here so we helped him leave.”
oulipo
Just did a slightly more elaborate version (mostly colors)
run it with `uv run –script script.py`
iancmceachern
I cannot overstate how big a fan i am of this approach.
I've successfully applied this approach to so many projects. My favorite example is a $30 million surgical robot that was failing labs because of the "human factor".
machine_ghost
This same is also known by another name when it's used to teach new programmers: pseudo code.
Sadly, because it's associated with learners/juniors, pseudo code gets a bad rap. But really, all engineering is translating English into code … which means almost any complex operation (setting up a new employee's account, or anything else) can be made clearer by utilizing such an "in-between English and code" step.
In the article they used Python, but for all the important stuff, they didn't: they used English. A "do-nothing" script is really just a script where instead of converting pseudo code to code (like programmers normally do), you just leave the English/pseudo code in, and wrap it with a print.
layer8
I would use a checklist for that. Most wiki/notes/todo software supports checklists or checkboxes. It has the benefit that you can still see all steps at once, perform some steps out of order if the dependency isn’t linear, and it can be simply part of your process documentation instead of having to find and run a script. It’s also easier to edit.
The article even mentions “just another checkbox checked”.
outofpaper
This is a waste of energy as there a cleaner clearer more effective ways of dealing with this script. You can go much cleaner and just do a Markdown Slideshow. Alternatively allow for a little real automation and use something like Expect or Pexpect.
https://pexpect.readthedocs.io/en/stable/overview.html
Don't pretend to yourselves that presenting a series of instructions and only advancing on when someone clicks enter will prevent people hit enter.
Even worse is someone missed up there's no going back. You have to either restart the whole process or you have to go and read over the python script.
This is not something that prevents or helps with slog.
aqueueaqueue
Oh yeah! I've seen this pattern used at work. I don't have an opinion on if it is better than a checklist. The lack of will or time to automate affects a do nothing script as much as runbook.
jiggawatts
My variant of this is putting the manual steps in between scripted steps as a comment, but having read the article I can see the benefit of this approach.
In PowerShell especially there are elegant ways to prompt users for “continue y/n?”.
PaulRobinson
1. Write the runbook
2. Automate the runbook
3. Delete the runbook
This is just a means to get #2 done iteratively. I've done variations for a while, its a powerful technique.
po84
We taken this approach using streamlit to incrementally turn Markdown lists of instructions into mini, self-service web apps. Steps in lists are gradually replaced with widgets, collecting inputs for the automation to do the work instead.
m463
do each of the steps, as an object, look overengineered?
I follow the advice of this column, but I differ in my implementation. It is lightweight and not so OO.
midtake
Then it really isn't doing nothing. Rather, it's a to-do app in the terminal.
The reason it is helpful for decreasing activation energy is that it handles the "checklist" aspect of creating a user account for you. This is especially helpful for user account creation, which has to be done completely and then logged or it can lead to cybersecurity incidents.
remram
Objects with only a single method to run it are already built into Python, they are the functions.
20after4
I found this a few years ago and was quite inspired by it. I then more recently found https://xcfile.dev/ which is a great way to implement these sort of scripts directly in the project's README.
sagarpatil
I’ll probably get down voted to oblivion, but this is really silly. I do not see any value proposition. To each their own, I guess.
stego-tech
This is, to a degree, how I’ve always written my SOPs. If anything, this approach is encouraging me to make two changes to my documentation approach:
1) Add a “Quick Run” section with just the steps/commands absent images and details, so it’s easier to perform routine slogs without having to scroll through unnecessary information.
2) Make (and link!) a companion code document that has the pseudocode for the process written out, so folks can contribute to it as time allows. If I’m a manager, I’d even incentivize it somehow – maybe covering a team lunch if we convert an SOP into an automation, for instance.
In the meantime, I am totally cribbing this for documenting my homelab stuff and getting it out of Obsidian, as a means of encouraging me to automate routine tasks somehow.