Escaping Surprise Bills & Over-engineered Messes: Why I Left AWS
I love building side projects. They’ve been a way to push myself and explore new ideas and technologies. Each site has needed hosting. I started my hosting journey with WordPress. I moved on to raw Linux servers and finally ended up on AWS. Hosting on AWS felt like a badge of honor, but it also felt like a ticking time bomb of complexity and cost.
Surprise Bills
I’ve heard horror stories of accidental bills. They ranged from a few hundred to several hundred thousand dollars. Sometimes they’re refunded, like when the story hits the front page of Hacker News and Reddit, but most times not.
I searched for ways to control my financial exposure on AWS. I expected a simple monthly max spend option. I found none. There were mentions of setting up alerts. What if I got DDoSed at 3 a.m.? Some suggested setting up intricate systems to shut everything down. What if they were setup incorrectly?
Over-Engineered Messes
I have noticed that we as engineers have a tendency over-engineer systems. We tend to mimic the practices of companies like Facebook or Netflix. We over-engineer solutions, choosing technologies that are often overkill for our actual needs. For me, AWS was far more than I needed.
NearlyFreeSpeech to the Rescue
After some research and conversations with other engineers, I discovered NearlyFreeSpeech (NFS). Their approach to hosting was exactly what I was looking for. One blog post stood out to me:
“[…] since our service is paid in advance, you always have complete control over your maximum financial liability simply by controlling the balance of your account. If you feel your site is attack-prone and you are primarily concerned about costs, we encourage you to maintain a low account balance to limit your exposure.” – Blog Post
This model was perfect. By prepaying, I could ensure complete control over my financial liability.
It’s possible with NFS to setup several billing accounts with prepaid balances. I have a primary account that I add a few
22 Comments
Insanity
That is pretty cool. I tend to default to AWS, luckily not too expensive for my side projects (about $15/month) – nothing accessible to the public though so my cost is relatively predictable.
That said, I do wish you could hard shutdown at a certain budget limit.. but guess that is not in AWS’s best interested.
faizshah
Personally I use lightsail on AWS and cloudflare cause there is always an off ramp to try some of the fancy stuff but then you can always go back to just using cheap VMs behind cloudflare. You can also put it all behind a VPC and you can use CDK/CloudFormation so that’s also nice.
I gave up on using GCP even though the products like BigQuery are way better just because I got burned too many times like with the Google Domains -> Squarespace transition.
I’m thinking of switching back to a bare metal provider now like Vultr or DO (would love to know what people are using these days I haven’t used bare metal providers since ~2012).
Also, completely unrelated does anyone know what the best scraping proxy is these days for side projects (data journalism, archiving etc.)?
deathanatos
I've used NearlyFreeSpeech for years (as registrar & DNS), and I've loved their service. Their site is plain, and you just trade money for a plain, simple product, with basically 0 bullshit between you and that exchange. Their site is so refreshing in today's landscape of upsells and other corporate dark patterns.
The article implicates AWS, but AFAICT the other major cloud, GCP, behaves similarly. The docs for "budget alerts"[1] do call it out directly,
> Setting a budget does not automatically cap Google Cloud or Google Maps Platform usage or spending. Budgets trigger alerts to inform you of how your usage costs are trending over time. Budget alert emails might prompt you to take action to control your costs, but they don't automatically prevent the use or billing of your services when the budget amount or threshold rules are met or exceeded.
But still. But wait, you say, those docs go on to suggest,
> One option to automatically control spending is to use budget notifications to programmatically disable Cloud Billing on a project.
And the linked page states,
> Following the steps in this capping example doesn't guarantee that you won't spend more than your budget.
sigh "Over-Engineered Messes", TFA hits it on the nose.
There's also limiting API usage, but that's on requests … not on cost.
I avoid it all for personal stuff.
At work, we pipe all these cloud bills into a BigQuery account which then pipes into graphs in Grafana, which all tells us that engineers have no idea what the actual horsepower of 3 GHz * 32 cores is when they request a bajillion more cores.
It's probably also reasonably categorized as an "Over-Engineered Mess".
(We also import Azure's billing data, and boy do they make that obnoxious. CSV files dumping into a bucket, and they might go back & edit CSVs, or drop new ones, and if there is there a schema for those CSV files … I've yet to find it. Some columns you might think were guaranteed non-"" are not. Dates in American. Severely denormalized. Etc.)
[1]: https://cloud.google.com/billing/docs/how-to/budgets
bschwindHN
Truth is, most web projects made today can run on a raspberry pi or mini PC and be just fine. If you have enough users that you need to scale to more machines, you'll be in a position to know what to do to handle it, or hire someone who does.
ufmace
I don't think "surprise bills" is a good reason to avoid AWS. Yes, you avoid them on other providers by doing simple bare-metal Linux boxes. But doing a bare-metal Linux box with no other services is also fixed-cost at AWS. You only get into tricky to determine in advance variable costs when you string together a bunch of AWS's extra services in ways that you don't understand well or don't/can't set limits on.
On the other hand, doing single bare-metal boxes being much more complex and usually more expensive is a good reason to skip AWS for simple projects. In addition to the profusion of instance types and billing/usage options.
I also think the lack of options on limiting max billing for flexible services is pretty reasonable actually. For most of them, there's no single obvious reasonable thing to do when the money hits the limit. Storage costs money too, but I don't think much of anyone really wants to have their data get deleted when the cost hits the limit, for example.
sema4hacker
Ironically, NearlyFreeSpeech still didn't exactly provide the "simple monthly max spend option" you were looking for, because now you're maintaining multiple prepaid accounts. It's silly that online services won't provide a billable account with a hard max.
placardloop
AWS isn’t and has never been economical for side projects or hobby tinkering, unless you specifically want to tinker with AWS.
I’m a big AWS fan. I’d recommend any company of decent size to use AWS. But seriously, if your project is just a personal blog or some rails app you tinker with on the weekend, just get a $5/mo instance on Digital Ocean or a raspberry pi.
wao0uuno
Reads like an ad.
scrose
Over the years I’ve
spent a lot of time talking engineers and managers out of using serverless AWS options for various reasons. I’ve found that most non-infra focused engineers and managers see serverless marketed as “simpler” and “cheaper”.
It’s often the opposite, but most people don’t see that until after they’ve built their infrastructure around that, get locked in, and then start seeing the surprise bills, difficult to diagnose system failures, and hard-limitations start rolling in.
A bit of early skepticism, and alternative solutions with a long-term perspective in mind, often go a long way.
taurknaut
I think it's fairly obvious why people try to escape aws. What is far less clear is why folks try to embrace their own solutions when this clearly ain't their forte
irjustin
I see these articles on HN enough almost always end up agreeing. My gross over simplification of this problem is:
At that low end, you can get a mix of basic-but-okay DB, web servers, caching, CDN and maybe a few Lambdas all with logging/x-ray. All you'll need to run a good site/service w/ low-medium traffic and solid reliability. At around $3k/mth, you likely know whether you're in AWS to stay (using many services) or are eyeing the self hosting route.
Side projects really need to optimize under this $5-50/mth and that's just too low on AWS. The expensive foot-guns are just too risky.
djtango
Found the article light on details here.
What kind of AWS setup leads to surprise bills? If your blog ends up on front page of HN how much is the traffic spike going to cost? $100? $1000? $10000?
Ofc I've seen the horror stories of recursive lambdas causing 1M+ in compute costs but for a small site I'm unsure what kind of issues we're talking about here…
If pages are mostly static doesn't a CDN make most of this go away?
abrookewood
Talk about burying the lede: "My bill has increased from about $1 to $7 a month."
I agree with much of the sentiment, but I don't see how complex things you could possibly be making things if you're paying $1 a month …
sieve
> One app gave me trouble – a Python Flask app. It had several complicated dependencies including OpenCV, Numpy, and Matplotlib. My belief is that this had more to do with the complex nature of the libraries and less to do with NFS.
It has to do with the allround lunacy that surrounds the python ecosystem the moment you step outside the stdlib nursery. I picked up python because I wanted to contribute to a project. Was immediately met with ** of the first order.
After decades of C-like languages, I like (some of) the syntax. But I hate the ecosystem and having to ride a bullock cart after cruising in a plane. The language is slow. So all the critical work is done in C/Rust. And then you write python bindings for your library. Once that is available, you can "program in python."
The dependencies are a nightmare, particularly if you do anything LLM-related. Some libraries simply won't work with newer versions of the interpreter. After decades of Java, this came as a big surprise.
If it were not for uv, I might have given up on the language altogether.
andrewstuart
You can get 1Gbps unlimited traffic VPS on IONOS 12 vCores, 24GB RAM, 640GB storage for $50/month.
No need to pay 9 cents per GB egress to the big clouds.
https://www.ionos.com/servers/cloud-vps
blackeyeblitzar
I like that the service he recommends, NearlyFreeSpeech, will host anything that is legal. But they do require your real identity info. Is there a way to anonymously host websites today?
rob_c
Most developers aren't sysadmins and most sysadmins don't work closely with their developers.
After you factor that in most of the disasters of this type involving the cloud or home spun rust are easily explained away.
The former are missing skills that should have been beet into them by any decent education and the later is normally overwhelmed and in need of a team to replace them.
zerof1l
I use Lambda + DynamoDB to run a very basic app for managing access to my servers. The monthly cost is $0.02 – $0.05.
I'm also looking into using S3 glacier for my server backups. It appears to be the cheapest option.
tietjens
I’m very confused by this blog post. The risk of exploding costs he’s talking about comes from an attack, right?
So he doesn’t have the risk appetite for a DDOS attack nor the desire to figure out how to properly mitigate one. And this is AWS’s fault?
rendaw
I was using Azure for a cloud windows desktop which I need rarely, for using something like itunes or kobo to download books. It cost ~$5 a month.
One day windows update bricks the system (hanging while trying to revert some security patch), and over a couple months from time to time I try random janky official azure recovery tools and voodoo from user forums with people who don't really know what they're doing either.
I notice my bill has crept up to several hundred dollars a month. Each of the recovery tools was cloning the system + disks, and I ended up with a bunch of disks that chewed up my bill.
I raised a support ticket and they refunded part of it with a bit of "you're a bad person", but wow… although the primary lesson I got here is that I never want to use windows again.
bambax
I am self hosting on a NAS at home, with Cloudflare in front (which does the most of the work) and Cloudflare tunnels to avoid exposing anything directly. The tunnel communicates with various Docker instances depending on the services.
It works flawlessly for now, and costs almost zero, since the NAS is always on in any case, and Cloudflare is free.
These are all small projects of course, but two of them stayed on HN frontpage for a day and didn't break a sweat.
Havoc
Other good option for hardcaps is to put as much as possible on bunny cdn
Still means the dynamic logic need to go elsewhere but for basic storage it works well for prepay