This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outages. Rust won't help, people will always make mistakes, also in Rust. Robust architecture addresses this by not allowing a single mistake to bring down myriad of unrelated services at once.
First, let’s set aside the separate question of whether monopolies are bad. They are not good but that’s not the issue here.
As to architecture:
Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so.
But there’s a more interesting argument in favour of the status quo.
Assuming cloudflare’s uptime is above average, outages affecting everything at once is actually better for the average internet user.
It might not be intuitive but think about it.
How many Internet services does someone depend on to accomplish something such as their work over a given hour? Maybe 10 directly, and another 100 indirectly? (Make up your own answer, but it’s probably quite a few).
If everything goes offline for one hour per year at the same time, then a person is blocked and unproductive for an hour per year.
On the other hand, if each service experiences the same hour per year of downtime but at different times, then the person is likely to be blocked for closer to 100 hours per year.
It’s not really bad end user experience that every service uses cloudflare. It’s more-so a question of why is cloudflare’s stability seeming to go downhill?
And that’s a fair question. Because if their reliability is below average, then the value prop evaporates.
> On the other hand, if each service experiences the same hour per year of downtime but at different times, then the person is likely to be blocked for closer to 100 hours per year.
I think the parent post made a different argument:
- Centralizing most of the dependency on Cloudflare results in a major outage when something happens at Cloudflare, it is fragile because Cloudflare becomes the single point of failure. Like: Oh Cloudflare is down... oh, none of my SaaS services work anymore.
- In a world where this is not the case, we might see more outages, but they would be smaller and more contained. Like: oh, Figma is down? fine, let me pickup another task and come back to Figma once it's back up. It's also easier to work around by having alternative providers as a fallback, as they are less likely to share the same failure point.
As a result, I don't think you'll be blocked 100 hours a year in scenario 2. You may observe 100 non-blocking inconveniences per year, vs a completely blocking Cloudflare outage.
And in observed uptime, I'm not even sure these providers ever won. We're running all our auxiliary services on a decent Hetzner box with a LB. Say what you want, but that uptime is looking pretty good compared to any services relying on AWS (Oct 20, 15 hours), Cloudflare (Dec 5 (half hour), Nov 18 (3 hours)). Easier to reason about as well. Our clients are much more forgiving when we go down due to Azure/GCP/AWS/Cloudflare vs our own setup though...
> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so.
The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet.
If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the world aren’t BofA customers.
I'm not sure I follow the argument. If literally every individual site had an uncorrelated 99% uptime, that's still less available than a centralized 99.9% uptime. The "entire Internet" is much less available in the former setup.
It's like saying that Chipotle having X% chance of tainted food is worse than local burrito places having 2*X% chance of tainted food. It's true in the lens that each individual event affects more people, but if you removed that Chipotle and replaced with all local, the total amount of illness is still strictly higher, it's just tons of small events that are harder to write news articles about.
No it's like saying if one single point of failure in a global food supply chain fails, nobody's going to eat today. And which is in contrast to if some supplier fails to provide a local food truck today their customers will have to go to the restaurant next door.
Ah ok, it is true that if there's a lot of fungible offerings that worse but uncorrelated uptime can be more robust.
I think the question then is how much of the Internet has fungible alternatives such that uncorrelated downtime can meaningfully be less impact. If you have a "to buy" shopping list, the existence of alternative shopping list products doesn't help you, when the one you use is down it's just down, the substitutes cannot substitute on short notice. Obviously for some things there's clear substitutes though, but actually I think "has fungible alternatives" is mostly correlated with "being down for 30 minutes doesn't matter", it seems that the things where you want the one specific site are the ones where availability matters more.
The restaurant-next-door analogy, representing fungibility, isn't quite right. If BofA is closed and you want to do something in person with them, you can't go to an unrelated bank. If Spotify goes down for an hour, you're not likely to become a YT Music subscriber as a stopgap even though they're somewhat fungible. You'll simply wait, and the question is: can I shuffle my schedule instead of elongating it?
A better analogy is that if the restaurant you'll be going to is unexpectedly closed for a little while, you would do an after-dinner errand before dinner instead and then visit the restaurant a bit later. If the problem affects both businesses (like a utility power outage) you're stuck, but you can simply rearrange your schedule if problems are local and uncorrelated.
If utility power outage is put on the table, then the analogy is almost everyone solely relying on the same grid, in contrast with being wired to a large set of independent providers or even using their own local solar panel or whatever autonomous energy source.
Look at it a user (or even operator) of one individual service that isn’t redundant or safety critical: if choice A has 1/2 the downtime of choice B, you can’t justify choosing choice B by virtue of choice A’s instability.
The world dismantled landlines, phone booths, mail order catalogues, fax machines, tens of millions of storefronts, government offices, and entire industries in favor of the Internet.
So at this point no, the world can most definitely not “just live without the Internet”. And emergency services aren’t the only important thing that exists to the extent that anything else can just be handwaved away.
> If Cloudflare is at 99.95% then the world suffers
if the world suffers, those doing the "suffering" needs to push that complaint/cost back up the chain - to the website operator, which would push the complaint/cost up to cloudflare.
The fact that nobody did - or just verbally complained without action - is evidence that they didn't really suffer.
In the mean time, BofA saved cost in making their site 99.95% uptime themselves (presumably cloudflare does it cheaper than they could individually). So the entire system became more efficient as a result.
> The fact that nobody did - or just verbally complained without action - is evidence that they didn't really suffer.
What an utterly clueless claim. You're literally posting in a thread with nearly 500 posts of people complaining. Taking action takes time. A business just doesn't switch cloud providers overnight.
I can tell you in no uncertain terms that there are businesses impacted by Cloudflare's frequent outages that started work shedding their dependency on Cloudflare's services. And it's not just because of these outages.
Terrible examples, Github and Google aren't just websites that one would place behind Cloudflare to try to improve their uptime (by caching, reducing load on the origin server, shielding from ddos attacks). They're their own big tech companies running complex services at a scale comparable to Cloudflare.
> If everything goes offline for one hour per year at the same time, then a person is blocked and unproductive for an hour per year.
> On the other hand, if each service experiences the same hour per year of downtime but at different times, then the person is likely to be blocked for closer to 100 hours per year.
Putting Cloudflare in front of a site doesn't mean that site's backend suddenly never goes down. Availability will now be worse - you'll have Cloudflare outages* affecting all the sites they proxy for, along with individual site back-end failures which will of course still happen.
On the other hand, if one site is down you might have alternatives. Or, you can do something different until the site you needed is up again. Your argument that simultaneous downtime is more efficient than uncoordinated downtime because tasks usually rely on multiple sites being online simultaneously is an interesting one. Whether or not that's true is an empirical question, but I lean toward thinking it's not true. Things failing simultaneously tends to have worse consequences.
"My architecture depends upon a single point of failure" is a great way to get laughed out of a design meeting. Outsourcing that single point of failure doesn't cure my design of that flaw, especially when that architecture's intended use-case is to provide redundancy and fault-tolerance.
The problem with pursuing efficiency as the primary value prop is that you will necessarily end up with a brittle result.
> "My architecture depends upon a single point of failure" is a great way to get laughed out of a design meeting.
This is a simplistic opinion. Claiming services like Cloudflare are modeled as single points of failure is like complaining that your use of electricity to power servers is a single point of failure. Cloudflare sells a global network of highly reliable edge servers running services like caching, firewall, image processing, etc. And more importas a global firewall that protects services against global distributed attacks. Until a couple of months ago, it was unthinkable to casual observers that Cloudflare was such an utter unreliable mess.
Your electricity to servers IS a single point of failure, if all you do is depend upon the power company to reliably feed power. There is a reason that co-location centers have UPS and generator backups for power.
It may have been unthinkable to some casual observers that creating a giant single point of failure for the internet was a bad idea but it was entirely thinkable to others.
That's an interesting point, but in many (most?) cases productivity doesn't depend on all services being available at the same time. If one service goes down, you can usually be productive by using an alternative (e.g. if HN is down you go to Reddit, if email isn't working you catch up with Slack).
Many (I’d speculate most) workflows involve moving and referencing data across multiple applications. For example, read from a spreadsheet while writing a notion page, then send a link in Slack. If any one app is down, the task is blocked.
Software development is a rare exception to this. We’re often writing from scratch (same with designers, and some other creatives). But these are definitely the exception compared to the broader workforce.
Same concept applies for any app that’s built on top of multiple third-party vendors (increasingly common for critical dependencies of SaaS)
> If everything goes offline for one hour per year at the same time, then a person is blocked and unproductive for an hour per year.
This doesn’t guarantee availability of those N services themselves though, surely? N services with a slightly lower availability target than N+1 with a slightly higher value?
More importantly, I’d say that this only works for non-critical infrastructure, and also assumes that the cost of bringing that same infrastructure back is constant or at least linear or less.
The 2025 Iberian Peninsula outage seems to show that’s not always the case.
> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so.
I disagree; most people need only a subset of Cloudflare's features. Operating just that subset avoids the risk of the other moving parts (that you don't need anyway) ruining your day.
Cloudflare is also a business and has its own priorities like releasing new features; this is detrimental to you because you won't benefit from said feature if you don't need it, yet still incur the risk of the deployment going wrong like we saw today. Operating your own stack would minimize such changes and allow you to schedule them to a maintenance window to limit the impact should it go wrong.
The only feature Cloudflare (or its competitors) offers that can't be done cost-effectively yourself is volumetric DDoS protection where an attacker just fills your pipe with junk traffic - there's no way out of this beyond just having a bigger pipe, which isn't reasonable for any business short of an ISP or infrastructure provider.
Paraphrasing: We are setting aside the actual issue and looking for a different angle.
To me this reads as a form of misdirection, intentional or not. A monopolist has little reason to care about downstream effects, since customers have nowhere else to turn. Framing this as roll your own versus Cloudflare rather than as a monoculture CDN environment versus a diverse CDN ecosystem feels off.
That said, the core problem is not the monopoly itself but its enablers, the collective impulse to align with whatever the group is already doing, the desire to belong and appear to act the "right way", meaning in the way everyone else behaves. There are a gazillion ways of doing CDN, why are we not doing them? Why the focus on one single dominant player?
I don’t the answer to the all questions. But here I think it is just a way to avoid responsibility. If someone choses CDN “number 3” and it goes down, business people *might* put a blame on this person for not choosing “the best”.
I am not saying it is a right approach, I just seen it happens too many times.
CloudFlare doesn’t have a good track record. It’s the third party that caused more outages for us than any other third party service in the last four years.
That’s fine if it’s just some random office workers. What if every airline goes down at the same time because they all rely on the same backend providers? What if every power generator shuts off? “Everything goes down simultaneously” is not, in general, something to aim for.
> Cloudflare has had some outages recently. However, what’s their uptime over the longer term? If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so.
Why is that the only option? Cloudflare could offer solutions that let people run their software themselves, after paying some license fee. Or there could be many companies people use instead, instead of everyone flocking to one because of cargoculting "You need a CDN like Cloudflare before you launch your startup bro".
What you’re suggesting is not trivial. Otherwise we wouldn’t use various CDNs. To do what Cloudflare does your starting point is “be multiple region/multiple cloud from launch” which is non-trivial especially when you’re finding product-market fit. A better poor man’s CDN is object storage through your cloud of choice serving HTTP traffic. Cloudflare also offers layers of security and other creature comforts. Ignoring the extras they offer, if you build what they offer you have effectively made a startup within a startup.
Cloudflare isn’t the only game in town either. Akamai, Google, AWS, etc all have good solutions. I’ve used all of these at jobs I’ve worked at and the only poor choice has been to not use one at all.
What do you think Cloudflare’s core business is? Because I think it’s two things:
1. DDoS protection
2. Plug n’ Play DNS and TLS (termination)
Neither of those make sense for self-hosted.
Edit: If it’s unclear, #2 doesn’t make sense because if you self-host, it’s no longer plug n’ play. The existing alternatives already serve that case equally well (even better!).
All of my company's hosted web sites have way better uptimes and availability than CF but we are utterly tiny in comparison.
With only some mild blushing, you could describe us as "artisanal" compared to the industrial monstrosities, such as Cloudflare.
Time and time again we get these sorts of issues with the massive cloudy chonks and they are largely due to the sort of tribalism that used to be enshrined in the phrase: "no one ever got fired for buying IBM".
We see the dash to the cloud and the shoddy state of in house corporate IT as a result. "We don't need in-house knowledge, we have "MS copilot 365 office thing" that looks after itself and now its intelligent - yay \o/
Until I can't, I'm keeping it as artisanal as I can for me and my customers.
Sorry for the downvotes but this is true many times with some basic HA you get better uptime than the big cloud boys, yes their stack and tech is fancier but we also need to factor in how much CF messes with it vs self hosted, anyway the self hosted wisdom is RIP these days and I mostly just run cf pages / kv :)
Usually I am allergic to pithy, vaguely dogmatic summaries like this but you're right. We have traded "some sites are down some of the time" for "most sites are down some of the time". Sure the "some" is eliding an order of magnitude or two, but this framing remains directionally correct.
Does relying on larger players result in better overall uptime for smaller players? AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team.
If so, is it a good or bad trade to have more overall uptime but when things go down it all goes down together?
From a societal view it is worse when everything is down at once. Leads to a less resilient society: It is not great if I can't buy essentials from one store because their payment system is down (this happened to one super market chain in Sweden due to a hacker attack some years ago, took weeks to fully fix everything, and then there was that whole Crowdstrike debacle globally more recently).
It is far worse if all of the competitors are down at once. To some extent you can and should have a little bit of stock at home (water, food, medicine, ways to stay warm, etc) but not everything is practical to do so with (gasoline for example, which could have knock on effects on delivery of other goods).
users want to do things, if their goal depends on a complex chain of functions (provided by various semi-independent services) then the ideal setup would be to have redundant providers and users could simply "load balance" between them and that separate high-level providers' uptime state is clustered (meaning that when Google is unavailable Bing is up, and when Random Site A, goes down their payment provider goes down too, etc..)
So ideally sites would somehow sort themselves nearly to separate availability groups.
Otherwise simply having a lot of uncorrelated downtimes doesn't help (if we count the sum of downtime experienced by people). Though again it gets complicated by the downtime percentage, because likely there's a phase shift between the states when user can mostly complete their goals and when they cannot because too many cascading failures.
> AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team.
Is it? I can’t say that my personal server has been (unplanned) down at any time in the past 10 years, and these global outages have just flown right past it.
When only one thing goes down, it's easier to compensate with something else, even for people who are doing critical work but who can't fix IT problems themselves. It means there are ways the non-technical workforce can figure out to keep working, even if the organization doesn't have on-site IT.
Also, if you need to switchover to backup systems for everything at once, then either the backup has to be the same for everything and very easily implementable remotely - which to me seems unlikely for specialty systems, like hospital systems, or for the old tech that so many organizations still rely on (and remember the CrowdStrike BSODs that had to be fixed individually and in person and so took forever to fix?) - or you're gonna need a LOT of well-trained IT people, paid to be on standby constantly, if you want to fix the problems quickly, on account of they can't be everywhere at once.
If the problems are more spread out over time, then you don't need to have quite so many IT people constantly on standby. Saves a lot of $$$, I'd think.
And if problems are smaller and more spread out over time, then an organization can learn how to deal with them regularly, as opposed to potentially beginning to feel and behave as though the problem will never actually happen. And if they DO fuck up their preparedness/response, the consequences are likely less severe.
AWS and Cloudflare can recover from outages faster because they can bring dozens (hundreds?) of people to help, often the ones who wrote the software and designed the architecture. Outages at smaller companies I've worked for have often lasted multiple days, up to an exchange server outage that lasted 2 weeks.
Would you rather be attacked by 1,000 wasps or 1 dog? A thousand paper cuts or one light stabbing? Global outages are bad but the choice isn’t global pain vs local pleasure. Local and global both bring pain, with different, complicated tradeoffs.
Cloudflare is down and hundreds of well paid engineers spring into action to resolve the issue. Your server goes down and you can’t get ahold of your Server Person because they’re at a cabin deep in the woods.
It's not "1,000 wasps or 1 dog", it's "1,000 dogs at once, or "1 dog at once, 1,000 different times". Rare but huge and coordinated siege, or a steady and predictable background radiation of small issues.
The latter is easier to handle, easier to fix, and much more suvivable if you do fuck it up a bit. It gives you some leeway to learn from mistakes.
If you make a mistake during the 1000 dog siege, or if you don't have enough guards on standby and ready to go just in case of this rare event, you're just cooked.
I don't quite see how this maps onto the situation. The "1000 dog seige" also was resolved very quickly and transparently, so I would say it's actually better than even one of the "1 dog at once"s.
That's a reflect of social organisation. Pushing for hierarchical organisation with a few key centralising nodes will also impact business and technological decisions.
Actually, maybe 1 hour downtime for ~ the whole internet every month is a public good provided by Cloudflare. For everyone that doesn’t get paged, that is.
On the other hand, as long as the entire internet goes down when Cloudflare goes down, I'll be able to host everything there without ever getting flack from anyone.
It's not as simple as that. What will result in more downtime, dependency on a single centralized service or not being behind Cloudflare? Clearly it's the latter or companies wouldn't be behind Cloudflare. Sure, the outages are more widespread now than they used to be, but for any given service the total downtime is typically much lower than before centralization towards major cloud providers and CDNs.
> Rust won't help, people will always make mistakes, also in Rust.
They don't just use Rust for "protection", they use it first and foremost for performance. They have ballpark-to-matching C++ performance with a realistic ability to avoid a myriad of default bugs. This isn't new.
You're playing armchair quarterback with nothing to really offer.
What you've identified here is a core part of what the banking sector calls the "risk based approach". Risk in that case is defined as the product of the chance of something happening and the impact of it happening. With this understanding we can make the same argument you're making, a little more clearly.
Cloudflare is really good at what they do, they employ good engineering talent, and they understand the problem. That lowers the chance of anything bad happening. On the other hand, they achieve that by unifying the infrastructure for a large part of the internet, raising the impact.
The website operator herself might be worse at implementing and maintaining the system, which would raise the chance of an outage. Conversely, it would also only affect her website, lowering the impact.
I don't think there's anything to dispute in that description. The discussion then is if cloudflares good engineering lowers the chance of an outage happening more than it raises the impact. In other words, the things we can disagree about is the scaling factors, the core of the argument seems reasonable to me.
You're not wrong, but where's the robust architecture you're referring to? The reality of providing reliable services on the internet is far beyond the capabilities of most organizations.
I think it might be a organizational architecture that needs to change.
> However, we have never before applied a killswitch to a rule with an action of “execute”.
> This is a straightforward error in the code, which had existed undetected for many years
So they shipped an untested configuration change that triggered untested code straight to production. This is "tell me you have no tests without telling me you have no tests" level of facepalm. I work on safety-critical software where if we had this type of quality escape both internal auditors and external regulators would be breathing down our necks wondering how our engineering process failed and let this through. They need to rearchitect their org to put greater emphasis on verification and software quality assurance.
Obviously Rust is the answer to these kind of problems. But if you are cloudflare and have an important company at a global scale, you need to set high standarts for your rust code. Developers should dance and celebrate end of the day if their code compiles in rust.
Before LLMs programmers had pretty good intuition what GPL license allowed for. It is of course clear that you cannot release a closed source program with GPL code integrated into it. I think it was also quite clear, that you cannot legally incorporate GPL code into such a program, by making changes here and there, renaming some stuff, and moving things around, but this is pretty much what LLMs are doing. When humans do it intentionally, it is violation of the license, when it is automated and done on a huge scale, is it really fair use?
With self driving cars population on roads increasing, a side effect can be that all traffic will be shaped towards staying within the speed limits. With more cars staying within the limits, breaking the limits becomes more difficult.
For containers you will also have own TCP/IP stack similarly to what is shown for VM on the diagram, this is done when a container uses slirp4netns to provide networking. An alternative is to use kernel TCP/IP stack, this is done when pasta is used for networking, diagram on this page shows the details:https://passt.top/passt/about/.
I wonder how practical it would be to build a system that would let home appliances cheaply overuse energy when there is a peak in wind or solar production. For example:
* Let heat-pumps heat homes to say 23C instead of 20C
* Let freezers decrease the temperature to say -30C instead of -18C
* Let electric water heaters heat water to say 70C instead of 50C, such water can then be mixed with more cold water
Such overuse would then reduce energy consumption when the production peak is over (heat pumps could stop working for some time until the temperature decreases from 23 to 20, etc.)
You don't "build" such a system. You change the metering to follow supply, and everything else will follow naturally.
You'll have enthusiasts that'll do homebrew systems to take advantage of the economy, then you'll have companies catering to their (tbh, hobby), then you'll have products that are actually useful, then you'll see mass adoption. Like in everything else.
Trying to plan a huge strategy from the onset feels (and is!) daunting. Just make sure the price fits the reality, and savings will follow naturally.
> You change the metering to follow supply, and everything else will follow naturally.
Tell me your wonderland where this has happened . . .
There are whole countries with wireless meters. There must be papers showing how much effect it has on consumer consumption? Ignore one-off examples, I'm interested in population level effects and statistics.
There's a program called Hilo [1] in Québec where it's using the Hydro-Québec Rate Flex D [2] to automatically stop the heating during peak demand.
> With Rate Flex D, you can save quite a bit of money, since most of the time in winter, you’ll be charged less than the base rate, except during occasional peak demand events, when you’ll be charged more than the base rate.
"Ripple control" is vintage technology - hourly usage meters are not necessary.
Everyone imagines that consumers would change their behaviour if they were given price information. In my experience, I've yet to see any good data showing that on average consumers save electricity due to smart meters.
In New Zealand, I think the power companies design their consumer products to be unhelpful (what's the equivalent term here for dark patterns in marketing?). I believe few consumers watch their instant usage or review their hourly usage. Personally I changed away from a plan that used spot prices (after seeing the debacle from snow in Texas, and realising the rewards were low and that judging/managing the risks was hard).
Yeah right. Because building a freezer that goes to -30 C is as cheap as going to -18 C.
It's much beefier hardware with a lot more insulation.
Likewise a heat pump can only boost so much.
This, like other environment related changes never happen by market forces. Not once.
And small tweaks even on large scale produce small effects, insufficient for our needs.
Already kind of in place. I’m on the Octpus agile tariff that gives different electricity tariffs every 30 minutes - with 24 hour notice if tomorrow’s prices.
Whenever electricity prices go negative I have automations to force-charge my solar batteries from the grid, turn on hot water heaters in my hot water tank (normally heated by gas etc. ).
To add, so called 'dynamic energy contracts' are getting more and more popular, at least in my native Netherlands. The European day-ahead electricity market switched to 15-minute price blocks this month, to more accurately follow the supply and demand.
The market for power imbalance was already on 15 minute blocks.
I'm using a HomeWizard smart plug [0] to enable my electric boiler to only run during the cheapest hours of the day
You might find the crossover for hot water heating is higher than 0p; your boiler is likely only around 70% efficient. So at 6p/therm for gas, you'd break even with resistive electric heating at around the 10p/kWh mark.
You should absolutely re-run these numbers to be sure, but you might find you can use electric heating far more often than you might currently be doing.
Sure, I use Home Assistant running in a little raspberry pie in the lift.
There is an Octopus Integration that exposes current prices (and much else) to HomeAssistant.
There is another Integration that works with my solar panels and another that works with my batteries and can change mode (self use, force charge, force discharge etc.)
So from there it’s really just a question of setting up some if-then automations to turn on smart switches, charge the batteries if prices go negative.
You can also gradually add more nuanced automations like turning on water heaters if the panels are generating more than 1kW and the batteries are over 90% charged.
The only thing you would have to do to make this happen is to change electricity pricing from a fixed rate to a dynamic rate based on actual market conditions, along with a standardized way of accessing current pricing. This would drive consumers to shift their behaviors to take advantage of cheap prices, and smart appliances could access the price feed to make decisions like the ones you mention. Another simple one is washing machines, dryers and dishwasher offering to delay their start time to coincide with the cheapest energy price within X hours.
The issue is that most consumers don't like unpredictable prices. You can make a crude approximation by having 2-3 fixed rates for different times of day, but that leaves a lot of potential on the table
> The issue is that most consumers don't like unpredictable prices.
The key is to not take this away; make it so that those who want predictability can get it (but they end pay more for the privilege) but those who want to try to "game the system" can (and incidentally help with the overproduction problem).
Done well, things like Powerwalls, thermal mass storage, etc could absorb quite a bit of load during peak production times, reducing load at inopportune times.
For your old appliances you still pay the same on average. A fixed price contract isn't cheaper, it just smooths prices into a long-term average. And many of the changes can be done manually. On your old dishwasher or washing machine you decide when they start, and most of them even already have buttons to start with a fixed-time delay. Instead of starting them at the end of day you can just start them when the wind is strong or the sun is shining, or watch the price feed. You even get to feel smart for saving money
I agree on the popularity, but you'd absolutely see an effect even without anyone buying new appliances
The aim is net zero by 2050, lifespan of a fridge-freezer is about 10 years. Even assuming designing a system and putting it in place took 5 years, that's still enough time to have most appliances on it by 2040.
Given the current energy prices, it probably even makes sense to replace appliances sooner than their normal lifetime. My fridge-freezer is only 5 years old, but if it broke today and cost more than ~£150 to repair, I'd end up saving money by replacing it.
they are installing now smart meters with sim cards in Greece, and of course everyone started complaining, shaming the gov, claiming corruption, etc...
General population doesn't understand that fixed pricing includes an extra cost which is the risk that the electricity provider has to account for. That risk has a calculable price, which is passed down to the consumers. But because it's baked in the flat rate, nobody complains.
Smart/dynamic pricing actually benefits the consumer.
It does, but people are really bad at understanding it.
It's like how there's a substantial portion of the population that counts the best commute time ever as their commute time, and are perpetually late. "How can it take 30 minutes to get to work, one time it was only 15!" - ignoring the reality of traffic, subway delays, etc.
This will probably take a little longer for private use, but the industrial sector is already doing this. Cooling chambers being cooled down further during cheap electricity prices (or sunshine when they have their own solar) or storing heat/"cool" underground
When I was working with NREL back in 2017, they were thinking about coordinating water heater electricity use with a “smart grid.” Each device attached to the smart grid would measure the electricity spot price and would “store” energy to minimize cost. At the time the goal was to reduce peak load on the grid, but the same ingredients to maximize power use from intermittent power sources.
You have to get the energy to the appliances though, and there is the bottleneck.
It does looks like it will make some sort of sense for compute workloads to move around to be at locations near surplus energy generation. As someone else mentioned bitcoin mining (with the benefit of heat generation) could also be used, but if this practice becomes widespread the attachment of bitcoin pricing to what is in effect negative local energy prices may prove to be a structural problem with it.
I really don’t think that that’s the bottleneck. Peak demand is much higher than average demand. There is a lot of leeway in moving around domestic demand
This literally is the bottleneck which wastes the energy and is so stupidly expensive in the case of Britain (and Germany).
The issue is lots of renewable generation far from places where it is used and not enough transmission capability.
This is called curtailment and is really, really bad. Energy providers need to pay the windfarms for the energy that they (the grid operators) fail to transmit to where it is needed, and they have to pay backup generation (usually gas) at the place with the load.
It’s an issue right now because we lack the ability to steer demand. Connect a few million electric cars and heat pumps to the grid and allow the grid operators to talk to them and the issue is much less severe.
No. Steering demand will not work. Unless by steering demand you suggest forcibly moving millions of people to Scotland.
You have an intermittent power source (wind), far removed from where the energy is needed, and you do not have sufficient electric transmission capacity.
Heat pumps or EVs far removed from the source of generation will not do you any good. You need load where the energy is produced or you need more transmission capacity.
The situation GB has is that there is load, and there is enough renewable generation on the grid to meet that load, however they do not have the capability to bring the electricty to where the load is. You can lessen the demand, but the generation would not get less through that. The only benefit of that would be that you wouldn't have to spin up gas plants, but the same amount of wind energy would still be lost.
With flexible rate agreements, that's already possible, and some DIYers already are doing this - the problem is the interfacing. Heat pumps (and central heating systems in general) are notorious for being walled gardens, most freezers run on analog technology (i.e. a bi-metal strip acting as a thermostat).
I'm in the market for a heat-pump based system and I'm 100% worried about lock-in/walled gardens.
Take Google, which should have plenty of money and systems to provide long-term support, is regularly axing older products. (Of course, Google has a history of such actions, but they don't have to EOL products that should have long life-spans. Plenty of company won't really have a choice if they are facing bankruptcy, etc.)
You are correct. Some devices expose local API's, most have walled garden cloud API's
Before buying a device, it is a good idea to check if there are open source adapters for it for Home Assistant, those usually show if it can be controlled easily and preferably without cloud.
For heatpumps/heating in general - we have a after market-product here that you can install in your old "dumb" central heating system - you connect it between the outside thermometer and your boiler/pump/what have you. It then fiddles with the outside temperature readings as to trick the pump to run harder/easier depending on the electricity price (and thus in extension, towards peak production).
I used it in my previous house and it worked well! (no affiliation except as a former customer, but the product in question _I used_ is called ngenic tune [0])
It's practical enough that this is how it works now in many (most?) parts of Europe at least. Electricity at the wholesale level is priced hourly or quarter-hourly and households often elect to have a correspondingly hourly priced eletricity contract & program their appliances/ev charging/whatnot to follow the price.
We have this system in Finland and whilst I was sceptical at first, it works much better. Electricity prices are published about 24h in advance for 15m intervals (was 60m up until 2 weeks ago). You can therefore time your usage dependent on demand on the grid (which is correlated to production of course).
We've saved 100s of euros annually on our electric bill by limiting sauna, washing machine + dishwasher to low-cost hours. Sometimes it's impossible and it's days at a higher rate - but for a 2 person household it's costing us 15-20e a month (+ additional transmission costs)
> This includes switching between 'peak' and 'off-peak' meter registers as well as controlling the supply to dedicated off-peak loads such as night storage heating
1980's technology, recently switched off, I presume for internet based alternatives. The exact same principle applies, beats batteries as hot water tanks and storage heaters already exist.
Smart thermostats and water heaters can to that to some extent. Heat or cool the room a little more and run the water heater when electricity is cheap, so that they don't have to run when it is expensive. Of course, electricity price matches supply and demand.
Other options could be delayed start for large appliances like washing machines and charging electric vehicles. EVs have even been proposed as a distributed battery system to smooth out electricity use.
Already have thermostats that move based on signals from the utilities. There were some early pioneers in this stuff over 10 years ago in the bay area. They also aggregated the power to bid it, but I imagine they could aggregate to buy as well.
We have a low-tech version of something like this in South Australia: we pay the wholesale rate for electricity, which updates at 5 minute intervals. During the day when there’s oversupply of wind and solar, the rate is super low or even negative, which we take advantage of to charge an EV (and we’ll be adding a home battery soon).
The power company can integrate with car chargers and battery controllers to control all of this automatically, though we don’t bother - just check the app for the cheapest/greenest times and schedule the car to charge then.
It’s allowed us to switch to an EV without even really noticing any extra power cost for charging it.
Personally, homes and freezers should have a consistent temperature; if there's ways to store the excess heat / cold somehow that'd be neat. But for homes, the best ways to store excess energy would be batteries and electric cars, or worst case sink heat into underground storage.
The electric water heaters are a good idea, but you'd need the space for extra storage. There's existing heat exchanger systems with e.g. rooftop / sunlight water heating systems, if excess cheap energy could be used to also heat that storage you'd have something.
It would be much more effective to even out things , and trivial (engineering wise) to stop wasting the outputs of all these heat pumps by effective integration. Ie dump heat removed by ac or freezer into hot water heating, etc
I'm always highly amused when people have heated pools next to large outdoor ac units. They could probably dump all the heat from house into it the entire summer and not have a meaningful effect on the temperature
My parents had a off-peak hot water system when I was growing up. The insulated tank would fill and heat up during off-peak hours (i.e. late at night), and merely keep it warm during the rest of the day.
The downside was that once the hot water was gone, we had to wait until the next day for more. The last person to shower occasionally got a cold shower.
Good water heaters are key. Mine is 200 liters and I've experienced cold water exactly once in three decades: One day 3 guests took hour-long showers each. Normally a family of five will never experience cold water.
The one I'm getting now has two coils, one to quickly heat water at the top half, the second to heat from the bottom - they're never on at the same time. Internal heat around 75 C, mixed to cooler on the way out, and it can keep hot water for 2 weeks if disconnected from power.
This is already happening with electric car charging. However, part of the reason this can't apply here is that the UK doesn't have regional pricing. For this to work you'd need to vary people's prices depending on which pylon they're connected to.
A lot of thermostats already do that. Unfortunately these programs are not terribly popular. People see that the temperature is off and complain. Look up people talking about Nest Energy Shift (different but somewhat similar idea), most comments are quite negative.
I'm on the octopus agile tariff that has 30 minute pricing and an API to query it. Prices for tomorrow published at 4pm today. So the pricing bit is sorted. Just need to make the devices understand it now.
With phone hardware lifetime so short, would it be possible to catch-up with hardware update cycle? I guess each new version of a phone can ship with new versions of binary blob drivers. As mentioned in the announcement, reverse engineering the blobs is a huge effort, when it is done, hardware may already be out of sale and the effort would need to be repeated for new versions.
Depending on your system, passing secrets via environment can be more secure than passing secrets via a command line. Command line arguments of all running processes can be visible to other users of the system, environment variables are usually not.
Here is money printing scheme that looks to be at work:
Initial situation:
* Big corp M has X$ in cash where X is huge
Big Corp M invest X$ in AI startup O, with a provision that O needs to use most of the money to buy cloud infrastructure from M to power AI models.
End situation:
* Big corp M has X$ wort of shares in O, the value of which will rapidly grow
* Big corp M cloud division has ~X$ in extra revenue
The deal automatically turns X$ into ~2X$ in books.
Rinse and repeat with next round deals and next AI startups. The big corps are reporting increased cloud divisions revenue from AI spent, but it is their own investment money flowing back to cloud divisions.
> The deal automatically turns X$ into ~2X$ in books.
No it doesn't. The revenue it gets back is valued only at a fraction, because it's only worth its profit. Revenue != profit.
And your "the value of which will rapidly grow" is doing all the work here. That's not guaranteed. It might collapse as well.
It's not money printing at all. It's tying up cash long-term in exchange for a much smaller amount of profit short-term. Which is risky but entirely normal.
To be precise I wrote 'in books' which record also revenue, not just profits. Increasing cloud revenue is one of the things that drives big corps share price up, the missing bit is that this revenue comes from big corps themself. The growth investors mantra is that as long as revenue is increasing rapidly, they don't care much about today's profits, this is why so many unprofitable companies have crazy high valuation.
Hard to reproduce bugs often depend on an order of events or timing. Different architecture can trigger different order of execution, but this doesn't mean the bug is not in the application.
reply