Citizens in the developed world have been heavily surveilled for decades. All of them. And the penetration/rate of increase of surveillance has been increasing rapidly.
There are plenty of downsides to that, and I don't agree or think it's beneficial, on balance. But "citizens require a lack of surveillance" is far from the present-day truth, or even remotely practical as an aspiration.
I think the devil's advocate/libertarian reply would be roughly: its efficient to let consumers prefer venues with different pricing schemes. If dynamic pricing is bad, then competitors will differentiate by not doing it, and price out the ones doing dynamic pricing.
To be clear, I don't believe that (or even the premise that "making capitalism work" is a good social goal--some elements of capitalist economies are socially beneficial, but adopting it as an ideology rather than piecemeal is not). I think your point is generally correct: if your goal is an efficient free market, then price transparency is important. But that's just my hunch as to what the counterargument would be.
The core issue with pure libertarian ideas is that they ignore "shoe leather costs" and the role they play in making consumers "irrational", which the success of a libertarian society depends on.
In this case, plenty of places in the US only have one reasonably close grocery store.
Yes, but the incentives created by that system lead to insurance adjudicators operating with extreme adversariality towards the insured. Add to that the extreme inelasticity of demand for insured products (e.g. healthcare, or getting access to a car to use to commute after one is totaled), regulatory capture of insured products/services by insurers, and time, and you get pretty toxic systems wherein insurers exert upwards price pressure without significant checks.
Not as well as they can reason (or others can google) something as standardized as kubernetes. There’s just less context (in both senses of the term) needed to understand something running on a common substrate versus something bespoke, even if the bespoke thing is itself comprised of standardized parts.
For a project set up by a qualified engineer, there would be little difference to the end user in practice. The LLM would work out a solution with a negligible difference in speed. Maybe debugging would also be faster for the LLM without the abstraction layers and low level access?
Shit just gets really weird when your network isn’t split for k8s in an equivalent way to what GCP/AWS expect. Like, if you have other services running on the nodes that you want things inside k8s to talk to, or if the nodes are in a flat subnet with other stuff in it, things get annoying. Those are worst practices for a reason, but pretty common in environments with home rolled k8s clusters.
That is indeed a weirdly cursed requirement. Why? Black box of legacy stuff? A system that was never designed to be run in multiple does so if all the nodes think they’re the same machine? Defeating a license restriction?
This is a need it fails at miserably. k8s reminds me of the raid recentralization anti pattern problem where you fix a hardware failure that never occurs in exchange for knowing simple higher level mistakes or security problems will tank something now too large to fail again.
After peeking the source, a few possible areas of improvement:
- You can use `fstat` and keep a file handle around, likely further improving performance (well, reducing the performance hit to other users of the filesystem by not resolving vfs nodes). If you do this, you'll have to check for file deletions.
- If you do stick with stat(2), it might be a good idea to track the inode number from the stat result in addition to the time,size tuple. That handles the "t,s = 1,2; honker gets SIGSTOPped/CRIU'd; database file replaced; honker started again", as well as renameat/symlink-swap fiddling. Changing inode probably should just trigger a crash.
- Also check the device number from the stat call. It sounds fringe, but the number of weird hellbugs I've dealt with in my career caused by code continually interacting with a file at the same time as something else mounted an equivalent path "over" the directory the file was originally in is nonzero.
- It's been a few years since I fought with this, but aren't there edge cases here if the system clock goes backwards? IIRC the inode timestamp isn't monotonic--right? There are various strategies for detecting clock adjustment, of various reliability, that you could use here, if so. Just checking if the mtime-vs-system-clock diff is negative is a start.
That covers the more common of the "vanishingly uncommon but I've still seen 'em" cases related to file modification detection. Whether you choose to cope with people messing with the file via utime(2) is up to you (past a point, it feels like coping with malicious misuse rather than edge cases). But since your code runs in a loop, you're well-positioned to do that (and detect drift/manipulations of the system clock): track a monotonic clock and use it to approximate the elapsed wall time between honker poller ticks (say it fast with an accent, and you get https://www.bbc.com/news/world-latin-america-11465127); if the timestamp reported by (f)stat(2) ever doesn't advance at the same rate, fall back to checksumming the file, or crashing or something. But this is well into the realm of abject paranoia by now.
It's been a decade or so since I worked in this area, so some of that knowledge is likely stale; you probably know a lot more than I do after developing this library even before considering how out-of-date my knowledge might be. When I worked on this stuff, I remember that statx(2) was going to solve all the problems any day now, and then didn't. More relevant, I also remember that the lsyncd (https://github.com/lsyncd/lsyncd) and watchman (https://github.com/facebook/watchman) codebases were really good sources of "what didn't I think of" information in this area.
But seriously, again, nice work! Those are nitpicks; this is awesome as-is!
I actually looked at fstat, but the "check for deletions" piece, given I'm polling at 1kHZ, was the reason I decided not to use it. Older hardware actually made this a big issue but it's fast enough now I decided it wasn't a problem.
I'll ignore the malicious ones bc [out of scope declaration]. Object paranoia is an artifact of build trama and I respect that lmao.
I've just looked into the device number and system clock issues. I think what i'll end up doing is actually a combo of ncruces's above comment and your feedback: a 1kHZ data_version and a 10HZ stat() with version check. This gets around syscall load, avoid clock issues, avoids the WAL truncation issues that others have mentioned, and is both lighter weight and less bugabooable than my previous design.
One clarification: by "check for deletions" I didn't mean that you need to read back through the filesystem; you can check for deletions for free using fstat(2)'s result. The number of hard links to a file descriptor's underlying description returned by fstat includes the "existential" hard link of the file itself, and drops to zero when the file's deleted and the open handle is an orphan:
import os
import time
from threading import Thread, Event
f = '/tmp/foo.test'
ev = Event()
Thread(target=lambda: ev.wait() and os.unlink(f), daemon=True).start()
with open(f, 'w+') as fh:
print("before delete:", os.fstat(fh.fileno()).st_nlink)
ev.set()
time.sleep(1)
print("after delete:", os.fstat(fh.fileno()).st_nlink)
reply