SD-card corruption is a real problem on RPi (see OpenHAB forums). Supposedly the SD-card driver is not the greatest which together with the system's design makes writes fail silently. This cost is half a day once, since we did not understand what was going on with a config not being persisted over reboots.
Mounting root read only and only writing explicitly through the "ro/re" scripts helps a lot and should be the default IMO.
The biggest issue some people have with RPis though, is their hackish software, e.g. the kernel. This has become better apparently, but it always was a shame that you couldn't just run a generic kernel/distro.
I had good experiences with the BeagleBoneBlack, they come with eMMC and quite a few smart additions (e.g. a power button). Twice as expensive though.
We're using hundreds of raspberry's as digital signage devices and before deploying them I checked a few of them and found out that sometimes the source files of the app were empty.
After a lot of headscratching and googling we boiled it down to a write delay to the SD card.
If you do echo "test" > test.txt and power off the device like 5 sec. after, and power it on again, there was a file called test.txt with 0 bytes.
I'm still baffled how something like this can be a common problem especially since raspberry pi usage is so widespread
I operate a digital signage service (https://info-beamer.com). It's using a custom built Linux OS that is highly optimized to deal with file system corruption and unexpected reboots:
* The OS is is always read-only. While this doesn't prevent all possible corruption scenarios, it's pretty close and you can be quite sure the system always boots.
* System updates are implemented as A/B. So even during updates, the OS the system booted from isn't touched.
* All content data is on a third partition and together with the OS files is constantly checked for indications of corruption.
* If the OS detects any bit error on the OS files, it is reinstalled (using the A/B mechanism above)
* If there's a bit error on any of the content files, they are deleted and downloaded again
* If the content filesystem is broken beyond repair the system reformats everything and restores everything automatically by redownloading
* Various watchdogs and checks constantly monitor system health and notify you on the dashboard.
The service has multi-million hours of service by now and the only case of filesystem corruption was due to a really crappy SD card. When done properly, the Pi can be reliable. That's also the reason why there is no way to properly shut down the system. Just unplugging power shouldn't be an unexpected circumstance but something you optimize for. Starting with a normal Raspbian and expecting that to work is a bad idea.
I have a use-case in mind where the power might disappear suddenly and often. Is there a different SBC board and OS which is optimal for this use-case?
You might want to look at some of the 'Pi UPS' systems which have been developed. Once they indicate the power has failed you can start a graceful shut down. One example can be found here:
In my opinion it seems the Pi was too good at fast and cheap prototyping and people started shipping products based on it, overlooking the fact that the hobbyist and unreliable nature of the board wouldn't necessarily scale as hoped.
It always looked to me like the Pi was supposed to be used some steps before hardware integration and shipping.
I am glad products are built on it but I was a bit worried when I had to evaluate a Pi based solution for digital signaling last year. Maintenance was a key factor in the proposals.
Yes, in fact you'll find lots of source code out there that will write test.txt.tmp and then close and rename it to test.txt, exactly because this is an issue in all filesystems. From the sounds of it the time window during which RPi is vulnerable is longer than usual. A careful choice of filesystem journaling/integrity flags would probably help.
The fact that there's a 0-byte file instead of a completely broken file system is already the result of a journal. You can reliably observe this with ext4.
It was an unsafe shutdown, meaning I just pulled the plug. But it was at least several seconds after saving it. I've never seen behaviour like this before.
Most sd cards are formatted fat32, and do not have a virtual filesystem cache and write scheduler buffering work in ram for them.
Also raspis do NOT like big fast SD cards with tight tolerances. I've had the best luck with old class 4 cards. Class 10 cards don't last nearly as long with regular power cycling.
You can force a partition to be mounted "synchronously", where writes aren't buffered in RAM and then flushed later -- every write blocks until it is written to disk.
The options are the third column in your /etc/fstab file. For example, if you had "errors=remount-ro", you'd change it to "errors=remount-ro,sync".
Did you ever run linux with ext2 (not ext3 or ext4) on a desktop computer in the early days? It used to fail spectacularly on unclean shutdowns. IIRC linux used to delay/buffer writes up to a minute or so. (Maybe it still does?)
I would never SD card boot a Pi in a production environment unless your software can handle a read only file system, the card corruption issues are so widely documented at this point. Thankfully USB boot volumes are supported now.
Except that you don't need to buy an SD card. Or a special 2.5A power supply. And you can access the it solely over USB so you don't need to hook up an external keyboard, mouse, and HDMI to get it running headless. And you can actually get the chips and their datasheets for the BBB. etc.
These are great specs indeed. I just wish it wasn't so expensive, since it does take it from "something I can use" to "something I'll put in a gilded cage and only use when that very demanding problem arrives", which is never.
Mounting root read only and only writing explicitly through the "ro/re" scripts helps a lot and should be the default IMO.
The biggest issue some people have with RPis though, is their hackish software, e.g. the kernel. This has become better apparently, but it always was a shame that you couldn't just run a generic kernel/distro.
I had good experiences with the BeagleBoneBlack, they come with eMMC and quite a few smart additions (e.g. a power button). Twice as expensive though.