Detecting and resolving bitrot on consumer hardware

I just finished rebuilding my home server, and I got to thinking, are there any data integrity scenarios I'm missing? All the files I care about are on the server, with Windows Offline file mirrors to machines that need them. The server drives are mirrored. The server backup drive is mirrored.

I was recently reading at work about the non-zero occurence of memory soft errors - hilariously, it does happen on hosted compute solutions like Azure or EC2; they apparently use commodity hardware, not the expensive ECC memory stuff, so it's surprisingly common. Similarly, the TCP checksum is rather weak; I recall Blizzard discussing how a terrifying number of users couldn't download World of Warcraft and get it to work just because something kept corrupting packets.

Hrm, what about hard drive bit rot? I haven't thought about that in years, but I remember seeing it on hard disks multiple times in the 1990s, where files would just mysteriously get corrupted though no one had touched them. Alternatively it could be firmware bugs or something, but it's the same effect.

I'm sure there's high end server solutions, but what do you do for this on a Windows commodity pc? Is there some software out there that checksums files as an automated detection solution?

Nowadays, bitrot is more usually called "unrecoverable read error". Apropos of your fear, here's a recent article I came across describing how the sort of bit-rot you're worried about can still ruin the lives of people on high end servers.

As for low end Windows solutions to this problem, I got nothin'. The files I personally keep like this are stored with par2 archives and are manually copied to an offline disaster recovery USB drive every 6 months or so. After I copy the files, I check them with par2 which finds problems about half the time, so bit rot is still very much a real thing.

For low end Unix solutions, I would be pretty tempted to use zfs via your Unix of choice (OpenSolaris, FreeBSD or Ubuntu), and think about migrating to btrfs in five years or so. zfs is perfect for your fears because it can be instructed to scrub arrays for precisely these errors and doesn't do physical device level failing when read errors are detected, which maximises your chances of getting good data off.

i was also going to link that article. it isn't bad and has some good information, though it doesn't really get into a lot of nitty-gritty details.

as for the zfs / *n*x combo, that is probably the best long-term solution. btrfs still has quite a long way to go before it comes anywhere near the trust that zfs receives- it's just too new and untested. i say this despite having used btrfs for the last 2 years on my Ubuntu storage 'server' box. my personal plan is to, at some point, build a box from a low-end Xeon or Opteron -class cpu and then use either FreeBSD/zfs or SciLinux/zfs and be done with it. the big gotcha with zfs is that it really likes a lot of memory. both zfs & btrfs are going to have that when you start turning options on like real-time compression & deduplication as well (though only zfs is stable in these regards, imo).

I remember reading something about how we're now at the point where statistically you'd expect there to be a few bit errors when copying one large hard drive to another. Whenever I'm copying a huge amount of data like that, I do an 'md5sum' against the source files to generate checksums for all of them, and then after the copy I rerun it in '--check' mode on the new drive, to verify that they haven't been altered. I also have a script that runs over my stored files and checks for corrupt files by fairly simple tests (e.g., if it's a zip file, does 'unzip' handle it without errors).

That's about it though; I don't specifically do anything to proactively protect them besides my usual backup process (which doesn't really help if it just goes ahead and backs up a corrupted file, of course), though the par2 idea sounds interesting.

The hell files are you storing where a bit here or there will cause a problem? I'm genuinely curious; shouldn't most media files be fairly robust to a stray bit here or there going bad?

This is relevant to my interests. *scratches chin* Does it suffice to simply run md5 checksums against every file every few months? The article talks about running into errors because there are volumes that aren't accessed for years being the main thing...

Regularly reading the files in question (as md5sum would do) doesn't reduce the problem incidence, it simply lets you detect the problem incidence sooner. Whether this is helpful or not depends on your problem recovery mechanism. If it's a RAID rebuild, probably better to rebuild when there is only a problem on one drive. If it's a par2 recovery, you don't really care if n% of the drive fucks up, as long as the filesystem is readable and you have somewhere else to write the good data out.

The vast bulk of my shit it'd only be a minor annoyance, but if my TrueCrypt encrypted list of passwords and accounts dies it's going to seriously fucking suck. I'm not all that upset if my mp3s get corrupted, by contrast.

Once you get up past 40k feet, things get worse.

Well, let's design the solution.

Knowing it's valid
The only way to know that a file is still correct is either using it, visually inspecting it, or the like. A sector could go insane a millisecond after its read, but it was still good when you read it. Therefore, the only checksum that can be trusted is one you create before you create before you open the file, verify it, and then don't change it. I'm not even going to try worrying about demand paging fucking with this; nope, it's all or nothing. Backup per your normal system.

Verifying it's valid
To periodically verify it you just recalculate the checksum and compare to the saved checksum. If it fails, one or the other is wrong; diff and restore from your backups. Double failure scenario is no different than your drive dying entirely and your offsite too at the same time.

Handling changes
Calculate a checksum after writing, verify the file still looks ok. Because very applications just blit memory to disk you can't automate this by checksuming in memory and then comparing to a checksum of the written copy. If it fails, restore the last version from backup, which is the last revision.

Given that, what's the actual practical solution? Near as I can tell:

For full backups, it's easy, just checksum every file beforehand and save that too. Flag any files with changed checksums but unchanged modification dates (yes, I know this is not perfect in general, oh well).

For a full backup and checksum + nightly incremental backup and checksum system it's more complicated:
1. You changed it, it's fine. Incremental backup and checksum will find this.
2. You didn't change it, it's fine. Incremental backup and checksum will find this.
3. You changed it, it's broken. Incremental backup and checksum will find this.
4. You didn't change it, it's broken. Incremental backup and checksum will not find this.

.....which implies that you should checksum and flag everything on every backup, but only backup the changes. Which I guess makes sense. The problem is that it takes a long, long time to read terabytes of data off a non-SSD drive; maybe run it weekly? The goal is to catch silent failures quickly.

Poking around for tools:

Checksum for Windows: full command line and windowed support for all this.
ZFS is the nuclear-powered aircraft carrier that solves everything, but it's not supported on Windows beyond some read-only project. I guess you could theoretically run it in a VM, but ugh.

it's like you just designed zfs! srsly, please look at: http://en.wikipedia.org/wiki/ZFS

that said: yes, back up data which you deem to be important enough to have backups of! none of this fancy-schmancy bullshit will save your ass when your entire disk array fails for no goddamned reason and ends up with several 2+TB disks failed when JUST MINUTES PREVIOUS everything was working just fine. and if none of this shit will save even the most failsafe disk array environment, think of how great it'll be with your puny 2 or 3 drives using a filesystem whose design is horribly inferior.

catching the silent failures is a bitch. that's why the big boys (zfs) do disk scrubbing (periodically reading the disk, comparing the content to the stored checksums).

it isn't practical for most people to build a kick-ass home fileserver / disk array. if you have large storage capacity needs (>8TB) at home, you are likely wanting to look into a NAS storage box (Drobo, Synology,etc), storage server of some sort, or "roll your own" (my favorite and one of a few current projects).

as for backing these huge things up, it really is going to depend on how valuable the data is. if you are just putting a "small" (<100GB) amount of truly critical data on these things, then do some kind of backup to a cloud provider (or 2 or 3 or..). media (music, movies) ought not be "critical" but whatevs. a true backup regimen for large amounts of data really depends a lot on how much money you are willing to spend. none of the solutions are speedy in terms of what any reasonable person would usually consider, but they get the data where it needs to be.

anyways, back to the matter at hand: your existing storage system sucks for bitrot and there isn't an easy way to resolve that.

P.S. this is what i currently have for 'generic storage' on my home server:

Code (text):

1. sd 6:0:0:0: [sdb] 35156247552 512-byte logical blocks: (17.9 TB/16.3 TiB)

Honestly the failure scenario I can't solve right now with just backups is "silently corrupted, but I don't look at it for a year and it keeps merrily backing up garbage." Which requires the full ZFS stack.

I should probably segment out my convenience restore-from-bare-metal when hard drive fails scenario from the "this actually needs to be reliable" scenario.

If all you care about this much is a single 1mb truecrypt volume you're seriously overengineering here.

That's just one end of the spectrum, the rest would be nice to keep working too.

Then scrub every week. Or even every month. Problem solved?

While ZFS is great and I wouldn't want to run my servers without it, I just had a machine fail on me - all the disks in the mirror showed errors so it's probably a bad controller or motherboard. Scrub repaired tons of damage but I got a bunch of the dreaded "Permanent errors have been detected in the following files". So yeah. You are always going to lose data, even if it's just the bit between the last backup and the failure point.

Print it. I keep a printout of all passwords for just that reason. Hide it with the chickens if you don't trust your wife.

I do and it's in the safe deposit box. The scanned-in duplicates of all important documents are too damn big to fit in the box though, so again the corruption problem.

Ignoring security issues, that requires hand fiddling. Chores that require crapping around with stuff like that are exasperating.

You're right they probably do solve this; wonder if I can poke around internally and see what Skydrive does.

You can script most of it, and public key cryptography is pretty solid.

What I ended up doing was using Md5Deep daily to hash the files I really care about.

The first file sets up the directories and timestamp, the second actually executes it. Annoyingly there appear to be zero utilities that will handle intentionally new, deleted, or modified files by design; you have to spit all the relevant file lists and process them separately, so I just compare the hash files in a diff tool occasionally; it's usually obvious if a file has started throwing up.

I really should get around to learning powershell one of these days instead of writing these batch file abominations.

REM BEGIN FIRST BATCH FILE HashDirectoryXYANDZ.cmd
REM @echo off
@setlocal
@set ARGS= %*
for /F "tokens=2-4 delims=/ " %%i in ('date /t') do set yyyymmdd=%%k%%i%%j
for /F "tokens=1-3 delims=: " %%l in ('echo %TIME%') do set hhmmss=%%l%%m%%n

set HashFileStorageDirectory=C:\Hashes

set HashFilePrepend=Directory1DirectoryA
set HashFolder=c:\Directory1\DirectoryA
@call "%~dp0CreateHash.cmd" "%HashFileStorageDirectory%\%HashFilePrepend%%yyyymmdd%%hhmmss%.hash" "%HashFolder%"

set HashFilePrepend=Directory2DirectoryB
set HashFolder=C:\Directory2\DirectoryB
@call "%~dp0CreateHash.cmd" "%HashFileStorageDirectory%\%HashFilePrepend%%yyyymmdd%%hhmmss%.hash" "%HashFolder%"
REM END FIRST BATCH FILE HashDirectoryXYANDZ.cmd

REM BEGIN SECOND BATCH FILE CreateHash.cmd
@echo on
@setlocal
set NewHashFileName=%1%
set DirectoryToHash=%2%

"%~dp0sha1deep64.exe" -r -o f -j0 %DirectoryToHash% > %NewHashFileName%
REM END SECOND BATCH FILE CreateHash.cmd

I think we need to whistle up a *nix beard to put you out of your misery on that.

nerd.

It's even more damn work to install mostly-but-not-always compatible unix utilities!

Takes about 2 minutes to install Cygwin. Or just do your development on a real OS.

When I run a Rise of Flight server for my friends on my web server, I use cygwin on my Windows gaming desktop for the ssh session to the Linux server, which runs wine to run the RoF server.

rmdir /s C:\Cygwin

Like what? I'm poking around my registry, I can't find anything that references anything under the cygwin directory that's of any significance. Maybe if you install a bunch of services or something, I guess, but a typical install seems absurdly clean.

*caveman voice* Fish like bash. Bash good shell, shell not break.

Low level registry hooks huh?

Do you also regularly run a registry cleaner that sounds klaxons when it finds a registry key's value is a path that points to a nonexistent file?