Comments on Fluffytek Art Blog: Improbable Drive Problems

"Improbable" only if the drives were independent (...

2008-03-19T04:27:00.000Z

"Improbable" only if the drives were independent (in the statistical sense of the word), which they weren't. For all we know it could have been a stray cosmic ray hitting a controller in your machine which caused the corruption.

I too have been struggling with backups, although I have only a quarter of the data that you do. Two machines, though, and thousands of files and versions of said files, and so on.

I've been thinking of how to design a backup methodology for a while now. The way I see it, it needs to be:

- Simple and open. At the most it uses some scripts and some open source tools to manage things.
- Tiered. Not all data is equally important.
- Robust. It should be easy to attain as much reliability as you want.

With that in mind I should mention the following:

- RAID is not a backup.
- The only true backup is WORM: write once, read many. Like DVDs. Once you start overwriting data and changing filesystem structures you no longer have a true backup.

My very rough plan is:

- Dump my new photos onto my main machine, in the "least important" dir.
- Create a SHA1/MD5 for each photo. This will detect corruptions later on.
- Work on the photos and eventually separate them into different dirs, like "least", "moderate", and "very important".
- "very important" holds the smallest amount of data. Maybe a gig.
- "moderate" holds 3.7 GB.
- "least" holds an unlimited amount.
- The trick is to backup the different dirs to different media. Every 5 days "very" is backed up to Amazon S3 (online). Every 15 days "moderate" + "very" is burned to a DVD. Every 30 days "least" + "moderate" + "very" are backed up to an external HD.
- Amazon S3 can hold as many copies of "very" as you want. There's all those DVDs too.
- The external HDs are more complicated. I'd have several HDs and rotate them. First, scan all the files on the HD and verify none are corrupted. Then sync your main computer to the HD.

Yeah, this system sounds complicated, but I think treating all data as equally important is a mistake.

Now I just need to follow my own advice.

I understand...as I sit here at my studio computer...

2008-03-17T19:20:00.000Z

I understand...as I sit here at my studio computer with 5 external drives, plus two in the computer. I have all my photos on at least 2 hard drives, but I don't have any automated system for backups. I really hope to get this better organized this year.

But, I'm also working on what I consider my most important backup. I'm making prints of all my favorite photographs and putting them into archival boxes. One box for each of my children, one for me and my wife, and one for the imaginary photo history grad student who will write their thesis on me 50 years after I die.

Or, one could just use film.

2008-03-17T13:50:00.000Z

Or, one could just use film.

I'd like to help.But I have a 1kb mind in a multi ...

2008-03-16T19:44:00.000Z

I'd like to help.

But I have a 1kb mind in a multi TB world.

D.L. Wood

Does it make you yearn for the olden days of negat...

2008-03-16T17:15:00.000Z

Does it make you yearn for the olden days of negatives and chromes?

Me neither.

Yikes,That sounds like a server system problem. ei...

2008-03-15T18:48:00.000Z

Yikes,

That sounds like a server system problem. either a spike got thru the UPS or something went wrong in the server software..Had a UPS go bad on me (the one on my 1 TB server) during a generator test but didn't lose any data. If you had a power & UPS failure during a write, it MIGHT have the effect you described, I'm not a IT type so maybe someone who is can comment.

I'm using the photo storage system Kevin Ames published last year, keeping my main files on the server on-line but 2 complete backup DVD's, one of them in a safe deposit box offsite. I anticipate having to copy, or pay to have someone copy, all the files to new storage technology in a few years (probably before I have any problems with the disks) but am feeling pretty secure right now.

Good luck and my best to Lin.