Best archival method for raw footage from tapeless camcorders?

[NJRoadfan]

I have been considering a future purchase of an HD camcorder and have been wondering, is there a decent way to archive the footage? In the old days, it was easy. The source tape was the archive due to it being generally robust and tangible (I can store it away on a shelf, its contents were viewable after a few years, and footage isn't easily lost/damaged). Now tape is no longer an option with newer camcorders. I have come up with a few options, but none of them seem very good at the moment. I am curious what others have come up with, particularly in the professional broadcasting field.

1. Memory Cards
Pros
- Easy access to contents
- Somewhat durable, they can survive water damage and exposure to extreme temperatures better then tape.
Cons
- VERY expensive
- Limited storage space
- Unknown long term reliability
- Difficult to recover data if corrupt

2. Optical Storage (BluRay likely for HD content/DVD for SD)
Pros
- Relatively cheap
- Easy access to contents, drives readily available
- Durable, can survive most environmental damage as long as it isn't abused
Cons
- Unknown long term reliability (BD-R in particular)
- Moderately difficult to recover data if corrupt depending on type of damage

3. Hard drives
Pros
- Dirt cheap at todays prices
- Easy access to huge libraries on PC
Cons
- Portability varies
- Drives can be easily damaged by dropping
- Reliability is a crapshoot, some drives outright die without warning, others give early warning signs of failure.
- Data recovery is generally possible as the market for it is mature. Recovery costs vary on drive damage.

[juhok]

I have my personal footage + client tapes archived on file server which uses ZFS/RAIDZ1. On ZFS data is verified at read/write and Z1 copes loosing 1 drive. In the future(or even now) when drives get bigger and bigger and risk of loosing another drive while recovering from the failure of the first one calls for Z2 or more, protecting against simultaneous failure of 2 drives. Data is occasionally backed up on USB HD's in tandom with file server. This setup is pretty reliable, but not very portable - depends how much I utilize the USB drives.

[admin]

NJR, your observations are all pretty much accurate there.

Solid-state drives have unimpressive longevity stats right now, so these are best left to temporary storage. (I'm somewhat tired of reading people complain about losing all their images on SD cards, because they treated it like a permanent storage for the camera. SD is one of the worst of the SSD formats, as far as I can see.)

For optical media, single-layer DVD is pretty safe -- the safest of them all. It's the dual-layer DVD media, CD and Blu-ray that have various structural issues that cause serious concern for longevity. At only 4.38GB, of course, SL DVD isn't always the most convenient.

Hard drive recovery starts at about $1,000 and goes up from there. Most drives suffer from basic mechanical failures over time, and data can often be recovered well into the 90th percentile range. Of all the available HDD configurations, a mirrored RAID is the best. I use a RAID-1 for photo archives. Other data is merely duped to single drives, as it's non-active backup.

Also remember to use off-site storage. For a business, this can be anything from a second office location to an off-site storage facility. For home users, this can be the home of a parent, sibling or other relative -- maybe even your "day job" private office (if you have one), in a locked drawer.

And tapes weren't really all that robust anyway. I've seen moisture ruin many tapes, be it audio cassettes, VHS, 8mm, Hi8, DV or even computer DAT. Tape is tape -- humidity ruins it, leeches into the substrate.

[jmac698]

These are great points. I have learned to make double copies of some important data, and also to inspect the backups every year. I think that is really important. I just read a study from Library of Congress who sampled discs from a 300,000 collection and found visible mold or bubbles on some discs, even though they tried to store them properly as a museum would. Other discs read poorly even with no visible problems.

I came here because I thougth I had an answer for a similar question, actually to archive uncompressed footage I did a test of *all* lossless codecs and found huffyuv from ffdshow has good compression and can be done real-time in capture for a modern computer. UTVideo is also very fast and great for editing work, and some of the best compression for it's speed. The famous Huffy is rather outdated now. x264 lossless is quite bad actually.

[juhok]

Quote:

Originally Posted by jmac698

x264 lossless is quite bad actually.

How so? With the material I've tried, it beats ffdshow-huffyuv and FFV1 by fair margin (and with decent settings it's even realtime in modern computers). Capture/intermediate is different than archival tho. I capture huffyuv myself and if it needs to be archived then re-encode with better compression.

[jmac698]

http://compression.ru/video/codec_co...s_2007_en.html
This was a professional comparison.
I did also my own
http://forum.doom9.org/showthread.php?p=1463689

From what I've read, x264 does work well with certain material like animation which can benefit from intra frame compression.

I also found a VFW codec for x264 to let VirtualDub use that directly when capturing.
Huffy (the original one) I found too slow for realtime on my old machine, and the compression isn't that great. I'm a big fan of UTVideo now.

[juhok]

x264 has come a long way in 3 years (compression.ru test is from 2007). I don't know about VFW "hacks", capturing directly to x264 seems counter intuitive. It's not very edit-friendly etc. But archiving is different story.

I ran a quick test. 1 minute denoised VHS with some static in the beginning:

ffdshow huffyuv: 363 442kB
ffdshow FFV1: 304 812kB
x264 (--preset superfast): 356 097kB (75fps)
edit: UTVideo 4:2:0 (Optimize for compression): 351 367kB
(all include 48/16 uncompressed audio)

So yeah, with this material x264 does not have so good compression. But it was 3x faster than FFV1. Using "--preset veryslow" I managed to get 2Mbps lower bitrate (343 706kB) and 15x slower compression time. It's not propably the way to go for archiving. But wide support - outside of ffdshow - for H.264 decoding counts as a plus IMHO.

[jmac698]

Quote:

x264 has come a long way in 3 years

Yes, that's exactly what I thought! It has, for lossy compression - no doubt x264 is the best quality codec compared to even professional ones. A test at compression.ru showed this a few months ago.

Quote:

But wide support - outside of ffdshow - for H.264 decoding counts as a plus IMHO.

What support? What electronic devices support lossless x264? I think ffdshow covers any computer application universally. I'm not aware of any lossless hardware devices (I'm sure there's some pro ones, redcode raw etc.).
x264 lossless however, has not changed. Your results practically mirror mine.

x264 ultrafast, one pass lossless.
Re-encode time: 1:40
Size: 498059 hmm.. doesn't seem right, I know it lost though.


It can be made editable by changing the keyint(?) setting to 1. And the VFW version is not a "hack" - but I can't find the keyint setting in the VFW configuration.
http://sourceforge.net/projects/x264vfw/

FFV1 compresses best with large context vs small, and AC is better than VLC - for this material (VHS cartoon).

Codec	Abbr Settings	Size	Recomp Time	Ratio	Speed	Realtime
[1][a]	Alpary realtime	791523	01:01:00	1.54	1.20	Yes
[1][b]	Alpary max	668964	06:36:00	1.82	7.76	No
[2][a]	Fastcodec	689627	01:20:00	1.76	1.57	Yes
[3][a]	Huffy Predict	658471	00:56:00	1.85	1.10	Yes
[3][b]	Huffy gradient	677685	01:00:00	1.79	1.18	Yes
[3][c]	Huffy Left	660053	00:54:00	1.84	1.06	Yes
[4][a]	Lagarith	623230	01:32:00	1.95	1.80	No
[5][a]	Mindvid		651051	01:18:00	1.87	1.53	No
[6][a]	MSU Speed	624649	07:38:00	1.95	8.98	No
[7][a]	Toponoky Speed	981436	04:57:00	1.24	5.82	No
[8][a]	Utvideo 422	628689	00:59:00	1.93	1.16	Yes
[8][b]	Utvideo 420	449833			2.70		Yes *nearly lossless (YV12 only)
[9][a]	Xvid L0 q=2	174589	03:36:00	6.96	4.24	No *Not lossless
[10][a]	FFV1 AC-Lrg	529714	03:58:00	2.29	4.67	No
[10][b]	FFV1 AC-Sm	573909			2.12		No
[10][c]	FFV1 VLC-Lrg	541541			2.24		No
[10][d]	FFV1 VLC-Sm	578978			2.10		No
[11][a]	FFHuf Left	617828	01:00:00	1.97	1.18	Yes
[12][a]	LJPG		504842	01:36:00	2.41	1.88	No *nearly lossless (YV12 only)
[13][a]	Uncomp		1215669	00:51:00	1.00	1.00	Yes *estimate from 2/3 24bit avi
[14][a] MLC Speed       574596  00:01:51        2.12    2.12    No

The huffyuv in ffdshow is better than the original huffy standalone. The adaptive Huffman checkmark makes an improvement.

[juhok]

You got surprisingly good ratio with UTVideo compared to my sample. I added the info to my previous post. I've been weary of UTVideo before, but it seems more mature now for candidate to intermidiate codec inbetween capture and encoding to broadcast formats. Generally I wouldn't recommend it for long term archival to people who don't know exactly what they're doing. It's not that mature yet.

[jmac698]

Remember to compare apples here. My result for UTVideo 4:2:0 is much better because there are less color samples. If you scroll over my chart you will see I marked as nearly lossless. I was setting 4:2:2 as a standard for this test.
Here's a tidbit:

Quote:

Using WMPlayer, a 1080p/25fps video (using 'fast decode' encoding) uses about 18-20% of my 3.4Ghz Intel quad (Q6600), so easily real-time playback without any glitches.

If you need to know it's bug free, I could do some work to qualify it. I would like to use it myself so it's no problem. It's good for editing and capturing because it's keyframe of 1 and incredibly fast, leaving more time for filtering.

It was also found to be the only codec to play SD lossless realtime on a netbook (Intel Atom).

[juhok]

Oops. Earlier I read your test more keenly. Now I forgot/missed the 4:2:2 vs 4:2:0 part. I know that UTVideo is fast and that's the very (only) appeal. If I would quantify my worries about the codec, in regard to recommending it to "general populace", it's new - not very widely used and thus tested, bitstream has already been changed atleast once (afaik), future support unknown (sources available which is a plus). My assesment is no doubt biased and lacks expertice in the field of lossless video coding, YMMV.

[jmac698]

No, really those are good points and I can understand. Others feel the same, they want opensource and continuous development. I have good feeling for this as latest version 8.5 is Nov/10. Will it work with the Directshow of 2015 in Windows 9? Who knows. But for capture/editing, probably OK, as long as I know it's bit exact in the software I use.