Best archival method for raw footage from tapeless camcorders?
I have been considering a future purchase of an HD camcorder and have been wondering, is there a decent way to archive the footage? In the old days, it was easy. The source tape was the archive due to it being generally robust and tangible (I can store it away on a shelf, its contents were viewable after a few years, and footage isn't easily lost/damaged). Now tape is no longer an option with newer camcorders. I have come up with a few options, but none of them seem very good at the moment. I am curious what others have come up with, particularly in the professional broadcasting field.
1. Memory Cards Pros - Easy access to contents - Somewhat durable, they can survive water damage and exposure to extreme temperatures better then tape. Cons - VERY expensive - Limited storage space - Unknown long term reliability - Difficult to recover data if corrupt 2. Optical Storage (BluRay likely for HD content/DVD for SD) Pros - Relatively cheap - Easy access to contents, drives readily available - Durable, can survive most environmental damage as long as it isn't abused Cons - Unknown long term reliability (BD-R in particular) - Moderately difficult to recover data if corrupt depending on type of damage 3. Hard drives Pros - Dirt cheap at todays prices - Easy access to huge libraries on PC Cons - Portability varies - Drives can be easily damaged by dropping - Reliability is a crapshoot, some drives outright die without warning, others give early warning signs of failure. - Data recovery is generally possible as the market for it is mature. Recovery costs vary on drive damage. |
I have my personal footage + client tapes archived on file server which uses ZFS/RAIDZ1. On ZFS data is verified at read/write and Z1 copes loosing 1 drive. In the future(or even now) when drives get bigger and bigger and risk of loosing another drive while recovering from the failure of the first one calls for Z2 or more, protecting against simultaneous failure of 2 drives. Data is occasionally backed up on USB HD's in tandom with file server. This setup is pretty reliable, but not very portable - depends how much I utilize the USB drives.
|
NJR, your observations are all pretty much accurate there.
Solid-state drives have unimpressive longevity stats right now, so these are best left to temporary storage. (I'm somewhat tired of reading people complain about losing all their images on SD cards, because they treated it like a permanent storage for the camera. SD is one of the worst of the SSD formats, as far as I can see.) For optical media, single-layer DVD is pretty safe -- the safest of them all. It's the dual-layer DVD media, CD and Blu-ray that have various structural issues that cause serious concern for longevity. At only 4.38GB, of course, SL DVD isn't always the most convenient. Hard drive recovery starts at about $1,000 and goes up from there. Most drives suffer from basic mechanical failures over time, and data can often be recovered well into the 90th percentile range. Of all the available HDD configurations, a mirrored RAID is the best. I use a RAID-1 for photo archives. Other data is merely duped to single drives, as it's non-active backup. Also remember to use off-site storage. For a business, this can be anything from a second office location to an off-site storage facility. For home users, this can be the home of a parent, sibling or other relative -- maybe even your "day job" private office (if you have one), in a locked drawer. And tapes weren't really all that robust anyway. I've seen moisture ruin many tapes, be it audio cassettes, VHS, 8mm, Hi8, DV or even computer DAT. Tape is tape -- humidity ruins it, leeches into the substrate. |
These are great points. I have learned to make double copies of some important data, and also to inspect the backups every year. I think that is really important. I just read a study from Library of Congress who sampled discs from a 300,000 collection and found visible mold or bubbles on some discs, even though they tried to store them properly as a museum would. Other discs read poorly even with no visible problems.
I came here because I thougth I had an answer for a similar question, actually to archive uncompressed footage I did a test of *all* lossless codecs and found huffyuv from ffdshow has good compression and can be done real-time in capture for a modern computer. UTVideo is also very fast and great for editing work, and some of the best compression for it's speed. The famous Huffy is rather outdated now. x264 lossless is quite bad actually. |
Quote:
|
http://compression.ru/video/codec_co...s_2007_en.html
This was a professional comparison. I did also my own http://forum.doom9.org/showthread.php?p=1463689 From what I've read, x264 does work well with certain material like animation which can benefit from intra frame compression. I also found a VFW codec for x264 to let VirtualDub use that directly when capturing. Huffy (the original one) I found too slow for realtime on my old machine, and the compression isn't that great. I'm a big fan of UTVideo now. |
x264 has come a long way in 3 years (compression.ru test is from 2007). I don't know about VFW "hacks", capturing directly to x264 seems counter intuitive. It's not very edit-friendly etc. But archiving is different story.
I ran a quick test. 1 minute denoised VHS with some static in the beginning: ffdshow huffyuv: 363 442kB ffdshow FFV1: 304 812kB x264 (--preset superfast): 356 097kB (75fps) edit: UTVideo 4:2:0 (Optimize for compression): 351 367kB (all include 48/16 uncompressed audio) So yeah, with this material x264 does not have so good compression. But it was 3x faster than FFV1. Using "--preset veryslow" I managed to get 2Mbps lower bitrate (343 706kB) and 15x slower compression time. It's not propably the way to go for archiving. But wide support - outside of ffdshow - for H.264 decoding counts as a plus IMHO. |
Quote:
Quote:
x264 lossless however, has not changed. Your results practically mirror mine. x264 ultrafast, one pass lossless. Re-encode time: 1:40 Size: 498059 hmm.. doesn't seem right, I know it lost though. It can be made editable by changing the keyint(?) setting to 1. And the VFW version is not a "hack" - but I can't find the keyint setting in the VFW configuration. http://sourceforge.net/projects/x264vfw/ FFV1 compresses best with large context vs small, and AC is better than VLC - for this material (VHS cartoon). Code:
Codec Abbr Settings Size Recomp Time Ratio Speed Realtime |
You got surprisingly good ratio with UTVideo compared to my sample. I added the info to my previous post. I've been weary of UTVideo before, but it seems more mature now for candidate to intermidiate codec inbetween capture and encoding to broadcast formats. Generally I wouldn't recommend it for long term archival to people who don't know exactly what they're doing. It's not that mature yet.
|
Remember to compare apples here. My result for UTVideo 4:2:0 is much better because there are less color samples. If you scroll over my chart you will see I marked as nearly lossless. I was setting 4:2:2 as a standard for this test.
Here's a tidbit: Quote:
It was also found to be the only codec to play SD lossless realtime on a netbook (Intel Atom). |
Oops. Earlier I read your test more keenly. Now I forgot/missed the 4:2:2 vs 4:2:0 part. I know that UTVideo is fast and that's the very (only) appeal. If I would quantify my worries about the codec, in regard to recommending it to "general populace", it's new - not very widely used and thus tested, bitstream has already been changed atleast once (afaik), future support unknown (sources available which is a plus). My assesment is no doubt biased and lacks expertice in the field of lossless video coding, YMMV.
|
No, really those are good points and I can understand. Others feel the same, they want opensource and continuous development. I have good feeling for this as latest version 8.5 is Nov/10. Will it work with the Directshow of 2015 in Windows 9? Who knows. But for capture/editing, probably OK, as long as I know it's bit exact in the software I use.
|
Site design, images and content © 2002-2024 The Digital FAQ, www.digitalFAQ.com
Forum Software by vBulletin · Copyright © 2024 Jelsoft Enterprises Ltd.