Go Back    Forum > Digital Video > Videography: Cameras, TVs and Players

Reply
 
LinkBack Thread Tools
  #1  
12-26-2010, 11:40 AM
NJRoadfan NJRoadfan is offline
Premium Member
 
Join Date: Sep 2010
Posts: 1,155
Thanked 357 Times in 293 Posts
I have been considering a future purchase of an HD camcorder and have been wondering, is there a decent way to archive the footage? In the old days, it was easy. The source tape was the archive due to it being generally robust and tangible (I can store it away on a shelf, its contents were viewable after a few years, and footage isn't easily lost/damaged). Now tape is no longer an option with newer camcorders. I have come up with a few options, but none of them seem very good at the moment. I am curious what others have come up with, particularly in the professional broadcasting field.

1. Memory Cards
Pros
- Easy access to contents
- Somewhat durable, they can survive water damage and exposure to extreme temperatures better then tape.
Cons
- VERY expensive
- Limited storage space
- Unknown long term reliability
- Difficult to recover data if corrupt

2. Optical Storage (BluRay likely for HD content/DVD for SD)

Pros
- Relatively cheap
- Easy access to contents, drives readily available
- Durable, can survive most environmental damage as long as it isn't abused
Cons
- Unknown long term reliability (BD-R in particular)
- Moderately difficult to recover data if corrupt depending on type of damage

3. Hard drives

Pros
- Dirt cheap at todays prices
- Easy access to huge libraries on PC
Cons
- Portability varies
- Drives can be easily damaged by dropping
- Reliability is a crapshoot, some drives outright die without warning, others give early warning signs of failure.
- Data recovery is generally possible as the market for it is mature. Recovery costs vary on drive damage.
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Ads / Sponsors
 
Join Date: ∞
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
12-26-2010, 02:05 PM
juhok juhok is offline
Free Member
 
Join Date: Sep 2009
Posts: 379
Thanked 107 Times in 87 Posts
I have my personal footage + client tapes archived on file server which uses ZFS/RAIDZ1. On ZFS data is verified at read/write and Z1 copes loosing 1 drive. In the future(or even now) when drives get bigger and bigger and risk of loosing another drive while recovering from the failure of the first one calls for Z2 or more, protecting against simultaneous failure of 2 drives. Data is occasionally backed up on USB HD's in tandom with file server. This setup is pretty reliable, but not very portable - depends how much I utilize the USB drives.
Reply With Quote
  #3  
12-27-2010, 02:33 PM
admin's Avatar
admin admin is online now
Site Staff | Web Development
 
Join Date: Jul 2003
Posts: 4,310
Thanked 654 Times in 457 Posts
NJR, your observations are all pretty much accurate there.

Solid-state drives have unimpressive longevity stats right now, so these are best left to temporary storage. (I'm somewhat tired of reading people complain about losing all their images on SD cards, because they treated it like a permanent storage for the camera. SD is one of the worst of the SSD formats, as far as I can see.)

For optical media, single-layer DVD is pretty safe -- the safest of them all. It's the dual-layer DVD media, CD and Blu-ray that have various structural issues that cause serious concern for longevity. At only 4.38GB, of course, SL DVD isn't always the most convenient.

Hard drive recovery starts at about $1,000 and goes up from there. Most drives suffer from basic mechanical failures over time, and data can often be recovered well into the 90th percentile range. Of all the available HDD configurations, a mirrored RAID is the best. I use a RAID-1 for photo archives. Other data is merely duped to single drives, as it's non-active backup.

Also remember to use off-site storage. For a business, this can be anything from a second office location to an off-site storage facility. For home users, this can be the home of a parent, sibling or other relative -- maybe even your "day job" private office (if you have one), in a locked drawer.

And tapes weren't really all that robust anyway. I've seen moisture ruin many tapes, be it audio cassettes, VHS, 8mm, Hi8, DV or even computer DAT. Tape is tape -- humidity ruins it, leeches into the substrate.

- Did this site help you? Then upgrade to Premium Member and show your support!
- Also: Like Us on Facebook for special DVD/Blu-ray news and deals!
Reply With Quote
  #4  
12-30-2010, 04:19 AM
jmac698 jmac698 is offline
Free Member
 
Join Date: Dec 2010
Posts: 387
Thanked 73 Times in 56 Posts
These are great points. I have learned to make double copies of some important data, and also to inspect the backups every year. I think that is really important. I just read a study from Library of Congress who sampled discs from a 300,000 collection and found visible mold or bubbles on some discs, even though they tried to store them properly as a museum would. Other discs read poorly even with no visible problems.

I came here because I thougth I had an answer for a similar question, actually to archive uncompressed footage I did a test of *all* lossless codecs and found huffyuv from ffdshow has good compression and can be done real-time in capture for a modern computer. UTVideo is also very fast and great for editing work, and some of the best compression for it's speed. The famous Huffy is rather outdated now. x264 lossless is quite bad actually.
Reply With Quote
  #5  
12-30-2010, 04:59 AM
juhok juhok is offline
Free Member
 
Join Date: Sep 2009
Posts: 379
Thanked 107 Times in 87 Posts
Quote:
Originally Posted by jmac698 View Post
x264 lossless is quite bad actually.
How so? With the material I've tried, it beats ffdshow-huffyuv and FFV1 by fair margin (and with decent settings it's even realtime in modern computers). Capture/intermediate is different than archival tho. I capture huffyuv myself and if it needs to be archived then re-encode with better compression.
Reply With Quote
  #6  
12-30-2010, 05:37 AM
jmac698 jmac698 is offline
Free Member
 
Join Date: Dec 2010
Posts: 387
Thanked 73 Times in 56 Posts
http://compression.ru/video/codec_co...s_2007_en.html
This was a professional comparison.
I did also my own
http://forum.doom9.org/showthread.php?p=1463689

From what I've read, x264 does work well with certain material like animation which can benefit from intra frame compression.

I also found a VFW codec for x264 to let VirtualDub use that directly when capturing.
Huffy (the original one) I found too slow for realtime on my old machine, and the compression isn't that great. I'm a big fan of UTVideo now.
Reply With Quote
  #7  
12-30-2010, 06:09 AM
juhok juhok is offline
Free Member
 
Join Date: Sep 2009
Posts: 379
Thanked 107 Times in 87 Posts
x264 has come a long way in 3 years (compression.ru test is from 2007). I don't know about VFW "hacks", capturing directly to x264 seems counter intuitive. It's not very edit-friendly etc. But archiving is different story.

I ran a quick test. 1 minute denoised VHS with some static in the beginning:

ffdshow huffyuv: 363 442kB
ffdshow FFV1: 304 812kB
x264 (--preset superfast): 356 097kB (75fps)
edit: UTVideo 4:2:0 (Optimize for compression): 351 367kB
(all include 48/16 uncompressed audio)

So yeah, with this material x264 does not have so good compression. But it was 3x faster than FFV1. Using "--preset veryslow" I managed to get 2Mbps lower bitrate (343 706kB) and 15x slower compression time. It's not propably the way to go for archiving. But wide support - outside of ffdshow - for H.264 decoding counts as a plus IMHO.

Last edited by juhok; 12-30-2010 at 06:41 AM. Reason: Added UTVideo codec
Reply With Quote
  #8  
12-30-2010, 06:25 AM
jmac698 jmac698 is offline
Free Member
 
Join Date: Dec 2010
Posts: 387
Thanked 73 Times in 56 Posts
Quote:
x264 has come a long way in 3 years
Yes, that's exactly what I thought! It has, for lossy compression - no doubt x264 is the best quality codec compared to even professional ones. A test at compression.ru showed this a few months ago.
Quote:
But wide support - outside of ffdshow - for H.264 decoding counts as a plus IMHO.
What support? What electronic devices support lossless x264? I think ffdshow covers any computer application universally. I'm not aware of any lossless hardware devices (I'm sure there's some pro ones, redcode raw etc.).
x264 lossless however, has not changed. Your results practically mirror mine.

x264 ultrafast, one pass lossless.
Re-encode time: 1:40
Size: 498059 hmm.. doesn't seem right, I know it lost though.


It can be made editable by changing the keyint(?) setting to 1. And the VFW version is not a "hack" - but I can't find the keyint setting in the VFW configuration.
http://sourceforge.net/projects/x264vfw/

FFV1 compresses best with large context vs small, and AC is better than VLC - for this material (VHS cartoon).

Code:
Codec	Abbr Settings	Size	Recomp Time	Ratio	Speed	Realtime
[1][a]	Alpary realtime	791523	01:01:00	1.54	1.20	Yes
[1][b]	Alpary max	668964	06:36:00	1.82	7.76	No
[2][a]	Fastcodec	689627	01:20:00	1.76	1.57	Yes
[3][a]	Huffy Predict	658471	00:56:00	1.85	1.10	Yes
[3][b]	Huffy gradient	677685	01:00:00	1.79	1.18	Yes
[3][c]	Huffy Left	660053	00:54:00	1.84	1.06	Yes
[4][a]	Lagarith	623230	01:32:00	1.95	1.80	No
[5][a]	Mindvid		651051	01:18:00	1.87	1.53	No
[6][a]	MSU Speed	624649	07:38:00	1.95	8.98	No
[7][a]	Toponoky Speed	981436	04:57:00	1.24	5.82	No
[8][a]	Utvideo 422	628689	00:59:00	1.93	1.16	Yes
[8][b]	Utvideo 420	449833			2.70		Yes *nearly lossless (YV12 only)
[9][a]	Xvid L0 q=2	174589	03:36:00	6.96	4.24	No *Not lossless
[10][a]	FFV1 AC-Lrg	529714	03:58:00	2.29	4.67	No
[10][b]	FFV1 AC-Sm	573909			2.12		No
[10][c]	FFV1 VLC-Lrg	541541			2.24		No
[10][d]	FFV1 VLC-Sm	578978			2.10		No
[11][a]	FFHuf Left	617828	01:00:00	1.97	1.18	Yes
[12][a]	LJPG		504842	01:36:00	2.41	1.88	No *nearly lossless (YV12 only)
[13][a]	Uncomp		1215669	00:51:00	1.00	1.00	Yes *estimate from 2/3 24bit avi
[14][a] MLC Speed       574596  00:01:51        2.12    2.12    No
The huffyuv in ffdshow is better than the original huffy standalone. The adaptive Huffman checkmark makes an improvement.

Last edited by jmac698; 12-30-2010 at 06:37 AM.
Reply With Quote
  #9  
12-30-2010, 07:20 AM
juhok juhok is offline
Free Member
 
Join Date: Sep 2009
Posts: 379
Thanked 107 Times in 87 Posts
You got surprisingly good ratio with UTVideo compared to my sample. I added the info to my previous post. I've been weary of UTVideo before, but it seems more mature now for candidate to intermidiate codec inbetween capture and encoding to broadcast formats. Generally I wouldn't recommend it for long term archival to people who don't know exactly what they're doing. It's not that mature yet.
Reply With Quote
  #10  
12-30-2010, 08:05 AM
jmac698 jmac698 is offline
Free Member
 
Join Date: Dec 2010
Posts: 387
Thanked 73 Times in 56 Posts
Remember to compare apples here. My result for UTVideo 4:2:0 is much better because there are less color samples. If you scroll over my chart you will see I marked as nearly lossless. I was setting 4:2:2 as a standard for this test.
Here's a tidbit:
Quote:
Using WMPlayer, a 1080p/25fps video (using 'fast decode' encoding) uses about 18-20% of my 3.4Ghz Intel quad (Q6600), so easily real-time playback without any glitches.
If you need to know it's bug free, I could do some work to qualify it. I would like to use it myself so it's no problem. It's good for editing and capturing because it's keyframe of 1 and incredibly fast, leaving more time for filtering.

It was also found to be the only codec to play SD lossless realtime on a netbook (Intel Atom).
Reply With Quote
  #11  
12-30-2010, 08:14 AM
juhok juhok is offline
Free Member
 
Join Date: Sep 2009
Posts: 379
Thanked 107 Times in 87 Posts
Oops. Earlier I read your test more keenly. Now I forgot/missed the 4:2:2 vs 4:2:0 part. I know that UTVideo is fast and that's the very (only) appeal. If I would quantify my worries about the codec, in regard to recommending it to "general populace", it's new - not very widely used and thus tested, bitstream has already been changed atleast once (afaik), future support unknown (sources available which is a plus). My assesment is no doubt biased and lacks expertice in the field of lossless video coding, YMMV.
Reply With Quote
  #12  
12-30-2010, 08:41 AM
jmac698 jmac698 is offline
Free Member
 
Join Date: Dec 2010
Posts: 387
Thanked 73 Times in 56 Posts
No, really those are good points and I can understand. Others feel the same, they want opensource and continuous development. I have good feeling for this as latest version 8.5 is Nov/10. Will it work with the Directshow of 2015 in Windows 9? Who knows. But for capture/editing, probably OK, as long as I know it's bit exact in the software I use.
Reply With Quote
The following users thank jmac698 for this useful post: admin (06-12-2011)
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
MP3 for Archival Quality? Is MP3 a good CD backup? deter Blank Media 19 04-14-2010 12:17 PM
Blu-ray disks for archival media? admin Blank Media 0 02-12-2010 11:25 AM
Video 8 Capture/Transfer Method 1 or Method 2? mjk340 Capture, Record, Transfer 10 11-01-2009 06:11 AM
Method of turning off my external hard dr Sossity Computers 1 12-16-2008 06:38 AM
The lordsmurf method using DVD recorders lordsmurf Capture, Record, Transfer 0 02-13-2004 12:16 AM

Thread Tools



 
All times are GMT -5. The time now is 03:36 AM