Quantcast Converting RAW video data losslessly for archiving? - digitalFAQ Forum
  #1  
03-02-2017, 08:52 AM
barlent barlent is offline
Free Member
 
Join Date: Mar 2016
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Hi everyone,

I am well into a fairly big project of digitizing some old VHS home videos from the early/mid 90s. These recordings were not very high quality - most were recorded on an old 80s camcorder and VCR (which I no longer have) on the extended speed setting in order to fit more footage onto a tape - man I sure wish I hadn't done that!!

Anyhow from reading these forums I stocked up with some decent equipment since regular VCRs could barely even produce a usable video signal from these tapes. Using a Panasonic AG-1980 VCR fed through a Panasonic DMR-ES10 I was able to get a stable picture. In some cases I had to also feed this through an AVT-8710 to further clean up the picture but that was only necessary on some of the really bad tapes. I am digitizing using a Diamond VC500 USB capture device.

I was having issues capturing within Videodub (audio sync issues), so ended up using the software shipped with the VC500 (EZ Grabber). I set this software to 'AVI' mode which produced a large uncompressed AVI file which mediainfo reports as "166 Mb/s, 720*480 (3:2), at 29.970 (29970/1000) FPS, YUV (NTSC)".

At this point I have 90% of my source material digitized into huge uncompressed AVI files (about 1.5TB worth)

The video looks great in the format it is in now. But it would be NICE to shrink it down a bit for archiving in a lossless fashion. HuffyUV (compressing using Videodub) seems to produce a seemingly identical copy - however the resulting file appears to end up as an RGB colorspace which I understand is not optimal.

I also attempted to use Handbrake to convert these files into x.264 using RF=0 and no filters applied - thinking this SHOULD result in an identical picture. The files were certainly much smaller than the uncompressed source files, but I found the quality to be less than perfect. Motion appeared rougher in the x.264 copy, and dark areas appeared slightly 'blotchy' or 'noisey'. Many of these videos are VERY dark with poor lighting so I cannot afford to be throwing away variances in the detail of dark scenes. Again, the uncompressed video looks great.

So... a few questions:

1. Is there anything wrong with the YUV uncompressed format I currently have these files in? I hear quite a bit about YUV2.. and honestly I am not sure what the difference is.

2. Are there additional handbrake settings (or alternate methods of encoding to x.264 lossless) that I could use to produce a truly LOSSLESS archive copy of this footage? The idea would be to create secondary copies at higher RF values for actual viewing / streaming.

3. Should I de-interlace? This material will spend most of its life being viewed on a PC, possibly streamed from a website, and occasionally being watched on a TV via a connection to a PC or USB stick.

4. HuffyUV - I played with the GUI codec options from within VirtualDub but the resulting file seems to always end up RGB. Any suggestion how to avoid this? Or is that fine?

Sorry for the long post - I hope I provided enough details. I fully admit to being a newbie in this area... every time I think I have it all figured out, something else pops up! Can't wait to get this footage archived and backed up!

Thanks!
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Ads / Sponsors
 
Join Date: ∞
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
03-02-2017, 09:47 AM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: Tennessee and NY, US
Posts: 1,992
Thanks: 65
Thanked 556 Times in 511 Posts
A lossless archive can be built with Lagarrith lossless compressor, which gives slightly smaller files than huffyuv in YUY2 and much smaller than RGB: https://lags.leetcode.net/codec.html.
Open the original capture(s) in Virtualdub.
Set "Video..." -> "Color depth..." to "4:2:2 YCbCr (YUY2)".
Set "Video..." -> "Compression..." to Lagarith or huffyuv. If huffyuv, disable "always suggest RGB" in the dialog setup. If Lagarith, set colorspace to YUY2.
Set "Video..." to "fast recompress".
Save the AVI with a new name.

You can also experiment with UT Video codec suite, which can compress more tightly in lossless mode using its custom config settings, but note that using its highest lossless compression will take a long, long, long, long time.
Reply With Quote
  #3  
03-02-2017, 10:59 AM
barlent barlent is offline
Free Member
 
Join Date: Mar 2016
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Thanks for the suggestion! I didn't notice the "Color depth" setting in virtualdub... perhaps that was my issue with HuffyUV ending up in RGB.

I am not too concerned with the time required to do the compress - I'll leave my PC on for a week if I have to :-)
Reply With Quote
  #4  
03-06-2017, 05:33 AM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 5,886
Thanks: 546
Thanked 840 Times in 758 Posts
Do not deinterlace.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #5  
03-07-2017, 09:38 AM
barlent barlent is offline
Free Member
 
Join Date: Mar 2016
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Thanks lordsmurf.. after taking a closer look at the lossless x.264 files I generated I have noticed that they were in fact de-interlaced, even though I had the deinterlace filter in handbrake turned off. This could explain the differences I was seeing. It appears there may be additional x.264 flags required to force the encoder to work in interlace mode.

I have gone back to my original RAW avi copies and compressed them using the Lagarith codec as described above to created my archive masters - worked great!
Reply With Quote
  #6  
03-07-2017, 10:51 AM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: Tennessee and NY, US
Posts: 1,992
Thanks: 65
Thanked 556 Times in 511 Posts
Quote:
Originally Posted by barlent View Post
I am well into a fairly big project of digitizing some old VHS home videos from the early/mid 90s. These recordings were not very high quality - most were recorded on an old 80s camcorder and VCR (which I no longer have) on the extended speed setting in order to fit more footage onto a tape - man I sure wish I hadn't done that!!
Yes, molst of my old VHS tapes were slow speed. Wishj I hadn't done that msyelf!

Quote:
Originally Posted by barlent View Post
I was having issues capturing within Videodub (audio sync issues), so ended up using the software shipped with the VC500 (EZ Grabber). I set this software to 'AVI' mode which produced a large uncompressed AVI file which mediainfo reports as "166 Mb/s, 720*480 (3:2), at 29.970 (29970/1000) FPS, YUV (NTSC)".

At this point I have 90% of my source material digitized into huge uncompressed AVI files (about 1.5TB worth)
I get the point, and glad you worked it out with Lagarith. VirtualDub borrows capture settings from the EZGrabber configuration. If you've set EZGrabber to uncompressed AVI, and then use VirtualDub for capture, yoiu have to specify compression and colorspace in VirtualDub capture.

The audio sync issue can be solved with the proper sync settings in VirtualDub. A more recent and detailed settings guide is here at Capturing with VirtualDub [Settings Guide]. In particular, audio and other sync settings for USB cards are shown and discussed in part 5, which covers the capture options dialog, here: Capturing with VirtualDub [Settings Guide]. With the VC500 you should not enable audio playback during capture. You can preview audio during setup and prep work, but turn it off for capture. if you want to monitor whether you're getting audio during capture, use the "Audio" top menu and enable "volume meter" which displays an active volume meter at the bottom of the capture window. This is shown in Part b4, here: Capturing with VirtualDub [Settings Guide].

I don't use Handbrake or MEGUI or such for processing/deinterlace/telecine, etc., so I can't answer for them.
Reply With Quote
  #7  
03-11-2017, 03:13 AM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 5,886
Thanks: 546
Thanked 840 Times in 758 Posts
x/H.264 doesn't handle interlace well. It was always intended for non-interlaced use, be it broadcast or web streaming.

MPEG-2 is what you want for interlaced content.

I always used quality tapes (BASF, JVC, TDK, pre-2000 Maxell, and always the better grades) so I didn't face some of the issues many others do. I did use EP, but it was usually a good recorder. In the 90s, I swapped to S-VHS-ET, in EP mode, and quality on it was on par with SP VHS. Maybe better.

The big zonk was using SVHS-C. Stupid decision, should have opted for Sony Hi8.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #8  
03-19-2017, 10:54 PM
msgohan msgohan is offline
Free Member
 
Join Date: Feb 2011
Location: Vancouver, Canada
Posts: 779
Thanks: 40
Thanked 176 Times in 154 Posts
Quote:
Originally Posted by lordsmurf View Post
x/H.264 doesn't handle interlace well. It was always intended for non-interlaced use, be it broadcast or web streaming.
Say what? They specifically developed more efficient interlaced coding for H.264.

"I have mixed feelings about H.264's interlaced support. I'm impressed it was done so well. But I worry that it gives the industry one more excuse to keep the interlaced milestone around our necks another decade." - Ben Waggoner, Compression for Great Video and Audio: Master Tips and Common Sense

I don't think x264 supports the crushingly-slow 3-pass encoding mode of the H.264 reference encoder, though. Wonder how its shortcuts compare.

Fast Frame/Field Coding for H.264/AVC
"In the H.264/AVC reference software [5], a multi-pass approach is used to perform the PAFF and MBAFF decisions. In this scheme, the frame is first encoded in frame picture mode. When MBAFF coding is used, every macroblock pair is coded as two frame macroblocks or two field macroblocks respectively, and the mode with lower rate-distortion cost is selected, and the corresponding bitrate-distortion cost of all the macroblock pairs in the frame are recorded and summed to get the cost of whole frame picture. It is obvious that this MBAFF frame coding strategy needs two-pass coding. With PAFF coding, the same frame is also encoded in field mode, which means the frame is coded as two separate field pictures, and the rate distortion cost of each macroblock is again recorded and summed to get the cost of two fields in the current frame. So three-pass coding is needed for PAFF and MBAFF coding in reference software [5]. Mode decision of PAFF coding is then made based on summed rate-distortion cost values obtained above.
With the right rate-distortion optimization method, the mode selection scheme in reference software provides with an optimal decision for both MBAFF and PAFF coding, but the computational complexity triples compared to a single-pass approach..."
Reply With Quote
  #9  
03-20-2017, 02:37 AM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 5,886
Thanks: 546
Thanked 840 Times in 758 Posts
It may be more developed/advanced that MPEG, but there are issues.

From my understanding, the main focus in development was as a progressive format. Interlacing was not the main focus, but something merely added later on. As such, it's hackish.

And then the main problem is playback/decoding. Very often, interlace playback is a mess.

But recording/editing/etc is also an issue. For example, I think every NLE has issues with interlace H.264. I've read the same about advanced/complex broadcasting hardware as well. And I don't mean old reports, but some as recent as Q4 2016.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #10  
03-20-2017, 01:10 PM
msgohan msgohan is offline
Free Member
 
Join Date: Feb 2011
Location: Vancouver, Canada
Posts: 779
Thanks: 40
Thanked 176 Times in 154 Posts
Quote:
Originally Posted by lordsmurf View Post
From my understanding, the main focus in development was as a progressive format. Interlacing was not the main focus, but something merely added later on. As such, it's hackish.
ITU & MPEG make "hackish" standards? Not sure where to source info regarding levels of development focus, so I won't bother arguing that point. But interlaced support was present from the very beginning. They weren't about to drop that, since it was always designed to support broadcasting among the many other possible applications. (And I think only the exceedingly optimistic believed broadcasters would change everything from mastering up to transmission to support true progressive.)

Beside, interlacing was "something added later on" to MPEG in the first place, as you know! MPEG-1 being progressive, and MPEG-2 adding interlaced support and little else.

Quote:
And then the main problem is playback/decoding. Very often, interlace playback is a mess.
Unfortunately, I don't think "I've never had a problem" will be a useful response here. I only started using it after it was very mature.

You wanna talk decoding problems: the nightmare that is interlaced VC-1 support.
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
Where is audio/video sync data located in video files? merchantord Edit Video, Audio 5 07-26-2014 11:52 PM
Advice converting scientific data on S-VHS to digital? yellow_inspir Capture, Record, Transfer 4 08-24-2013 07:18 PM
Archiving Video Format for Capturing VHS JasonCA Project Planning, Workflows 3 03-09-2013 07:35 PM
Universal compression scheme for data for archiving ? Sossity Computers 29 03-10-2011 04:30 PM
Archiving Video For Viewing matt- Project Planning, Workflows 3 09-25-2009 12:12 AM

Thread Tools



 
All times are GMT -5. The time now is 07:49 PM  —  vBulletin Copyright Jelsoft Enterprises Ltd