Best SD video archive format?

[BmacSWA]

Let me start this conversation hoping it doesn't devolve into a confrontation.

I've been going through the process capturing and archiving SD VHS, Hi8 and DV with mostly 1st gen tapes with the purpose of making modern viewable video on contemporary consumer devices: Phones, tablets, TV, YouTube and so on. Below are my workflows...

DV:
It is what it is. I just use WinDV and store it as is. File size isn't insane and its as clean as the format will provide. I just leave it as is.

SD Analog:
I have been capturing with VirtualDub 1.9.x (1.9.1.1 I think?) compressing lossless with Huffy 2.1.1 then running through QTGMC to deinterlace and convert to 60fps outputting to ProRes422 HQ to be edited in Final Cut Pro.

Both processes work fine. It may be overkill, but I've been archiving both the Huffy AVIs and the ProRes Files. My thought being with the Huffy files I could have a lossless copy to convert to whatever the new editing software may be in the future I keep the ProRes so I can drop it into Final Cut if I decide to re-edit the clip.

However this is the issue posing the question. I've been experimenting with different capture processes and start with an 8 bit uncompressed .mov file after the capture. The files are obviously larger than a Huffy2.1.1 lossless capture, but considering I can go direct to a ProRes422 HQ file from there after using QTGMC, Is it worth converting the uncompressed video to Huffy or another format? My thoughts being I'd like a reasonably sized lossless file to archive that is in an "original" interlaced format that also could be used down the road for perhaps newer software or hardware that may replace QTCMC.

I'd love to hear everyone thoughts.

Bmac

[latreche34]

Keep the HuffYUV files before QTGMC, pretty simple.

[BmacSWA]

So the complete summary of that statement is archive as HuffYUV 2.1.1, period... or pretty simple as you say? No other archive formats being used out there?

[BW37]

I think you’ll get a broader set of options and opinions if you ask over at videohelp.

[latreche34]

There are other similar formats like Lagarith or prores, I think prores is not entirely lossless but I'm not a Mac guy so don't take my word for it, Since you are familiar with HuffYUV use it, "pretty simple" means that if you want to go back in the future and do some editing or restoration without repetitive generational loss the raw HuffYUV files you have are the only option, once de-interlaced it's already lost some quality there.

[aramkolt]

Advantage of ProRes is 10 bit color and significantly less storage space with humanly imperceptible losses when used at the higher bitrates. It will store interlaced content just fine, issue is I don't think all versions of AVIsynth/QTGMC will work in a 10 bit color space. I've seen some threads claiming that QTGMC will work with 10 bit color depth - I just haven't tried it myself.

Advantage of 10 bit would be less potential for banding in scenes where there is a a slow color gradient change such as sunsets/sky. See here:
https://chrisdobey.com/tech/2019/6/19/8-bit-and-10-bit

If hard drive space wasn't a concern, I think we could all agree that a 10 bit capture could be superior visually to an 8 bit capture in some cases, but should never be inferior visually to an 8 bit capture, especially in certain situations. Drawback is it usually takes up much more hard drive space, possibly for minimal benefit in most situations.

Sweet spot might be 10 bit ProRes capture that takes up less space than an 8 bit HuffYUV capture, especially if you plan to do all of your editing on a Mac anyway.

Eventually I'll post some captures showing raw AVI captures Vs ProRes as well as hardware/software deinterlacers on the same source material and that will probably mean more than talking about theoretical differences.

[lordsmurf]

Quote:

Originally Posted by aramkolt

Advantage of ProRes is 10 bit color
I just haven't tried it myself.

No.

Most consumer analog SD formats, including the "best" (relatively speaking) S-VHS/Hi8, and more comparable to 6-bit dithered, and 8-bit can more than capture the palette. 10-bit is just measurbating gear, zero utility for the captures.

(And before some chime in about DV, yes DV still degrdes it farther. The best of 90s video tech, yuck.)

Quote:

Advantage of 10 bit would be less potential for banding in scenes where there is a a slow color gradient change such as sunsets/sky. See here:
https://chrisdobey.com/tech/2019/6/19/8-bit-and-10-bit

That just does not apply whatsoever here. It's like discussing the nuances of jet fuel formulations, but you have a 1990 Honda Accord.

Quote:

If hard drive space wasn't a concern, I think we could all agree that a 10 bit capture could be superior visually to an 8 bit capture ,

No.

Quote:

but should never be inferior visually to an 8 bit capture,

Also no. Why? Because it's not just about 2 more bits, but the processing to get there. Inferior processing, and/or processing an unexpected source, leads to issues. This is a great example on why "pizza box" broadcast rack-mount TBCs so utterly fail to perform on consumer analog sources -- not expected.

Quote:

Sweet spot might be 10 bit ProRes capture that takes up less space than an 8 bit HuffYUV capture, especially if you plan to do all of your editing on a Mac anyway.

There's really no reason to go beyond standard ProRes422 (non-HQ) at 8 bits..

Quote:

Eventually I'll post some captures showing raw AVI captures Vs ProRes as well as hardware/software deinterlacers on the same source material and that will probably mean more than talking about theoretical differences.

You must always be aware of all variables.

This is why Youtube is so full of BS "comparisons". For example, here's a video I commented on just a few days ago. And that Youtuber is a perfect example of a schmuck. Rather than learn from my comment pointing out the obvious oddities and flaws, he chose to just disable comments. His "comparison" is really between composite and Y/C, not about the 710-USB or vhs-decode at all. Worse yet, his "after" sample has this odd horrid halo/ringing to it (mush with cartoonish outlines), which was not present on the "before". Meaning it was never a 1:1 comparison, he's fudging data (intentionally or not).

And again, this is why Youtube videos are nonsense in general, the Youtuber is able to remove comments, or block all. He/she only wants to see gushing comments, literary fellatio.

So when you post samples, don't try to back into a conclusion made before testing even began. Especially not if it's laughably obvious false BS like that Youtube clip was.

Furthermore, the idea of "blind" is nonsense, when you're not an expert designing the study. People with knowledge need to see what is what, in order to eliminate errors made, variables that exist. I don't want to see two random video clips, I want to see labeled A and labeled B. We need to see if expected aspects are present, and why not if not. Science isn't blind. Bias is real, but most researchers can set aside bias based on actual results. Blinding is for the laymen, not the researchers. When laymen design a study, they don't understand this.

[dpalomaki]

Some interesting notes on 10-bit vs lower bit count gray images.
https://pixelcraft.photo.blog/tag/human-eye/

Many argue that the typical human eye can do maybe 6- to 7-bit on normal TV viewing. The human eye is not linear and the brain does interpretation. Resolvable bit depth will vary with average intensity and adaption over time.
Specialized medical imaging and trained readers (e.g., x-ray) may be able to 9-bit under the right viewing environment and brightness range. But that is not typical video.

Another limitation on useful bit depth is noise level. Is 10-bits (60 dB) useful if the noise floor is 7 bits (42 dB) which is arguably good for consumer analog video.

I suspect the main advantage of 10-bit is for pristine, noise free material that will be extensively processed. The added two bits will reduce the truncation errors that can accumulate with repeated processing, and for applications such as medical imaging where the viewing environment is tailored to use take advantage of the added depth.

[lordsmurf]

Quote:

Originally Posted by dpalomaki

I suspect the main advantage.

I think the usual main advantage is "my bits are bigger than your bits, nanny nanny nyah nyah!"

Childish pissing contests.

- "My dad makes more money than your dad."
- "My mom's car is better than your mom's car."
- etc etc

[BmacSWA]

I think in my original post I hoped this CONVERSATION would not devolve into a CONFRONTATION. Besides LS, if you get all excited it may affect the work you're doing to get me the TBC and VCR that has been pre financed. I'm simply concerned for your well being... and getting some electronics in the mail.

[lordsmurf]

Quote:

Originally Posted by BmacSWA

I think in my original post I hoped this CONVERSATION would not devolve into a CONFRONTATION. Besides LS, if you get all excited it may affect the work you're doing to get me the TBC and VCR that has been pre financed. I'm simply concerned for your well being... and getting some electronics in the mail.

I had a post reply window open, just didn't submit.

I don't see confrontation at all, just some harsh reality, mostly due the machinations of marketing and the American consumer psyche ("more = better!"). ie, bits. You'll notice that the real pushers of bits are desperate companies trying to edge out competiting (and often superior) products. It reminds me of Toshiba's "6 head" (BS) VCRs.

Shipping updates on gear are pending, imminent. The frozen weather has finally moved on, allowing some gear movement again.

Quote:

Originally Posted by BmacSWA

I've been going through the process capturing and archiving SD VHS, Hi8 and DV with mostly 1st gen tapes with the purpose of making modern viewable video on contemporary consumer devices: Phones, tablets, TV, YouTube and so on. Below are my workflows...

Standard stuff.

Quote:

DV:
It is what it is. I just use WinDV and store it as is. File size isn't insane and its as clean as the format will provide. I just leave it as is.

Yep.

But note this: the digital transfer/"capture" has lags, and can not capture footage. Sometimes seconds matter, so you may need/want to at least view the DV via analog, to be assured you have all the footage. It can get tedious, so I sometimes capture both (if I think anything was clipped/missing), and then sort it later in post.

Quote:

SD Analog:
I have been capturing with VirtualDub 1.9.x (1.9.1.1 I think?) compressing lossless with Huffy 2.1.1 then running through QTGMC to deinterlace and convert to 60fps outputting to ProRes422 HQ to be edited in Final Cut Pro.

Hopefully you mean 59.94 -- it's not 60.
Otherwise, yep.

Quote:

Both processes work fine. It may be overkill, but I've been archiving both the Huffy AVIs and the ProRes Files. My thought being with the Huffy files I could have a lossless copy to convert to whatever the new editing software may be in the future I keep the ProRes so I can drop it into Final Cut if I decide to re-edit the clip.

Not overkill, 22tb+ drives are $300. You have space for cheap. Backup drives correctly, maybe Huffyuv on one, ProRes422 on the other. Preferably different brands, meaning Seagate and WD.

Quote:

However this is the issue posing the question. I've been experimenting with different capture processes and start with an 8 bit uncompressed .mov file after the capture. The files are obviously larger than a Huffy2.1.1 lossless capture, but considering I can go direct to a ProRes422 HQ file from there after using QTGMC,

This is viable. Large, but visable. ProRes422 HQ may not be needed, however -- referring to HQ.

Quote:

Is it worth converting the uncompressed video to Huffy or another format? My thoughts being I'd like a reasonably sized lossless file to archive that is in an "original" interlaced format that also could be used down the road for perhaps newer software or hardware that may replace QTCMC.

Either way involves conversion.
- capture ProRes422, convert Huffyuv lossless copy
- capture Huffyuv, convert ProRes422 copy for Mac processing.

Half a dozen one way, 6 the other.

MagicYUV is a lossless native to Mac+Windows (+Linux), but odds of it being futureproofed are near zero. It's a minor league one-man codec project. It's outstanding "for now", but when longevity is a consideration, no. Huffyuv has longevity, ProRes, maybe FFV1 (but it's lousy for non-HD, also not really lossless in same way ProRes is not).

Quote:

Originally Posted by BmacSWA

So the complete summary of that statement is archive as HuffYUV 2.1.1, period... or pretty simple as you say? No other archive formats being used out there?

The keyword for archive is "longevity". Huffyuv has it. How many other codecs can you find online, where users have done their best to keep it functioning on new OS? (Rhetorical. Answer = none. Just Huffyuv.)

Quote:

Originally Posted by BW37

I think you’ll get a broader set of options and opinions if you ask over at videohelp.

Perhaps. But also a lot of really dumb opinions, too, Be careful of consensus opinions. The general population is generally stupid. One must vet, eliminate wrong opinions (wrong opinions = opinions based on bunk).

Quote:

Originally Posted by latreche34

There are other similar formats like Lagarith or prores, I think prores is not entirely lossless but I'm not a Mac guy so don't take my word for it, Since you are familiar with HuffYUV use it, "pretty simple" means that if you want to go back in the future and do some editing or restoration without repetitive generational loss the raw HuffYUV files you have are the only option, once de-interlaced it's already lost some quality there.

ProRes is "visually lossless", but not 100%, not true lossless. Still, close enough. Huffyuv is really the only SD lossless format that has longevity. Lagarith is 2nd, all the rest does not (Ut, Magic, whatever). FFV1 is really just for HD, and ProRes422 is not too different from DNxHD in MXF (I hate those).