Uncompressed for archiving?

[Mejnour]

We are just talking

From a preservationist point of view, about archiving VHS over the long run (decades, century); considering the size of storage available today (10-12 TB hard drive), I was wondering
if the is some advantages capturing uncompressed AVI?

I read that when capturing with huffyuv codec YUY2, there is always a small loss during color space conversion depending of the software you use for editing. For example, Adobe Premiere do the color space conversion from YUY2 to RGB.

I guess that capturing uncompressed AVI YUY2 would give the same result, color space conversion to RGB.
But does the "loss" is the same? since it's "not codec dependant"?

Then I read that "uncompressed video file" do not guarantee universality (trouble free) for editing since several kind of uncompressed AVI exist.

All thoses questions that coming in my mind are LordSmurf fault, yeah I wish I never read this thread

https://forum.videohelp.com/threads/...ile-size/page2

[sanlyn]

Quote:

Originally Posted by Mejnour

From a preservationist point of view, about archiving VHS over the long run (decades, century); considering the size of storage available today (10-12 TB hard drive), I was wondering
if the is some advantages capturing uncompressed AVI?

Yes. No digital compression artifacts are added to the capture.

Quote:

Originally Posted by Mejnour

I read that when capturing with huffyuv codec YUY2, there is always a small loss during color space conversion depending of the software you use for editing.

Where did you see that? VHS is stored as YPbPr, and YUY2 is the most similar colorsp[ace available to most of us. There is no color loss.

Quote:

Originally Posted by Mejnour

For example, Adobe Premiere do the color space conversion from YUY2 to RGB.

Poor example. RGB is stored as 4:4:4. YUY2 is 4:2:2. How would there be color loss going from YUY2 to RGB? There is damage, since we know that Adobe's colorspace convbersion is pretty sloppy, especially with i9n terlaced source.m And one has to be careful suring YUYr captur to observe safe video signal levels, or darks will get crushed and brights will be clipped during YUY2 to GB conversions from nonstandard signal levels. The loss is not recoverable.

Quote:

Originally Posted by Mejnour

I guess that capturing uncompressed AVI YUY2 would give the same result, color space conversion to RGB. But does the "loss" is the same? since it's "not codec dependant"?

What loss are you talking about> Capturing to YUY2 using Huffyuv, Lagarirth, or UT Video codec is not color loss. You can do a lot of processing without converting to RGB. But if you use Adobe for processing, then you're into sloppy conversions. We advise to make colorspace conversions with Avisynth for greater precision. Besides, what can you do for VHS captures with Adobe except cut-and-join edits, timeline effects, color work, and encoding? It has no provision for cleanup of video defects.

Quote:

Originally Posted by Mejnour

Then I read that "uncompressed video file" do not guarantee universality (trouble free) for editing since several kind of uncompressed AVI exist.

Why not? And where are you reading this nonsense? Lossless codecs are lossless from compression stage to compression stage. You can import a huffyuv lossless video, make changes, and losssleslly re-compress it as Lagairth or UT Video codec. These are all lossless. "Lossy" means that there is data loss with each compression. The data loss is not recoverable when lossy files are decoded for processing. The next next lossy encode of the intermediate working file involves more data loss and more compression artifacts. The loss and artifacts are accumulative in lossy encoding and are not recoverable.

All thoses questions that coming in my mind are LordSmurf fault, yeah I wish I never read this thread
https://forum.videohelp.com/threads/...ile-size/page2[/quote]

And did you see some of the replies to that rather iffy and vague comment by Lordsmurf? This, for example:

Quote:

I've tested HuffYUV 2.1.1 with thousands and thousands of frames. The YUY2 video that comes out is always exactly the same as the YUY2 video that went it.

The huffman compression scheme is lossless:

http://en.wikipedia.org/wiki/Huffman_coding

Huffman coding (and variations thereof) are used in most archiving programs (like WinZip, WinRar, etc.) where absolute losslessness is required. Any losses in HuffYUV would be due to a bug in the codec or mishandling of the video outside HuffYUV. I've seen no evidence of a bug like this in HuffYUV. Also note that HuffYUV hasn't been updated since 2003. So it's not possible that it has improved with time. I can't vouch for other HuffYUV implementations (ffdshow, for example) but I've seen no evidence of problems other than Lordsmurf's vague "I knew some guy had had a problem once" allegations.

https://forum.videohelp.com/threads/...e2#post1983999

I suggest you re-read that thread more carefully.

Or work strictly with uncompressed video if that's what's required to relieve your misgivings. When you find evidence of databit loss with huffyuv, Lagarith, or UT Video, you should report it to the developers thru their website. Be sure to include hard evidence and examples, and post here to let digitalfaq know about it.

And when lordsmurf encounters hard evidence of databit loss through lossless codecs, he will undoubtedly follow-up with a full report.

[Mejnour]

Thanks Salyn,

My english writing is pretty limited, something I use wrong word. But I really appriciate your answer because it point out many subtilities that I was missing. It even challenged me to do more research. And I admit that there were many ideas/questions that was blowing my mind (too much reading) and I should have digest it a bit more.

Roughly the first idea was about archiving in long term, best way to store a Master, trying to predict the impredictable things that may happens on the long run. Right now I am capturing in huffyuv. Would it be still workable in 50 years? I wondering why they discarted Huffyuv in their study? Not really for archiving?


http://download.das-werkstatt.com/pb...ontainers.html

=======================

It's pretty clear that if I understand you correctly, analog is the poor kid of video editing, and if you want to work "optimally" in the native color space, the choice of software is limited. By the way you may gave me the slap I needed to work with avisynth that always look to me as too tech for me. Scripts/line codes

On the same topic, I found a thread that seem to say that it is possible to stay in YUV color space in Premiere Pro depending of the filters you use, but yeah it mean you're limited.

https://forums.adobe.com/thread/825920


Thanks for your knowledge

Regards

[sanlyn]

Quote:

Originally Posted by Mejnour

Right now I am capturing in huffyuv. Would it be still workable in 50 years? I wondering why they discarted Huffyuv in their study? Not really for archiving?

I don't see where any part of that link discarded huffyuv for archiving. If anything, they showed a long list of lossless codecs, and none of them are 100% guaranteed to be mainstream 50 years from now. As new lossless codecs come along over the years, you can always re-encode to the new codec. Or just use no compression at all for archiving.

Quote:

Originally Posted by Mejnour

It's pretty clear that if I understand you correctly, analog is the poor kid of video editing, and if you want to work "optimally" in the native color space, the choice of software is limited.

Post processing might require filters and processing that call for different color spaces. This is why Avisynth has so many high-precision colorspace conversion functions.

Quote:

Originally Posted by Mejnour

On the same topic, I found a thread that seem to say that it is possible to stay in YUV color space in Premiere Pro depending of the filters you use, but yeah it mean you're limited.

What would you do in YUV with Adobe? Working with color in YUV isn't like working in RGB color. YUV controls and histograms don't work the same way, so you have to learn and adjust to different techniques. Remember that Adobe's color conversion methods aren't so wonderful.

[lordsmurf]

Even YUY2 uncompressed isn't guaranteed 50 years from now. That in itself is a type of codec, simply one that is defaulted to the system. Windows and Mac have different ideas of what "uncompressed" means, as an example.

Of all the lossless codecs, Huffyuv is probably the one with most staying power. Though Lagarith seems to have gained in the x64 era, and even I've adopted it more heavily in the past 2 years.

Many years ago, early 2000s, I saw some odd discrepancies in lossless encoded files. The reason is easy: Huffman has rounding errors. But I've not seen that in probably 10 years now. At this point, I think it was a combo of the Huffyuv version (not yet 2.1.1), and perhaps some sort of underrun of resources in that early P4 era.

The libavcodec/ffmpeg version of Huffyuv is terrible, don't use it.

The attached white paper gives a bit more details on Huffman for video encoding.

The space requirements of uncompresses make it undesirable. But it's also the processing overhead of the larger files. Even modern 2018 CPUs on SSD get slowed by uncompressed video.

[Mejnour]

Quote:

Originally Posted by sanlyn

I don't see where any part of that link discarded huffyuv for archiving. If anything, they showed a long list of lossless codecs, and none of them are 100% guaranteed to be mainstream 50 years from now. As new lossless codecs come along over the years, you can always re-encode to the new codec. Or just use no compression at all for archiving.

It was in the list in "Interesting for archiving: "lossless compression" section. (see attached image)

Quote:

Originally Posted by sanlyn

Post processing might require filters and processing that call for different color spaces. This is why Avisynth has so many high-precision colorspace conversion functions.

1-Are you using Avisynth with virtualdub only?
2-And if yes, only with the 32-bits version?
3-Virtualdub/Avisynth combo support batch processing?

Quote:

Originally Posted by sanlyn

What would you do in YUV with Adobe? Working with color in YUV isn't like working in RGB color. YUV controls and histograms don't work the same way, so you have to learn and adjust to different techniques. Remember that Adobe's color conversion methods aren't so wonderful.

Thanks this is why I will really look to learn to work with Avisynth

[Mejnour]

Quote:

Originally Posted by lordsmurf

Even YUY2 uncompressed isn't guaranteed 50 years from now. That in itself is a type of codec, simply one that is defaulted to the system. Windows and Mac have different ideas of what "uncompressed" means, as an example.

Of all the lossless codecs, Huffyuv is probably the one with most staying power. Though Lagarith seems to have gained in the x64 era, and even I've adopted it more heavily in the past 2 years.

Many years ago, early 2000s, I saw some odd discrepancies in lossless encoded files. The reason is easy: Huffman has rounding errors. But I've not seen that in probably 10 years now. At this point, I think it was a combo of the Huffyuv version (not yet 2.1.1), and perhaps some sort of underrun of resources in that early P4 era.

The libavcodec/ffmpeg version of Huffyuv is terrible, don't use it.

The attached white paper gives a bit more details on Huffman for video encoding.

The space requirements of uncompresses make it undesirable. But it's also the processing overhead of the larger files. Even modern 2018 CPUs on SSD get slowed by uncompressed video.

Thanks LS for you comment, that is exacly those kind of insight I was looking for.

I did some research, and I know that the BBC and NARA (national US archives) used uncompressed YUY2/YUV.
Then considering that the size of HDD are always increasing, I was just re-thinking if there would be some
signifiant benefits from keeping archives uncompressed vs huffyuv. Depending of the number of hours you have to archive, space is less a problem today but like you point it out, there is no more stability/security going uncompressed and it could be a problem from a editing point of view, still hard on modern CPUs.

Thanks also to point it ou about Lagarith, I don't know much about it, but if you mention it, I guess it deserve some attention. Does it means that from a x64 Hardware point of view, Lagarith have a edge over Huffyuv? Could it be a important consideration for the future?

Have you "play" with FFV1 codec? It seem to have a hype around it recently! It's been adopted from some national archives countries and it is among the candidate from this study...

http://download.das-werkstatt.com/pb...ontainers.html

Thanks again!

[hodgey]

Quote:

Originally Posted by Mejnour

It was in the list in "Interesting for archiving: "lossless compression" section. (see attached image)

They don't mention the specific reasons for each codec, but given that huffyuv is not a proprietary format, it's probably due to the lack support for different bit depths and colour spaces (huffyuv only supports 8bpp YUY2 and RGB) that lead them to not see it as a viable alternative. For analog SD captures, YUY2 is fine. Analog SD is normally captured as some variant of yuv with 422 chroma subsampling, i.e a luma (brightness) and two chroma (colour) components, where the colour components have half the horizontal resolution of the brightness component. There are a few ways of storing the raw data, one of which is YUY2. For this huffyuv (or lagarigth for that matter) are perfectly fine, the space savings with FFV1 won't be massive, and FFV1 is significantly more processor intensive to encode and decode.

On the other hand, if you have video data from some other source that is in a different colour format, or is has a higher bit depth than 8, huffyuv won't work.

Though, in general, all these formats (huffyuv, lagarith, FFV1) are non-proprietary, with source code and specifications widely available for free, so for preservation either should be fine as long as the format can store the data losslessly (I mean if you are really really paranoid you could store a copy of the source code for the codec as well). Most lossless video codecs are also much less complex that modern lossy codecs like h264 and newer.

[sanlyn]

Quote:

Originally Posted by Mejnour

Are you using Avisynth with virtualdub only?

I've tried running Avisynth scripts in other programs, but VirtualDub is the easiest and most straightforward, and I can apply VirtualDub filters and undertake other operations while running and modifying scripts.

Quote:

Originally Posted by Mejnour

And if yes, only with the 32-bits version?

64-bit has a long way to go, and many 64-bit versions of excellent and essential filters will never appear. I use 32-bit only. There is no speed increase with 64-bit filters. And you can't mix 32-bit and 64-bit in the same application. Besides, Avisynth's vast arsenal of 32-bit filters is trouble enough to learn and update.

Quote:

Originally Posted by Mejnour

Virtualdub/Avisynth combo support batch processing?

Yes, but I work with and inspect one video project at a time and use specific filters and methods for each piece of video. I have no illusions about one-size-fits-all processing, automation, or anything like that. The videos I work with don't lend themselves to batch methods. Also, I don't waste time on internet streamed video sources. Video off the internet always ends up looking like garbage, so I leave it as-is. All of my projects are full-length videos and complex slide shows, and each requires thorough attention. Fortunately, samples submitted to this and other forums are short pieces that don't take up the time that I put into my personal projects -- I consider work on forum samples as learning tools, for myself as well as for readers.

One of these days I'll have to submit a sample of my own nightmare videos and see what readers come up with.

[Mejnour]

Quote:

Originally Posted by sanlyn

I've tried running Avisynth scripts in other programs, but VirtualDub is the easiest and most straightforward, and I can apply VirtualDub filters and undertake other operations while running and modifying scripts.

Apologize for my ignorance in advance, It may be as stupid question

If I capture Huffyuv to create a master, once the file is saved, I decide (many weeks later because I have no time before) to apply some Avisynth or virtualdub filters to "improve" my master file.
When I save the file with the applied filters, could it be still considered as a master?
I mean I assume that I was able to save losless again with the combo virtualdub/avisynth.

Thanks