digitalFAQ.com Forum

digitalFAQ.com Forum (https://www.digitalfaq.com/forum/)
-   Project Planning, Workflows (https://www.digitalfaq.com/forum/video-workflows/)
-   -   Storage/backups for large VHS-to-digital project? (https://www.digitalfaq.com/forum/video-workflows/8623-storage-backups-large.html)

koberulz 04-08-2018 03:43 AM

Storage/backups for large VHS-to-digital project?
 
What's the best storage option for a large VHS-to-digital project? I initially got an 8TB external, thinking that would last at least a good chunk of it, but I've run out of space almost immediately.

I also currently have no backups, which isn't ideal.

leeoverstreet 04-11-2018 12:48 AM

Quote:

Originally Posted by koberulz (Post 53646)
I've run out of space almost immediately.

Whoa. 8TB almost immediately? Define immediately. :-)

How many tapes (or how many hours) have you done? I've captured about 70 VHS tapes so far in my project, a few full but most not. Some with barely 10 minutes. I'm right at 1TB using a Matrox codec at 25Mbps (same rate but different format as DV). My 28 Hi8 tapes, which are typically more full, are up to about 600GB.

lordsmurf 04-11-2018 12:54 AM

Are you saving lossless?

koberulz 04-15-2018 06:41 AM

Lossless original caps + lossless intermediates for in-process restorations (deleted as appropriate). Uncompressed out of AviSynth/VDub into Premiere because there don't seem to be any lossless codecs that are compatible with both Mac and Windows.

lordsmurf 04-16-2018 02:44 AM

MagicYUV is lossless for both Mac and Windows, and even Linux. The 1.x version is freeware, the 2.x is payware.

It sounds to me as if you're trying to do too many concurrent projects at the same time, if you have that many intermediary files. Or perhaps capturing too many at once?

I don't know the answer here. I almost never have more than 4tb of temp files, and 4tb of captures. Usually half that.

Now then, I do have many more of the 15mbps 4:2:2 MPEG files, but those are much smaller than the lossless, and thus still only take up maybe 10tb. For now.

koberulz 04-16-2018 05:09 AM

I'm creating x264 files to save the final versions, but I want to retain the original lossless capture and the VDub-to-Premiere file.

Would MagicYUV be viable as a capture format?

lordsmurf 04-16-2018 04:30 PM

No. Intermediary only.
Even Lagarith isn't wise a capture format.
The problem here is the compression is aggressive, and a CPU cannot keep up for realtime encoding.

Huffyuv is about 25% larger files, but also a % less overhead.

koberulz 04-17-2018 06:55 AM

Does that mean longer encode times?

I use Lagarith for capture because HuffYUV causes all sorts of problems with Premiere.

Any thoughts on the best storage/backup methods?

lordsmurf 04-30-2018 08:10 PM

Encode times for Lagarith vs. Huffyuv is nominal on a modern CPU. Remember, VirtualDub and Avisynth can't run in all cores, and the lossless codec runs in a core as well.

(The reason Lagarith is not good for capture is that the CPU core use is higher, it spikes more with larger spikes, and could easily cause dropped frames if it hits 100% for whatever reason. Also remember that cores aren't dedicated to a task, but intelligently pooled. They still multi-task. So a large Lagarith spike + something else may cause 100% and therefore dropped frames. Of course, that's also why you don't do anything else while capturing.)

I see you have a concurrent thread at VH on this topic as well.

On semi-related note, I'm looking to redo my DLNA.
I'm actually eyeing 4x 5tb USD3 drives for it. Smaller, quieter, cooler. I'll dump 5x 3.5" drives (re-purposed as backup drives, mostly housed in a drawer).

My point is this: Before buying lots of noise/hot metal boxes (racks, RAIDs, NAS, etc), you need to think about the broader aspects of where you'll store those. For example, in the same room as the video/audio work is the pits. I sometimes want to yank a drive out of the bay and throw it out the window, due to heat and noise. With every upgrade, my first thought is "how can I make it even quieter and cooler".

Hopefully those H.264 encodes are using a 4:2:2 profile. ;)

koberulz 05-01-2018 03:40 AM

I'd actually stopped checking here after you replied there but not here, until lingyi mentioned there that you'd replied here.

I'm not sure if that sentence is coherent.

Any particular reason they'd need to be 4:2:2? MeGUI tends to yell at me if they're not YV12.

MrTIM 05-01-2018 10:15 AM

1 Attachment(s)
This is how I deal with storage:

I have a Qnap NAS and Expansion with at the moment 12x8TB hard drives. (Less usable since these are two Raid6 Pools).
I keep the original captures (HuffYuv). I always capture with two different hardware setups and keep both files temporarily After cutting I decided which of the two sets to keep. I then reencode to x264 after doing some restoration. These x264 files are then used for my Plex Server on the QNAP to serve to my family.

I am luckily soon done with my family VHS project and probably don't need to upgrade storage soon again. (See attachement - I am expanding my second storage pool right now. That's why there is a warning) I am also using the NAS for virtualization and such, so this might be overkill. But it has served very well for this project.

koberulz 05-08-2018 01:32 PM

Quote:

Originally Posted by koberulz (Post 53981)
Any particular reason they'd need to be 4:2:2? MeGUI tends to yell at me if they're not YV12.

lordsmurf, can you elaborate on this?

lordsmurf 05-08-2018 08:55 PM

I use the payware MainConcept. It takes full advantage of the specs.

You can do 4:2:0, it's the standard for basic H.264 as it is/was for MPEG (DVD, Blu,ray, often broadcast), but of course halves the color data. If you're doing delivery, that's fine. But for archival, especially if future editing or restoration is possible, then saving the original 4:2:2 is ideal.

I know almost nothing about MeGUI. It's a low-end crappy freeware ffmpeg front-end. Limited everything.

Handbrake is equally crap.

Avidemux is my freeware of choice for lossless>H.264, but never for 4:2:2. It lets you pick the high422 profile, but doesn't seem to have a setting to specify the chroma sampling like MainConcept does. So I'm not sure if simply picking the profile is enough (I highly doubt it!), or if another setting is buried.

Hybrid is my other choice freeware, but on the system where I encode, Hybrid will often freeze or crash on lossless input. It especially hates Huffyuv. You can both set the 422 profile, as well as select the 4:2:2 output colorspace. This is a non-WYSIWYG (but GUI) encoder, but it takes full advantage of ffmpeg and several other integrations (like Avisynth, which I've never gotten to work).

Freeware is nice, but you pay for it in terms of lack of documentation, bugs, and other misfires.

koberulz 05-08-2018 10:53 PM

I'm retaining the lossless capture so yeah, the H.264 is just delivery.

Never heard anything bad about MeGUI before.

jeekl 05-15-2018 06:36 AM

What's your level of commitment/technical skill? The correct answer to your original questions about storage and backup strategy is raid-array for storage and redundancy, offsite/online backup for backup security.

My personal setup is 10x6tb wd reds in a zfs raidz2 (two disk parity), and offsite/online (encrypted) backup to google drive using rclone. I'm guessing this is not the kind of answer you're looking for, but if you want to take this seriously, i recommend doing some lurking and reading on reddit.com/r/datahoarder, as a starting point.

dinkleberg 08-02-2018 09:59 AM

Here is what you need, in order:

Offsite backup (Cloud storage)
ECC RAM
Uninterruptible Power Supply
Onsite redundancy

For onsite redundancy you want RAID1 (2 disks) or RAID10 (3+ disks) array with ECC RAM. This is most easily accomplished with AMD Ryzen or certain Intel Celeron / Pentium CPUs. I recommend using mdadm software RAID as opposed to hardware RAID, because with hardware RAID you need to keep spare RAID cards on hand in case of a failure, as well as spare BBU's for the RAID cards. With software RAID you can just replace the drives and rebuild. It is not a bad idea to try and "stagger" the hard drives' ages within your array, there is a post here somewhere on why but basically drive failure is likely to occur at similar times within the same batch of disks. I don't have the exact link handy but it's here somewhere: https://www.backblaze.com/blog/

The hardware necessary for ZFS costs far more than redundancy, so it makes little sense. the /r/datahoarder people are... interesting...... but have far more money than sense.


All times are GMT -5. The time now is 07:49 AM

Site design, images and content © 2002-2024 The Digital FAQ, www.digitalFAQ.com
Forum Software by vBulletin · Copyright © 2024 Jelsoft Enterprises Ltd.