Quantcast Storage/backups for large VHS-to-digital project? - digitalFAQ Forum
  #1  
04-08-2018, 03:43 AM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
What's the best storage option for a large VHS-to-digital project? I initially got an 8TB external, thinking that would last at least a good chunk of it, but I've run out of space almost immediately.

I also currently have no backups, which isn't ideal.
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Ads / Sponsors
 
Join Date: ∞
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
04-11-2018, 12:48 AM
leeoverstreet leeoverstreet is offline
Free Member
 
Join Date: Nov 2016
Location: Alabama
Posts: 12
Thanks: 0
Thanked 3 Times in 3 Posts
Quote:
Originally Posted by koberulz View Post
I've run out of space almost immediately.
Whoa. 8TB almost immediately? Define immediately. :-)

How many tapes (or how many hours) have you done? I've captured about 70 VHS tapes so far in my project, a few full but most not. Some with barely 10 minutes. I'm right at 1TB using a Matrox codec at 25Mbps (same rate but different format as DV). My 28 Hi8 tapes, which are typically more full, are up to about 600GB.
Reply With Quote
The following users thank leeoverstreet for this useful post: lordsmurf (04-11-2018)
  #3  
04-11-2018, 12:54 AM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 7,318
Thanks: 752
Thanked 1,118 Times in 997 Posts
Are you saving lossless?

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #4  
04-15-2018, 06:41 AM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
Lossless original caps + lossless intermediates for in-process restorations (deleted as appropriate). Uncompressed out of AviSynth/VDub into Premiere because there don't seem to be any lossless codecs that are compatible with both Mac and Windows.
Reply With Quote
  #5  
04-16-2018, 02:44 AM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 7,318
Thanks: 752
Thanked 1,118 Times in 997 Posts
MagicYUV is lossless for both Mac and Windows, and even Linux. The 1.x version is freeware, the 2.x is payware.

It sounds to me as if you're trying to do too many concurrent projects at the same time, if you have that many intermediary files. Or perhaps capturing too many at once?

I don't know the answer here. I almost never have more than 4tb of temp files, and 4tb of captures. Usually half that.

Now then, I do have many more of the 15mbps 4:2:2 MPEG files, but those are much smaller than the lossless, and thus still only take up maybe 10tb. For now.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #6  
04-16-2018, 05:09 AM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
I'm creating x264 files to save the final versions, but I want to retain the original lossless capture and the VDub-to-Premiere file.

Would MagicYUV be viable as a capture format?
Reply With Quote
  #7  
04-16-2018, 04:30 PM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 7,318
Thanks: 752
Thanked 1,118 Times in 997 Posts
No. Intermediary only.
Even Lagarith isn't wise a capture format.
The problem here is the compression is aggressive, and a CPU cannot keep up for realtime encoding.

Huffyuv is about 25% larger files, but also a % less overhead.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #8  
04-17-2018, 06:55 AM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
Does that mean longer encode times?

I use Lagarith for capture because HuffYUV causes all sorts of problems with Premiere.

Any thoughts on the best storage/backup methods?
Reply With Quote
  #9  
04-30-2018, 08:10 PM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 7,318
Thanks: 752
Thanked 1,118 Times in 997 Posts
Encode times for Lagarith vs. Huffyuv is nominal on a modern CPU. Remember, VirtualDub and Avisynth can't run in all cores, and the lossless codec runs in a core as well.

(The reason Lagarith is not good for capture is that the CPU core use is higher, it spikes more with larger spikes, and could easily cause dropped frames if it hits 100% for whatever reason. Also remember that cores aren't dedicated to a task, but intelligently pooled. They still multi-task. So a large Lagarith spike + something else may cause 100% and therefore dropped frames. Of course, that's also why you don't do anything else while capturing.)

I see you have a concurrent thread at VH on this topic as well.

On semi-related note, I'm looking to redo my DLNA.
I'm actually eyeing 4x 5tb USD3 drives for it. Smaller, quieter, cooler. I'll dump 5x 3.5" drives (re-purposed as backup drives, mostly housed in a drawer).

My point is this: Before buying lots of noise/hot metal boxes (racks, RAIDs, NAS, etc), you need to think about the broader aspects of where you'll store those. For example, in the same room as the video/audio work is the pits. I sometimes want to yank a drive out of the bay and throw it out the window, due to heat and noise. With every upgrade, my first thought is "how can I make it even quieter and cooler".

Hopefully those H.264 encodes are using a 4:2:2 profile.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #10  
05-01-2018, 03:40 AM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
I'd actually stopped checking here after you replied there but not here, until lingyi mentioned there that you'd replied here.

I'm not sure if that sentence is coherent.

Any particular reason they'd need to be 4:2:2? MeGUI tends to yell at me if they're not YV12.
Reply With Quote
  #11  
05-01-2018, 10:15 AM
MrTIM MrTIM is offline
Free Member
 
Join Date: Dec 2015
Posts: 37
Thanks: 0
Thanked 0 Times in 0 Posts
This is how I deal with storage:

I have a Qnap NAS and Expansion with at the moment 12x8TB hard drives. (Less usable since these are two Raid6 Pools).
I keep the original captures (HuffYuv). I always capture with two different hardware setups and keep both files temporarily After cutting I decided which of the two sets to keep. I then reencode to x264 after doing some restoration. These x264 files are then used for my Plex Server on the QNAP to serve to my family.

I am luckily soon done with my family VHS project and probably don't need to upgrade storage soon again. (See attachement - I am expanding my second storage pool right now. That's why there is a warning) I am also using the NAS for virtualization and such, so this might be overkill. But it has served very well for this project.


Attached Images
File Type: png Untitled.png (28.1 KB, 13 downloads)
Reply With Quote
  #12  
05-08-2018, 01:32 PM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
Quote:
Originally Posted by koberulz View Post
Any particular reason they'd need to be 4:2:2? MeGUI tends to yell at me if they're not YV12.
lordsmurf, can you elaborate on this?
Reply With Quote
  #13  
05-08-2018, 08:55 PM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 7,318
Thanks: 752
Thanked 1,118 Times in 997 Posts
I use the payware MainConcept. It takes full advantage of the specs.

You can do 4:2:0, it's the standard for basic H.264 as it is/was for MPEG (DVD, Blu,ray, often broadcast), but of course halves the color data. If you're doing delivery, that's fine. But for archival, especially if future editing or restoration is possible, then saving the original 4:2:2 is ideal.

I know almost nothing about MeGUI. It's a low-end crappy freeware ffmpeg front-end. Limited everything.

Handbrake is equally crap.

Avidemux is my freeware of choice for lossless>H.264, but never for 4:2:2. It lets you pick the high422 profile, but doesn't seem to have a setting to specify the chroma sampling like MainConcept does. So I'm not sure if simply picking the profile is enough (I highly doubt it!), or if another setting is buried.

Hybrid is my other choice freeware, but on the system where I encode, Hybrid will often freeze or crash on lossless input. It especially hates Huffyuv. You can both set the 422 profile, as well as select the 4:2:2 output colorspace. This is a non-WYSIWYG (but GUI) encoder, but it takes full advantage of ffmpeg and several other integrations (like Avisynth, which I've never gotten to work).

Freeware is nice, but you pay for it in terms of lack of documentation, bugs, and other misfires.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #14  
05-08-2018, 10:53 PM
koberulz koberulz is offline
Premium Member
 
Join Date: Feb 2016
Location: Perth, Australia
Posts: 404
Thanks: 9
Thanked 2 Times in 1 Post
I'm retaining the lossless capture so yeah, the H.264 is just delivery.

Never heard anything bad about MeGUI before.
Reply With Quote
  #15  
05-15-2018, 06:36 AM
jeekl jeekl is offline
Free Member
 
Join Date: Jul 2017
Location: Sweden
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
What's your level of commitment/technical skill? The correct answer to your original questions about storage and backup strategy is raid-array for storage and redundancy, offsite/online backup for backup security.

My personal setup is 10x6tb wd reds in a zfs raidz2 (two disk parity), and offsite/online (encrypted) backup to google drive using rclone. I'm guessing this is not the kind of answer you're looking for, but if you want to take this seriously, i recommend doing some lurking and reading on reddit.com/r/datahoarder, as a starting point.
Reply With Quote
  #16  
08-02-2018, 09:59 AM
dinkleberg dinkleberg is offline
Free Member
 
Join Date: Aug 2016
Posts: 38
Thanks: 35
Thanked 6 Times in 6 Posts
Here is what you need, in order:

Offsite backup (Cloud storage)
ECC RAM
Uninterruptible Power Supply
Onsite redundancy

For onsite redundancy you want RAID1 (2 disks) or RAID10 (3+ disks) array with ECC RAM. This is most easily accomplished with AMD Ryzen or certain Intel Celeron / Pentium CPUs. I recommend using mdadm software RAID as opposed to hardware RAID, because with hardware RAID you need to keep spare RAID cards on hand in case of a failure, as well as spare BBU's for the RAID cards. With software RAID you can just replace the drives and rebuild. It is not a bad idea to try and "stagger" the hard drives' ages within your array, there is a post here somewhere on why but basically drive failure is likely to occur at similar times within the same batch of disks. I don't have the exact link handy but it's here somewhere: https://www.backblaze.com/blog/

The hardware necessary for ZFS costs far more than redundancy, so it makes little sense. the /r/datahoarder people are... interesting...... but have far more money than sense.
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
1000 VHS Tapes to Digital Project sunsetandgower Project Planning, Workflows 15 07-23-2014 05:41 AM
New Member Introduction - my VHS-C to digital project! Dissones4u General Discussion 20 06-28-2014 01:32 AM
Need some advice with VHS to digital project dukeletoatreides Project Planning, Workflows 14 10-24-2013 07:31 AM
Pull backups vs. push backups from production server via Email or PM Web Hosting 1 07-25-2013 07:18 PM
Archiving slides to digital: Blu-ray DVDs for storage? Scan quality? kpmedia Photo Processing, Scanning & Printing 0 10-17-2010 05:56 AM

Thread Tools



 
All times are GMT -5. The time now is 04:30 AM  —  vBulletin Copyright Jelsoft Enterprises Ltd