1000 VHS Tapes to Digital Project
 

Thanks to the recognizable name of lordsmurf who have been following on videohelp.com, I came across this board yesterday and have been absorbing all the information here.

I have a on again/off again project I have been working on, figuring out the method I have been using is crap, throwing everything away and starting over - this is now my third round of starting from scratch.

I have about 1,000 video tapes - most of them were recorded in EP and about 90% of them are of a daytime soap opera over the course of a 20 year archive. My goal is to convert them to digital files, and store them on how ever many external hard drives it takes, while making backup dvd's of the mp4 files.

The first time I started this project (which has since been abandoned) I was using a vcr/dvd recorder for the dvd portion, and using a Canopus to convert the analog to digital, however I was not as particular as I should have been with the end result quality - either the dvd portion or the mp4 portion (and using a crappy vcr to boot), and I am pretty much throwing away 2 years worth of work there.

I started over this past week, using the Canopus ADVC-300, my new (to me) Panasonic AG-1980 and my new and fast Macbook Pro.

I know that many don't feel that the Canopus/DV workflow is the right way to go, but I've been around the bend and back with this and it is the only thing that works for me. My ultimate goal is not to make tv ready dvd's, but to have mp4 files that can be played from my computer, as backups to my vhs tapes that I will never get rid of. I have been using visualhub to do the conversion of the raw dv files to mp4 files, and that conversion brings them down to size of about 1.27 GB per file.

I realize that it is most likely impossible for me to ever actually finish this project, but my focus is going to be on the tapes that I don't think will last another 30 years, and the ones that are in good shape are on the backburner, which I will get to converting if I have the time but are not my concentration.

Really, I probably haven't watched most of these tapes in a good 20 years, it is only because of the fact that vcr's are on the way out, and I have so much invested in this - both time and money - that I am doing what I can to preserve my collection.

Any input from the experts here and advise on how to make my workflow as smooth as possible, would be greatly appreciated.

Starting over was a wise move. Even the best of us have been there ourselves!

EP mode VHS tapes (or SLP mode VHS, as it was named in the 80s), with 6 hours per tape compressed onto the video tape, can be the most difficult recording mode to work with. You generally need a wide player head, on a VCR made to work with EP/SLP recordings. Of all the VCRs out there, the Panasonic AG-1980P S-VHS VCR does work pretty well for this. (For PAL users, it's the AG-1980 equivalent known as the Panasonic NV-FS200 PAL S-VHS VCR.) So you've made a great choice there.

All VHS tapes suffer from chroma noise and unstable timebase. You'll want to plan for correcting chroma errors (if possible), as well as correcting the timebase on all axis. The TBC inside the AG-1980 only works to correct visual elements, it does not purify the signal. You'd be wise to add a full-frame external timebase corrector, such as the AVT-8710 from B&H, available for about $225 shipped. Although I will note that your exact Canopus DV box may or may not ignore some of those signal-quality timebase issues, allowing for a quality transfer. It really depends on the tapes.

When it comes to VHS-to-DVD / VHS-to-MP4 transfer work, a Mac isn't the best tool for the job. It's like running in a marathon with a few missing toes and a broken foot. You'll still get there, but it won't be first place, and you might fall a few times. Mac video workflows are very DV and HDV centric, meant for stuff you shoot on cameras and transfer over Firewire. That's why the Canopus works for you, it's essentially forcing VHS transfer work into a DV workflow, although that has drawbacks. The drawbacks of using DV compression on VHS have already been explained in a number of places, both on the site guides, as well as various forum posts in past months/years (as you've mentioned), so I won't go over it again.

VHS-to-MP4 won't be as bad as VHS-to-DVD, however. You'll be processing it anyway, so you can attempt to correct some of the DV damage.

Remember that MP4 is a container, not a format. It's just like AVI. The MP4 container was made for MPEG-4 format videos, usually used for H.264 AVC MPEG-4 Part 10 compression. Are you making H.264 videos? If so, realize the Apple Quicktime version of H.264 is crappy compared to MainConcept or even the open-source x264 version. Thankfully, there are x264 options for Mac! You can use Handbrake or a few others. I've not recently looked at any Mac H.264 x264 apps, as I don't have Mac access at the moment. I can schedule some Mac time at a friend's office in early June, if I want/need to review anything. (Let me know if you want me to try anything, or if you'll be fine on your own.)

You've mentioned Visualhub, which uses FFMPEG for MPEG-4. I would avoid that, and instead use x264 for better quality.

One primary disadvantage of only archiving to H.264 over MPEG-2 comes with deinterlacing. A high bitrate MPEG-2, or a max bitrate DVD-Video MPEG-2, left interlaced, may appear much higher quality than even the best software-deinterlaced MP4 file. You'd be best leaving the de-interlacing to the display, and retain the interlacing in the actual video file.

Yes, technically H.264 supports interlacing, but you'll be hard-pressed to play it back easily. Software playback may run out of resources, and then I'm not readily aware of media players that will play interlaced content correctly or at all. This is an area where I've not strayed very much, as there's been no need for it to date with projects I do.

It's just assumed that anybody converting to H.264 generally has to deinterlace it, in order to play it later on. But again, that comes with quality consequences.

I'd be all for an additional stream being H.264 MP4, in addition to a primary MPEG-2 archive. I'm not too fond of a workflow that ends in a single MP4 file, with no added digital archives.

If you do several tapes weekly, 1,000 tapes can be done in 3-5 years, at a casual basis. It takes closer to 1 year, with multi systems, running daily.

I would only bother archiving important tapes -- don't archive that HBO movie you recorded in 1987 that was released three times on DVD in the past 15 years already. Family videos, old TV shows, TV-only movies, TV specials, and maybe even commercials, are the things worth saving. Anything else tends to be a complete waste of time.

I recorded things years ago myself, that I always said I'd watch "someday". While I never again graced a VCR with that tape, I did transfer it to DVD and finally watch it. For very rare things or family movies, I kept the tapes. For TV shows, I generally pitched the tapes when done. (Realize, of course, that I had gotten the full potential quality from the tape -- something most people can't do, and therefore most people should NOT throw away tapes after conversion!!)

I see things to suggest additionally to your workflow, which probably doesn't make it "smoother" as much as it might make it more archivally sound.

I see nothing bad, wrong or dumb in your proposed workflow. That in itself is a good sign. :)

And thanks for becoming a Premium Member, to support this site. Much appreciated.

Thanks so much for your thoughtful response! A couple of follow up notes:

1 - videohelp.com is completely responsible for my choice of the Panasonic AG-1980. It has been a lifesaver for me, it reads tapes correctly that none of my "regular" vcr's can. That accounts for quite a bit of my SLP collection, so it was completely worth the investment for me.

2 - At this point with all the money I invested - I can't really put any more money into equipment. Could you please clarify for me what an external TBC would accomplish that I am not getting in my current setup? Keeping in mind that about 90% of what I am transferring is soap opera content, so it is tape as opposed to film. The rest of my collection are either movie of the weeks, episodes of series that aired on the WB about ten years ago, and things of that nature. If it is a worthwhile investment that will help with the project I am open to this, but I definitely need a better education as to how this would help my workflow.

3 - In my visualhub settings, I am indeed using H.264. Can you please clarify why you don't believe visualhub is a good tool here? I admit I don't know much about FFMPEG for MPEG-4. This is a tool that has been recommended to me in DV conversion in many Mac communities, so I have always used it. What I will not do this time around is use imovie for the encoding to mp4, as I have never been happy with the quality when I've gone that route.

Also, can you please confirm that I can use Handbrake to convert DV to mp4? I do indeed use Handbrake, but specifically to convert VOB files from dvd's, not to convert the raw DV coming from imovie.

3 - regarding deinterlacing, I am using an older version of imovie - it is imovie 6HD, which does not have the crappy autoation setting like the current versions of imovie that deinterlace the videos upon import. And I am also careful about that in the visualhub conversion, to not have deinterlace checked off.

The videos all play in quicktime - this I have been testing out, they are the same format that they would be as if I was running them through appletv, so I can play them full screen on my computer, and watch them hooked up to my HDTV as a monitor.

4 - As I previously mentioned I am not making dvd's that are playable in a dvd player, and that is intentional as this is not how I watch media anymore. For me it makes more sense to make backup dvd's of the mp4 files, so if I have a hard drive crash I do not need to go through this process again. I am getting about 3 episodes on a dvd. Not ideal in the fact that I need to account for this extra storage space, but I'm also not comfortable with a hard drive being the only source of the content with all the work I am going to be putting into this project.

4 - As far as only archiving important tapes, they are all important to me:) There is very little, outside of duplicate content, that I do not want to salvage here. So I realize what I am in for. I couldn't possibly get through more than 2 six hour tapes in a week. The importing of the content is just the first step. With breaking the episodes up into individual files, exporting the DV out of imovie and then the compression via visualhub, it is quite a lot of hours in a week that my computer is processing this video. Which is why I feel like this project will never really be finished, I just need to think about priorities in regards to the tapes and which ones are more important to preserve in the short term, and which have a longer shelf life.

At the same time, I'm not pitching any of these tapes. I'm in this for the long haul, and the tapes themselves are important to me. This is more of a backup, and an easier way to watch content when I want access to it.

Any other thoughts you have in regards to making this project more archivally sound would be appreciated - that I think is ultimately what I am looking for at this point, for the most part I have the technology nailed down I think.

I will probably have lots more to say on the topic as I start to make my way through my collection, very glad to have found this board.

checking in, I've been at this for about a week now. Things are going smooth, but sloooow. That's to be expected though since I really only have nighttime and very early morning before I leave to for work to play with this project.

doing my best to not let it consume my life:D

when you have this many tapes, its hard to not get overwhelmed by not knowing where to start.

Will get you an answer here after the weekend. Thanks!

thanks, looking forward to the feedback:)

I have to say the more I use my Panasonic 1980 the more I am blown away with how it plays my tapes. It's like a friggin miracle working in making them viewable.

1. The Panasonic AG-1980P advice was probably from one of our members here, or from somebody that is repeating advice they originally got from this site or our members. Regardless of the source, it's great that you have that S-VHS deck to play your VHS tapes back in high quality.

2. What TBCs do is explained fully at this link already: http://www.digitalfaq.com/forum/show...=9889#post9889
If you think you can skip an external TBC, then try it.

3. The video quality of FFMPEG is inferior to x264 or MainConcept. There is more noise, and it requires more bitrate for equal quality (or actually, still lower quality). It's just not good. It's like comparing a hot dog to steak. Yeah, you can eat both, both are meat, but one is obviously much higher quality. FFMPEG is the hot dog. I suggest you switch to an encoder that use x264 for its encoder engine.

4. Hard drives are fine, but you need to have a good "backup policy" in place. That includes duplicates of the hard drive, or even a RAID1 solution, and with good drives (Seagate or Western Digital only, not Maxtor/others). DVD is fine, too -- but again, you need multiple copies, using good media (Mitsubishi, Taiyo Yuden). More on good discs at http://www.digitalfaq.com/reviews/dvd-media.htm

5. (aka, the other #4) - Yep, it will take a while to backup all 1,000+ tapes, 6 hours each, in their entirety. For long-term archives, I'll still impress upon your that MPEG-2 is probably a better choice, as opposed to H.264 MPEG-4 Part 10 AVC, for an archival solution. I advise in an ethical manner, so I have to mention this. However, you're free to choose your method.

It sounds like you're all set, aside from possibly changing to another H.264 encoder.

Just wanted to update this thread.

The hard drive idea has gone poof - literally. After months of work, plus the older video I had captured the past few years, I was a moron and didn't have a backup drive of my newer content. The drive I have been working with is no longer readable on my computer.

So while I can probably rescue the data with the help of a professional, this really made me realize that hard drives are not a good long term archive solution - I'm going to go back to saving what really matters to me on dvd. I don't think its realistic with the amount of dvds involved to focus on my entire collection, but rather the aspects of it that I would be upset if the vhs tapes were degraded to the point that they were no longer viewable.

Just a big ole ugh. But lesson learned, loud and clear. Yes, I was a moron. Waiting too long to get a backup policy in place is an idiotic, moronic thing to do. And it's not like I don't know better, because I am that anal about my music collection.

I don't think I could have been any stupider here....

Yes, it is. But I've been guilty of slow backups myself, sometimes to my own detriment. So don't be too hard on yourself. Hard enough to learn better, yes. But as my baseball coach used to say, "shake it off" and "you'll get 'em next time". And I did. (I'll spare you the swat on the butt, however.)

The only place I suggest is Kroll Ontrack Data Recovery: http://www.krollontrack.com/data-recovery

It will run at least $1,000 to recover a bad drive. However, you need to understand it requires precision tools and training, in an ISO clean room. Not exactly easy work, or a cheap sort of business to maintain. Is it worth it? It was to me, and it is to their customers. If that drive reflects a year of work, 365 days x $3 = ~$1,000 range. So $3 per day -- is your time worth $3 per day? I would imagine so. When you send them a drive, they'll email a spreadsheet listing the recoverable files, including the % of recovery possible. If you agree, they'll recover the files to a new hard drive (which they provide). If you deny, you'll have to pay return shipping to retrieve your unfixed drive.

I still have my recovery list, and thanks to a recent defrag just prior to the crash, I recovered something like 95% of the files perfectly, with the remaining files merely being damaged (although still playable, with bad frames here and there). I think a few tiny TXT note files were completely lost, but not a big issue.

In my case, the drive's death was from an internal mechanical failure. The platter itself was fine, save the area where the arm likely struck the surface of the disk. Literally, the "crash" of the drive. And the data at that location was damaged.

That's still not a wise solution. Duplicate, duplicate, duplicate! Triplicate. Quadruplicate. Put files on disc, copy those discs to more discs, copy those discs to drives, copy those drives to more drives. Then put some at home, some at another location (office or home of family/parents/siblings). That's a good redundant backup solution that would pretty much only fail if the world ended (atmospheric EMP, nuclear war, alien invasion, killer meteor, etc). It costs you a few bucks for the blank media, yes. It takes a good bit of time to do all that copying, yes. But long-term, it's a great insurance policy for your data.

I feel for you, I really do. Been there, done that. It's a lousy feeling to lose everything.

In total, I probably spend 3 solid work weeks per year simply backing up terabytes of data. It's manually triplicated, with multi-site storage, and repeated at least twice per year. That doesn't include any time spent on incrementals, which I also manually monitor and duplicate from time to time, just to be sure. I don't have the time to redo work, and often times, it's impossible to redo it (photo/videos of events, for example).

RAID-1 and RAID-10 are my friends, too!

I will have to figure this all out. I do thankfully have all my vhs tapes which have kept up well, thankfully they have also been under climate control. The only thing I know for sure is that I will never rely on a harddrive for my only point of backup again. I think it went poof when I tripped over the cord and it detatched from the computer without ejecting - that's the last time it was working, anyway.

What I really need to think about now is realistically, which vhs materially is valuable enough for me to go through this exercise again. I really barely even made a dent in my collection, even with loosing about 500 gigs or so of data.

Hmmm.... If that's the case, then you may have only blown out the enclosure itself. This happened to a friend of mine. I ripped the external enclosure apart (it wasn't one of those easy-to-remove screwed-together models), and then put the hard drive inside a computer. It worked perfectly that way. Only the enclosure was damaged; the power board was fried. He bought a new external drive (bigger, better), and I got a free 500GB SATA drive!

And always velcro-wrap cables behind the desk, or to the side, where it's not in your way.

That would be a good plan. :thumb:

Oh thank you! It hadn't even occurred to me that it was the enclosure. If that's the case this should be a much, much easier fix to retrieve my content.

Doesn't change the fact that I have come to the conclusions that hard drives are too fragile for a long term archiving system, but still - this could be a big relief.

Right now what I'm trying to focus on is the order of importance of how upset I would be if my vhs suddenly stopped working, and converting to dvd in that order.

Just as an FYI, I was researching something on Google earlier, and this dFAQ thread was on pg2 of the SERPs. Odds are that it's in Google for a number of keyword results. So your chronicle here may help others from repeating your mistake. You're not only learning, you're sharing, and becoming a case study for others to learn from. So some good will still come of this. Maybe not for you directly, but it's a partial consolation ("booby prize") to know you're helping others to avoid repeating your mistakes. Wise people learn from the mistakes of others, not just their own mistakes.

Take that drive apart, and I'm crossing my fingers for you. I really hope it's just a dead enclosure. That would be great for you.

Post back if you need anything else. Thanks for sharing your story, and updating this thread.

Just wanted to update that after a very long break I'm back on this project. A lot has changed technology wise in how video is delivered and my thoughts now are to buy 3tbdrives and save the DV files as well as the lossy mp4's,at least for the tapes I am willing to get rid of. The end goal here is really to clear out space and not have tapes completely overflowing throughout my apartment. Losing that harddrive really paralyzed me in regards to moving forward and I needed to take time off from looking at video and thinking about this overwhelming task.

I did a test run over the weekend and my workflow seems to be efficient - I have the magnavox hdd recorder that I can move over a hundred hours of VHS content to at a time, and then batch convert with iMovie and handbrake.

I cannot stress enough how much I learned my lesson with that harddrive catastrophe, it really brought home the point about backups.

In this day and age I don't believe that DVD is viable for long term storage, it is clearly headed the way of VHS in regards to longevity so the message has come across loud and clear - backup, backup, backup.

Here I go again....wish me luck!:)

DVD is the most archival storage format currently available -- outlasting tape, hard drive, solid state, and more other optical formats. So there's really no reason to dismiss it.

Wishing you luck! :)