Quantcast Best formats for long-term archiving of data? - digitalFAQ Forum
Go Back    Forum > Digital Life > Tech Myths, Misinformation

Reply
 
LinkBack Thread Tools
  #1  
03-30-2018, 04:45 PM
ehbowen ehbowen is offline
Premium Member
 
Join Date: Feb 2018
Location: Houston, Texas
Posts: 203
Thanked 21 Times in 21 Posts
First, I do understand that you don't put all your eggs in one basket...you believe that the video on that M-Disc really will hold up a thousand years (or at least a hundred) but then find...not so much. But, those baskets can get expensive. Which are the best ones to consider investing in?

For storage of very large files (read: original lossless captures), if I even want to hang on to them at all, I'm thinking lotsa 4TB hard drives or else invest in one of the modern tape drive units. For final video files M-Disc might be worth considering, but are there other alternatives which are worth looking at as well?

I've got friends and family who are entrusting me with their memories. I'd like to leave them in better shape than I received them.
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Ads / Sponsors
 
Join Date: ∞
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
09-30-2018, 04:39 PM
lordsmurf's Avatar
lordsmurf lordsmurf is online now
Site Staff | Video
 
Join Date: Dec 2002
Posts: 8,082
Thanked 1,306 Times in 1,153 Posts
Quote:
Originally Posted by ehbowen View Post
First, I do understand that you don't put all your eggs in one basket...
I've always hated cliches, and this one is especially onerous.

Quote:
you believe that the video on that M-Disc really will hold up a thousand years (or at least a hundred) but then find...not so much.
Yes. Reality vs. marketing. Everybody claims to have the best, but many fail to meet those claims with independent validation and testing. As if the case with blank media especially.

Quote:
Which are the best ones to consider investing in?
If referring just to recordable DVD, then Taiyo Yuden and Verbatim. To a lesser extent, maybe some Fuji oxonol dye on Ritek, but I've not seen any in a while now (RITEKFxx media codes).

Quote:
For storage of very large files (read: original lossless captures), if I even want to hang on to them at all, I'm thinking lotsa 4TB hard drives or else invest in one of the modern tape drive units.
Get Seagate. Pay attention to Backblaze tests.
Hitachi if you don't care about noise.

Quote:
For final video files M-Disc might be worth considering, but are there other alternatives which are worth looking at as well?
TY and MCC/MKM Verbatim test better, have longer success record.

Quote:
I've got friends and family who are entrusting me with their memories. I'd like to leave them in better shape than I received them.
Store extra copies at their locations, not just your own. And hoping that some are in different geographic areas, not all huddled in the same city/area (as natural disasters can wipe out a large area, ie Houston).

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #3  
10-06-2018, 10:48 PM
paples paples is offline
Free Member
 
Join Date: Oct 2018
Posts: 5
Thanked 1 Time in 1 Post
I know this isn't a "cloud" topic but one option to even a cloud is Usenet. You'd run a paid "free" trial and then you could encrypt the content and upload it to Usenet. If you use a less popular group, you could have 10 year retention on that server. Of course you could cross post it too. Anyways, after about 8 years, you could just get another "free" trial then download it, then upload it again to refresh the retention.

I'm thinking you're looking for a silver bullet option, I feel it doesn't exist. I'd like to add to smurf's physical location tip. I've known people (with the money) who ship to P.O. boxes all over the world. I've never been clear on how they destroy the material once the box is full, I assume thy just let the encrypted discs go... wherever. I know they just open another box at some point, but I'm not in this financial bracket to do such a thing, but know it's being done.
Reply With Quote
The following users thank paples for this useful post: lordsmurf (10-06-2018)
  #4  
10-06-2018, 10:52 PM
lordsmurf's Avatar
lordsmurf lordsmurf is online now
Site Staff | Video
 
Join Date: Dec 2002
Posts: 8,082
Thanked 1,306 Times in 1,153 Posts
Quote:
Originally Posted by paples View Post
I know this isn't a "cloud" topic but one option to even a cloud is Usenet. You'd run a paid "free" trial and then you could encrypt the content and upload it to Usenet. If you use a less popular group, you could have 10 year retention on that server. Of course you could cross post it too. Anyways, after about 8 years, you could just get another "free" trial then download it, then upload it again to refresh the retention.
I would be wary of public uploading for sensitive information, as encryption is only safe until cracked. And in a decade or more, sometimes even less, once-impenetrable encryption can be broken by script kiddie or even IT help desk tools.

- Did my advice help you? Then become a Premium Member and support this site.
- Find television shows, cartoons, DVDs and Blu-ray releases at the TVPast forums.
Reply With Quote
  #5  
10-06-2018, 11:27 PM
paples paples is offline
Free Member
 
Join Date: Oct 2018
Posts: 5
Thanked 1 Time in 1 Post
That's true, it can't be, but friends and family where the source. Well, unless the material is... well, adult :-).

To beat off the path, also there is methods of obfustication that are insanely time consuming to reconstruct without a seed/map. I know this isn't the place to mention this, but on Usenet you can randomly generate usernames and randomly choose proxies. Besides that, you can chunk encrypted files into such small parts and post them to random groups, thus breaking any file up into thousands of pieces across thousands of groups. To feasibly reconstruct you'll need a seed, a map (probably .nzb) and a .par2 file. Sure, the encryption can be reversed along with the obfustication, but even a NSA agent would take a deep breath in thinking how to do it in timely manner (probably start at the Usenet provider).

To end, I once wrote a script in python years ago that did the above, but I don't do it anymore. However, I still use Usenet in this manner for content in regards to friends and family, and I can vouch for at least 12 years of retention. The real problem i have with it is, you can't exactly "read" from Usenet in the same similar manner you can with anything else.
Reply With Quote
  #6  
10-07-2018, 04:24 PM
lingyi lingyi is offline
Premium Member
 
Join Date: May 2011
Posts: 54
Thanked 21 Times in 15 Posts
From the title, I thought this was going to be one of gamemanico's endless threads!

As lordsmurf stated, store on multiple media in multiple locations and don't place the responsibility of storage on yourself. Put at least one set in a safe deposit box.

Burn everything on optical discs (Verbatim or TY as lordsmurf stated)and distribute. If they want a secondary backup (i.e. hard drives), have them pay or share the cost of the drives.

Tape drives are an additional form of storage only if you have very large (i.e. 10's of terabytes of data) collection or else it won't be cost effective. In addition, the high cost of the drives and continually changing standards makes it difficult to replace in the future.

The cloud, Usenet or your own domain and webspace are an alternative method of storage, but are only part of the 3-2-1 (3 backups, 2 local on different media, 1 offsite).

Getting back to placing the responsibility of store of yourself. My ex-girlfriend's family would charge each other for duplicate photo prints. At first I thought this was odd, but I now realize the demand and necessity of those "precious moments" diminishes once a cost is assessed for them. Everyone will line up for free, but the line will shrink once the cost it shared.

The next time you're at an event where a video or slideshow is being viewed by a group. watch the group instead of the show for a while. You'll see a lot of "SQUIRREL!" moments when the group's interest peaks only when it them involves them or their immediate group or family. Have them share in the cost of storage and the willingness to pay shrinks from 10 hours of video to 30 minutes!

Edit: I'm not the most sentimental guy (my brother and sisters meet once a year in memory of our parents), but last year when we were cleaning our family home, we looked through and distributed the family photos. Of the hundreds of photos, I walked away with 5 or 6, all from my youngest years. The rest (as well as the other family heirlooms) were placed in my oldest sister's care for storage at her home. A perfect example of "It's precious to me, but not enough to justify using my limited space and finances to keep!"

Last edited by lingyi; 10-07-2018 at 04:37 PM.
Reply With Quote
  #7  
03-13-2019, 08:07 PM
Tig_ Tig_ is offline
Premium Member
 
Join Date: Feb 2019
Location: Minnesota, USA
Posts: 23
Thanked 4 Times in 4 Posts
I shoot weddings, sometimes ~1TB/weekend. Data loss = career over and lawsuit for me. I also keep all files indefinitely since an experience two years ago: a couple whose wedding I'd shot asked me to photograph the bride's family, whose home burned down, taking their photos with it. Their original photographer didn't have copies/negatives. Storage is cheap, memories are not. Hoarding and paranoia established, here's my take.

Optical discs are dead to me. For the cost of Taiyo Yudens I could buy the same capacity of 60x faster SSD, or 5x the HDD capacity that's 7x faster.; and SSDs/HDDs can fit far larger backups in a safe.

My main data archive is a collection of HDDs in DrivePool (highly recommended) for duplication. One advantage of DrivePool over RAID is ease of adding/removing drives of any type and capacity. When space is low I move the smallest HDD to my backup pool and replace it with a new, higher-capacity HDD. DrivePool's sister app Scanner monitors and automatically evacuates drives showing signs of impending failure.

When high-capacity SSDs fall within 1.5x HDDs' cost, I'll switch. The faster you can restore data from a backup, the lower the odds of data loss.

Realtime duplication doesn't protect from accidental edits/deletion, theft, natural disaster, etc. Its primary purpose is to maintain data availability in the event of a failure. Backups are critical and must be offline to avoid ransomware etc. and offsite to avoid the aforementioned events, yet accessible enough you won't neglect them out of laziness.

Sorry for my late reply, but this is important stuff and I hope it helps. Check your drives' SMART health and back up your files regularly. RAID is not a replacement for backups.

Last edited by Tig_; 03-13-2019 at 09:03 PM.
Reply With Quote
  #8  
03-17-2019, 02:23 AM
Winsordawson Winsordawson is offline
Premium Member
 
Join Date: Sep 2010
Location: Behind you
Posts: 291
Thanked 7 Times in 7 Posts
This is relevant to the other thread, "HDD vs SSD in 2019", but what about Amazon Glacier or Microsoft Azure? They offer extremely affordable rates for archival data with redundant storage, though their technical support is not cheap. There is also Sony Optical Disk Archive, which is a tad more expensive at $7k for a drive.
Reply With Quote
Reply




Tags
archive, data backup, longevity, tape backup

Similar Threads
Thread Thread Starter Forum Replies Last Post
Advice: Archiving 8mm, VHS, MiniDV tapes (long-term project) AlexCatley Project Planning, Workflows 2 03-11-2017 03:03 AM
Max temperature to preserve optical discs long-term? samus87 Blank Media 8 12-27-2015 06:36 AM
VHS Capture & long-term archival suggestions... naripeddi Project Planning, Workflows 4 12-08-2011 10:06 PM
Long term (archival type) storage? admin Blank Media 11 01-07-2010 01:10 PM

Thread Tools



 
All times are GMT -5. The time now is 12:08 PM