HDDs: large capacities for long-term storage?
 

Hi all,

I've been saving files to 2TB external HDDs (currently 6 of them – 4 desktop Fantoms and 2 pocket-sized WD My Passports) for most of the last five years. I'm at the point where I may need to go to a 7th. My priority is long term storage of considerable amounts of movies, tv shows, photos, documents, etc. Before I jump for another 2 TB, I wanted to run down some advice I got here years ago and whether it's still the same advice, as I'd like to consider more convenient options if it means no backtracking on protocol. Let me explain.

So what I remember is that reliable capacities are 1, 2, 4 and 8 TBs. Ones to avoid are 3, 5 and 6. And lower capacities are a stronger choice presumably because less data is packed onto the same sized platters. For this reason I've only gone with 2 TBs - even choosing the pocket-sized portable drives over full-sized desktop externals that don't come in 2 TB configurations much anymore.

Here's how I see things. I'm at the point where my data isn't likely to top out a 7th drive anytime soon, so 14 TB total should be enough for a long while. I'm a little concerned about having too much data on the pocket drives because they're so small (everything is backed up so no worries there), and I'm a little tired of swapping out and managing so many external drives – though would absolutely continue doing so if it means I'm maintaining the same level of theoretical protocol. But it would be nice to move to some larger drives if it won't "break protocol." So here are my questions.

• Is the information I spelled out still true? For that matter, did I remember it all correctly?
• Is 2 TB still a stronger choice than 4 or 8 for reasons of less data? Or are larger capacities (10, 12, 14 included) just as reliable?
• Are the pocket drives less trustworthy than full-sized desktop externals, or are they all the same reliability at this point (capacities being equal)?
• If going with large capacities, should I do one large drive or split into two to reduce the burden of a large amount of loss on one? Or are 2, 8 and 14, for example, basically all the same risk at this point?

Thanks in advance. If I think of other questions I’ll add them. :cool:

Note: As of today, 06/05/21, >4TB hard drives 3.5" drives are commanding a high (up to 2-3X) premium because of Chia cryptocurrency mining (which uses hard drives). 2.5" drives which max out at 5TB are still normally priced for now. It's unknown how long this extreme demand will keep drive prices high, but there have been isolated cases (Costco and Bestbuy) with pre-Chia sale prices.

There is no such thing as guaranteed long term backup. Any storage media can fail at any time, for any reason, with or without notice.

Backup, backup, backup. Ideally 3-2-1 backup. 3 copies of your data, 2 copies on the same or different media (hard drive or optical discs in your case, but not SSDs or flash drives which are not archival), 1 copy kept offsite, physically or cloud. Check, verify and copy your data to new media every few years.

Edit: While there's no reliable metric for how long a hard drive will last, five years is good lifespan. Just because the drive is spinning doesn't mean it's in good condition. Check the SMART status of your drives with a utility like CrystalDiskInfo. If you get a yellow caution or red warning, backup your data immediately! You drive may live another five years or may fail immediately! SMART isn't 100% reliable, but is the best indicator or a drives health that we have. I check my drives every few months at the least and always check my drives if they haven't been used for a while. A drive that is green today, may be yellow or red tomorrow. True experiences.

The My Passports, and all WD and Toshiba portable drives have a major drawback. The USB interface and USB port are integrated in the the mainboard unlike Seagate which has a detachable interface. This is important because if/when the interface/port fails, recovering the data requires conversion to a SATA (regular hard drive interface) that's $$-$$$. This can be avoided by have proper backup(s).

There is a tiny sliver of fact in this statement, but it's long, long past. Years ago, the largest single hard drive platter was 1TB, with multiple platters added to achieve the required capacity. So single platter 1TB drives were quieter and supposedly more reliable. Now, the bigger issue is that all external 2.5" drives and all <10TB WD and <8TB Seagate externals contain SMR (Shingled Magnetic Recording) platters which have slower writes/rewrites due to the writing strategy and platter technology. Not a major issue if you're mainly using the drives for read only.*

Back in 2013-14, there was a single 3TB Seagate drive that failed at higher rates than other 3TB drives, including Seagate. This is where the 3TB and Seagate SUX misleading mantras come from. The drives are long off the market and the majority of them are long dead.

There was a slight issue with the initial 5TB drives (I believe Seagate) that used the then fairly new SMR technology to reach that capacity. I had a few 5TB desktop externals that failed relatively quickly, but I don't remember the brand. 5TB is the largest capacity available in 2.5" form factor and has fallen out of favor (both availability and low price) in the 3.5" category. FWIW, lordsmurf uses and recommends 5TB Seagate portables.

Largely no, as stated above.

In general, no. See my statement the necessity of backups. The exception is that portable externals, per their intended use may be more prone to failure because of the shock or physical damage (head crash) while moving the drive while it's still spinning.

If you have to choose buying two or three smaller capacity drives to ensure you have backup(s), definitely go with the multiple smaller drives. The catch is that price per TB decreases as you go with larger drives. For example, a 2TB portable is ~$55 and a 4TB portable is ~$90. For desktop externals, the best deals, ~$15-20/TB are for 8TB+ drives.

There is no good, better, best when it comes to hard drive capacities or brands (WD/HGST, Seagate, Toshiba). Premium priced NAS and Enterprise drives are meant for heavy datacenter and large business use which can't be duplicated or provide an advantage for home use.

Buy on price. Buy two or more for backup(s). Buy often.

If you want more opinions than you may get here, visit reddit.com/r/datahoarder. Warning, if you figure out my nick there, fair warning, I'm very rough and opinionated there because unlike most video topics, I'm very familiar with hard drives. Though as I very often say, my personal experiences with hundreds of drives over the years is meaningless compared to the billions of drives in use today.

Thank you so much, lingyi. This is a big help. So, ultimately, I guess what I’m wanting to know is how you’re defining "smaller" here. Is the 8 small, or is 2 better because it’s even smaller? Why do you advise 8 over 14 (my interpretation) when doing multiple drives, if there’s no good/better/best?

Sorry that I confused you. I mean multiple smaller drives vs more capacity if it means one or no backup. Better to have 8TB of storage with at least one 8TB backup, rather than 16TB of un-backed up data.

Edit: In your case, I'd go with a single 14TB+ drive or multiple smaller drives equal to 14TB+ and use your existing drives as backup.

Edit 2: Given the current high prices right now and my recommendation to backup your data immediately if you don't already have it properly backed up, is, assuming you're in the U.S., getting three 5TB portables for ~$90 each. Of if you're lucky, BestBuy will have another sale like last week when a 16TB Seagate external was ~$300. Sold out that day!

With the exception of 8TB Seagate externals and 1TB+ portables, which contain SMR drives, there's no practical* difference between drives of any size.

*At datahoarder there's an ongoing debate about which is better, air or helium filled drives. However this debate is centered around helium drives being slightly louder and possibly being more likely to fail as the helium slowly leaks out. The latter reminds me of stories of the plasma in plasma TVs needing to be refreshed after X number of years because no seal is absolutely gas tight.

Personally, I'm in the less drives = less chance of potential failure, at the risk of more data lost when, not if they eventually fail. Several years ago, I swapped out all my main 4TB drives with 8TB+ drives and am slowly replacing the 8TB drives with larger ones.

I see. So if I have this right:

• Two 8's are better than one 16 – though in my case it'd be 12-14 TBs total spread over two 8's, with one set per physical location.*
• Two 8's are no different than seven 2's running around making me crazy.

*My backup strategy has been a complete set of data across six 2TB HDDs (with a limited number of data DVD-Rs). One full set of six + discs at home, another full set of six + discs in storage. And I'm hoping to do a third set at another location soon.

Yes.

Glad to hear you have backups. Good going!

I and others at datahoarder cringe and criticize when posters brag about having X amount of data without backups.

I like those 5tb Seagates, but I've lost 3 in the past 2 years. That's too high. Admittedly, part of it may have been cooling, so I now have them sit flat on chucked Fantom enclosures (chunk of aluminum). If I do anything that causes more than 500gb of read/write sustained, I turned on a fan, blow over the drives. I use my 5tb regularly, not as mere backup or storage.

I recently bought a 8tb QVO, because NAND is supposed to go up. That's my new video editing drive. I know it won't be as fast as my EVO, yet still faster than the HDDs. I need my EVOs for something else now.

(On the plus side, I bought stock in Micron, Seagate and WD last year, when it was at bottoms. So at least my IRA will be happy from the price hikes. All of these stocks have at least doubled now.)

If you don't need the portability, have you considered shucking the 5TB drives and putting them into a multi-bay enclosure for better cooling? You may not need it, but I recommend this 2.5" to 3.5" adapter for making sure you drives are secure: https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1. Note that most adapters are meant for internal PC bays and don't relocate the SATA connectors in the correct position to be used in a 3.5" bay.

Hey again all,

Before I dive into another five year cycle of HD backups, I just wanted to see if anyone else wanted to chime in with comments. Here's what I've learned:

• TB capacity configurations don't matter much anymore (2 ,3, 6, any configuration is equally okay).
• Even-numbered TBs (2, 4 and 8) are preferrable to 3 or 5 = not really true anymore.
• Employing larger capacities (and therefore fewer individual drives obviously) is perfectly fine for five-year storage intervals nowadays. So 8 TBs are just as reliable as 2 TBs. One pair of 8's would be accessed throughout the year, another pair of 8's in storage just 2-3 times per year.
• Brand/manufacturer doesn't really matter. WD, Seagate, Fantom, Lacie, etc. All the big names are pretty much equally reliable. I'm of the mind that I'd like to split into two, across at least two different brands, just in case there's a manufacturing issue with a particular model discovered later on.

Any additional thoughts?

These are the ones I'm considering:

https://www.amazon.com/Book-Desktop-.../dp/B01LQQHLGC

https://www.amazon.com/Fantom-Drives.../dp/B073JF992B

https://www.amazon.com/LaCie-Profess.../dp/B07MK3T28F

https://www.amazon.com/Seagate-Deskt.../dp/B07CQJBSQL

Historically I'm a little leery of Seagate, but thought I'd include it here. The Lacie employs a Seagate Barracuda enterprise drive, so there's that. Does anyone know what Fantom uses nowadays?

Thanks!

I don't know that configuration ever mattered at all. The problem was certain configurations had limited drive models that existed, and for whatever reason those were largely bad too fast. For example, all the 3tb drives seemed to massively suck, but I doubt 3 had anything to do with it. It was more than we had few 3tb drives whatsoever. In the case of Seagate 1.5, 3, and 6 were all equally bad, but probably because those drives just had multiple platters/controllers. Again, very few of that size exists, far more common was 500/1/2/4/8. Modern 10/12/14/16 is SMR, and has no relation to older 1.5/3/6. The 1.5/3/6 was a "general rule" because it was easy. I may have even done that, though I probably said "avoid Seagate 1.5/3/6" drives, because it was mostly only Seagate with that issue (or that drive size).

What's always been the case is whether drive lines were found as being good. Not silly anecdotal evidence ("I bought 1 drive, and it was bad, wah, don't buy this drive!"), but vetted advice from somebody like Backblaze. What you'll find is that all drives have a % of fails, and most are low in the 1-2% ranges. And the more drives they test, the lower the % always go. Very rarely does a drive have an excessively high % of fails, and that was mostly those 1.5/3/6 drives almost a decade ago.

A drive is a drive, and I've never seen any evidence that larger drives are worse. That's going back to the 80s. I still remember when people poo-poo'd 1.2mb 5.25" floppies as worse than 360k. They were morons. Those same morons exist. Every new wave of drive sizing gets shunned as "worse", but it's all nonsense. I tend to get more comfortable with newer drives, as they usually lean from past mistakes. I just don't want to be the 1st in line, because the Seagate 1.5/3/6 left a bad taste, I'll let others be the beta testers. I bought Seagate 1.5s early on, and lost all but 1 of them within a few years. The last Seagate went for years before failing, and by then I wasn't even using it anymore. (I did, however, buy the 8tb QVO SSD here recently, and it's barely 6 months old now, but I feel more confident about Samsung SSDs than any HDD.)

There are only 4 drives manufactuers: WD and Seagate (most drives), and HGST and Toshiba (less drives). Everything else is 3rd-party rebadge/enclosures. LaCie is actually owned by Seagate, and will almost always be Seagate drives inside. Fantom has historically been good, uses any/all drives, but mostly WD.

When it comes to drives, what actually matters most is how well (and easily) data can be recovered. So plan accordingly, research the exact drives a bit closer.

I have
- 2x 8tb Seagate USB
- 1x 16tb Seagate Exos (internal, unplug when not being used)
- multiple 5tb Seagate for both backup, and regular use (for now)
- multiple 2tb Fantom that are being retired, drives reused, USB2 enclosure being shucked

I'm not a data hoarder, but I am serious about backing up data, always have been.

For a few years there, I slipped in my backup diligence, and it cost me $1k to recover it (Kroll Ontrack was used, about 10 years ago). They recovered 99% of my data from a 400gb (WD?) drive that during a backup. Yes, you read that right, I was finally backing up the data, and it failed right then. Back then, SMART didn't work all the well, not reliable, so I'd set my computer to ignore SMART warnings. It's still not perfect, but way better these days, and I have CrystalDisk monitoring all drives all the time now.

Over the many years of owning drives, WD was always the worst and noisiest of drives. So it made noise all the way until failures, what a drive! Hitachi was also mostly noisy, and I avoided those. Seagate has always been quiet, and lasted the longest of all drives. I still have IDE Seagate drives, while everything else has failed.

Thanks LS!

Do you know if there's a guide anywhere that breaks down what models use what actual drives inside? I've been looking for that for a while.

Also, do you agree with lingyi regarding SMR? I understand it might result in slower write times than CMR, but is there any reason to doubt reliability?

It's hard to avoid SMR.

The 5tb are SMR, but mine did fail somewhere after 4tb was used. I think my Exos is the X16 CMR, not SMR.

As stated, if mostly used for read, not an issue. I was using some 5tb for read+write, and it wasn't really slow. But I'm moving all of that to my new QVO, along with cleaning off my pair of EVO. And the 5tb will be entirely for storage and read.

Correct, the higher capacity Seagate Exos drives are (probably all) CMR. I have an older 8TB Archive and it's an SMR; it's harder to tell now what is SMR, but I think as new technologies come out that's being phased out.

There was a big scandal too where some WD RED NAS drives were actually SMR, and were getting terrible performance for heavy read/writes, despite being advertised as such
https://nascompares.com/2020/04/16/y...-need-to-know/

The QVOs, are those SSDs? I was looking at some high capacity SSDs but the pricing is too much right now. I don't think they're worth it for very idle data, archival data. Also because the high capacity SSDs 4TB+, some use 4 layer NAND meaning less durability but cheaper cost.

Not really any doubt in reliability, like others have shared any drive can fail anytime so if you care about your data, buy two drives, or use a reliable cloud storage provider. My SMR drives are fine, lasting many years, but then again I don't do many writes to them because of the poor speeds.

Hard drives tend to prefer not being turned on and off too much, it stresses the internal components out like anything else.

Yes, Samsung 8tb QVO SSD. :congrats:

It's worth it for heat and noise reduction. It writes at least as fast as any HDD, and I'm not using mine for archiving. It's for video editing large lossless files. Write is about 150-250 MB/s for tandem copying (a 115 MB/s USB drives, and a 4tb Samsung EVO, as the source with lossless files). That's the most aggressive I'll ever really use it, in terms of the data flow. It beats any HDD, even in a RAID.

When prices fall, then I'll add 1-2 more for archive. But it could be years before that happens.

HGST (Hitachi) is owned by WD and so there's only really only WD/HGST, Seagate and Toshiba. Some of the WD externals have HGST drives in them.

As I recall, the issue with 1.5, 3 and 6TB drives was the new 1.5TB platters, which I believe were used in the notoriously bad 3TB Seagates.

All current 500GB+, 2.5" drives are SMR. With the exception of the 1TB WD Red NAS drive.

Here's a list of all the SMR drives, which came out after WD and Seagate were caught "submarining" SMR platters into models that were previously CMR>

https://blog.westerndigital.com/wd-red-nas-drives/

https://www.seagate.com/internal-har.../cmr-smr-list/

WD just settled a class action suit for $2.7mil in damages for using SMR in drives without notice.

Note that SMR isn't inherently bad, it's [fine] for its intended use, write few, read many and for typical home use where fast write/rewrite times aren't critical. The issue came about because when used in RAID setups, the slow rewrite times when rebuilding the RAID [,to check for and correct bad files, the SMR drives] would cause the operation to fail [because of the slow rewrite time]. This is how it was discovered the WD Red NAS drives that were previously CMR were now SMR.

Significantly, in the settlement, WD acknowledges that the [SMR NAS] drives are not suitable for use in NAS setups using RAID, despite their being sold as NAS drives.

Note: NAS and RAID are [not] the same thing. NAS is for network accessibility. RAID is for data redundancy (and speed in certain configurations*), but not a backup.** You can have one without the other. RAID was introduced a decade (1987) before NAS, late '90's.

*Striped RAID writes/reads from two or more drives simultaneously, splitting the data stream between them and increasing the write/read speed. This is largely obsolete because of SSDs today. Though it's possible to Stripe RAID SSDs for increased speed.

**RAID is not a backup. RAID helps protect the data within it by checking (resilvering) the array and correcting any errors within the files using data from the additional (if any) parity drives. It's not a backup because if you overwrite or delete any file or format the drive, those files are gone just like a single drive, same as with a format.

SMR isn't going away. It's a necessary step (and a cost savings to the manufacturer) to larger platter drive sizes. While WD and Seagate have disclosed which drives use SMR, all <10TB WD and <8TB Seagate consumer line drives, SMR is used in the largest 3.5" drive, the 20TB WD which is currently not available to home consumers.

Even HAMR (Heat Assisted Magnetic Recording) and variants of the technology use SMR to achieve the greater storage capacity. Currently 20TB now, currently available only to Enterprise users and estimated up to 50TB by 2026.0

https://blog.seagate.com/craftsman-s...p-forward-now/

And here's a site that lists the size and type of platters used, though it's not complete:

https://rml527.blogspot.com/

As I stated above, if you want more extensive, usually but not always good discussions about anything hard drive or SSD related, visit reddit.com/r/datahoarder

-- merged --

As for what drives are in externals, it could be any drive from the manufacturer's line.* Highly speculated to usually be binned drives that didn't meet the full specs to be sold as retail internal. These drives may have their firmware changed or cache reduced. They're still good drives, just not up to the full specs of those sold as internal. They may also be production overruns or drives from cancelled orders because hard drives are a commodity, with a short sell shelf life. Unless it's been sitting in the seller's stock for a long time, you'll never a manufactured date of more than a few month before you bought the drive. The manufacture cost and retail margin is too small to keep them in storage.

*As of right now, there are reports of Seagate enterprise Exos drives being found in their 16TB externals. And in 2017-mid 2018, 8TB WD externals had original and white label NAS drives, which AFAIK were full spec. On the other hand, in the past, some found Purple Seagate Surveillance drives (not meant for typical home use) in their externals. Bottom line you never know what drive will be inside a manufacturers or third party external. What someone gets today, may be different from what you get tomorrow.

Sometimes you'll get White Label drives. White Label drives are either drives binned drives or drives meant for OEM or third party resellers. They could be full spec drives that the buyer cancelled or drives that are customized for the buyer, different PCB, firmware or cache size. Again, fine for typical home use**, but may not be for datacenter and enterprise use where known quality is critical.

**At the Datahoarder Reddit, I was corrected by a member when I said that Enterprise drives are meant for use in datacenter conditions that no home user could duplcate. He/she said that at Datahoarder, some members may well have a sophisticated setup that rivals a datacenter! I humbly corrected myself. LOL!

I have an WD EasyStore 18TB, purchased December 2020, I removed the drive from (shucking). Here is SMART data (smartctl -a output):
So far so good, drive is fast and can eat writes easily. No slowdowns that I've seen.
Can't comment on the Seagates, I haven't found good HDD deals ever since that stupid chia crypto-nonsense came around...

:yahoo2:
And you have an excellent use case for it, big files that need fast access. Excellent! I was thinking to replace some 8TB HDDs that have WORM data (write once, read many) but until prices drop I'll sit as you are doing.

This is relative to what you're writing and how.

All drives write speeds slow down with smaller files and are faster with larger multi-GB files. I get constant ~130-150MB/s writes when I'm transferring my videos.

Also, writes will always be slower as the drive gets fuller, as hard drives write from the outside in.

It's on rewrites of existing files that SMR really shows. Because of the shingling, blocks before and after the data to be overwritten must be erased and rewritten. Just like having to replace shingles on a roof. If you're working on the edge, you're overlapping only one set of existing shingles. If you're working in the middle of the roof, you have to remove and replace X number of surrounding shingles.

The drive cache also has a part to play in slowdowns. I've never bothered with it. But reports are that if you're experiencing a slowdown, if you let the drive slowly clear the internal cache, which can take a day or so, your write speeds will return to normal.

Bottom line is that for many (most users), including myself, the slower write/rewrite speeds of SMR isn't an issue. Yes, it takes longer to write/rewrite files large files, but I'm usually not in a rush. If I am, I use a SSD. And when I transfer a large amount of files, hundreds of GB or multiple TB for backups, I just set it up and walk away.

And the fact is, you can't get away from SMR with current 2.5" drives (except for the single WD NAS drive) and if speed is a concern, you're better off using faster, larger cache 3.5" CMR drives or SSDs.

As I said above, SMR is here to stay as larger capacity drives require it.

THANK YOU. I forgot to ask about this when I started the thread. I had concerns because the manufacturers don't like to tell you the manufacturing date on the packaging anymore, and my past experience getting it thru customer service was successful but so onerous I gave up even though I really cared. I assumed drives couldn't be sitting for too long but never had a reason why beyond the nature of merch moving quickly off the shelf. :congrats:

The manufacture date in on the label of the drive. For external drives use the below. The manufacture date is the default for the start of your warranty. Note that this may not be actual manufacture date. They sometimes as a month or two to allow for the full warrant. Another reason they usually don't sit on the shelf. If your purchase date is a few months after the manufacture date, you can usually use that for your warranty start if you present your receipt and/or register the drive with proof of purchase.

https://support-en.wd.com/app/warrantystatus

https://www1.hgst.com/warranty/index_gtech_serial.do

https://www.seagate.com/support/warr...-replacements/

https://toshiba.semicon-storage.com/...y-support.html

3-2-1 Backup. Local and cloud or physical offsite. 3 copies of your data, including your main drive, 2 on the same or different media and 1 kept physically offsite or cloud. The reason for the offsite copy is in case of catastrophic events such a s fire, flood, hurricane/tornado, power surge, theft, etc.

lingyi, do you know why Toshiba external desktops seem to have disappeared? It doesn't seem likely to be the chip shortage as everyone else is doing fine and prices seem to be coming down.

I'm planning to join Datahoarder and was going to ask there, but thought I should here too. Thanks.

3-2-1 Backup. Local and cloud or physical offsite. 3 copies of your data, including your main drive, 2 on the same or different media and 1 kept physically offsite or cloud. The reason for the offsite copy is in case of catastrophic events such a s fire, flood, hurricane/tornado, power surge, theft, etc.

I'm going to pass along a bit of a secret. :D

Check your local Walmarts as they may have external drives, Seagate and WD on closeout. A month ago I got three 8TB WD MyBooks, 1 for $116 and two for $80 each. There a recent report that someone got a 4TB Seagate Expansion for $21. https://www.reddit.com/r/DataHoarder...21_at_walmart/Note that this a huge YMMV as closeout prices vary widely.

Edit: As I explained above, this is definitely SMR.

This site allows you to check the local inventory of anything sold at Walmart and Target. Here's the URL for the 4TB Seagate: https://brickseek.com/walmart-invent...r?sku=44474061

Edit: At least for Walmart, using the UPC won't give you any results. You have to search for what you want at Walmart, then use their SKU.

I happened to see and by the MyBook at one location, then used Brickseek and saw that another location had them for $80. Got the last two. Brickseek isn't perfect. Not only can someone buy the drives before you arrive and sometimes if it says Out of Stock, there may be some in-store. When I bought the two 8TB drives, the 6TB drives were $95 and I thought about buying them, but didn't.

A couple of caveats. As I stated above, the default warranty date is the manufacture date and these drives have been sitting on the shelf for a good while, that's why there's on closeout. You could try and use your receipt to get a longer warranty, but at these prices, I don't worry about it.

The other potential issue that MyBooks use encryption on their USB interface. Which means if you shuck the drive, it won't be recognized outside the case. You'll have to format the drive without the interface to use it.

Those 8TB My Books are down to $174 at Amazon. I price-matched two out of my local Best Buy this afternoon and feel good about it because those drives haven't been sitting there long. I know because they were recently out of stock and of course were full price ($229), so I'm pretty confident in my purchase since these are for five-year backup cycles.

Now off to run diagnostics. :cool:

Good going and good backups!

I'm restoring a backup as we speak.

People don't often talk about this aspect. When you restore, don't restore everything in one go. That will overheat the drives. I'm doing 500gb chunks, short rest (10-15 minutes or so), 500gb more.

Just now getting around to installing my new Samsung 8tb SSD QVO, restoring the data from the failed Seagate 5tb, to a new Seagate 5tb, as backed up to the Seagate 8tb. I had to order a quality SATA power cable splitter from Amazon, ran out, so can't access my Seagate 16tb, swiped it for the SSD.

The backup was a bit stale, so I lost some data, but nothing (I don't think, I hope) that cannot be redone within a few days.

Overall, it takes some days to get data restored. I also audit everything, verify, before calling the restore a success.

You can automate all of this with $$$$ and NAS, but I don't want that. I want backup drives unplugged, no power, no connection, in a drawer until needed (routine backup or restore).

But come back here, ask questions, spread knowledge. ;)

I'm there as well, lingyi and nicholasserra are there.

This may be true for externals, but drives in a PC or in a properly cooled multi-bay enclosure can be safely run for days. When I transferred 9TB to my new 14TB drive, It took almost two days and I had to do it twice immediately after since I brainfarted and formatted the destination drive immediately after the transfer was done! :eek: I routinely copy TBs of data from drive to drive without stopping.

Part of my and others routine at DataHoarders is to stress test new drives by running a program to write and read to the entire drive or what I do is completely fill to the drive, then format and write the data again for actual use. Note that I always shuck my externals and use the bare drives in my multi-bay enclosures.

+1 to your process! Verification can be done automatically, immediately after the write. I use Teracopy, but there are other programs that can also do the job. Once the copying is done, I use ViceVersa to verify that both drives contents are identical.

Oh...don't get me started on NAS and RAID which are considered a necessity by many at Datahoarder. :mad4: :D

Not everyone, including myself has a need or desire for a NAS or RAID. I currently have 116TB RAW, ~80TB used, housed in non-NAS, non-RAID enclosures. I have two setups, with the second a mirror backup of the first and I'm in the process of create a third full mirror set of drives (I currently have ~50TB full backed up). I have enough space to backup everything, but want to save to an exact mirror of my two other setups, with drives ranging from 8-14TB. Stupid Chia put a hold to my plans to get 16TB drives to replace some of my 8TB ones.

Yeah nicholasserra's great thread (with your as always insightful and excellent posts) about video capturing is in the Datahoarder Wiki.

In regard to HDD heat...

I do video. Video has audio. I cannot wear headphones. Audio needs quiet. As such, my computers are not wind tunnels with mini-turbines. I go for far more passive and well-thought-out cooling, which is silent to quiet. So "properly" (fans) cooled doesn't happen. And yes, even slow-moving 140mm fans create wind tunnels that make audio restoration impossible. The HDD make enough noise already, and that's using quiet Seagates.

When I have to copy large amounts of data, I put a regular fan next to the system, and that makes wimpy little spinny computer fans look quaint and amusing, in terms of cooling power. Regular 30s (C) temps drop way down in the 20s. The copy processes never get over 40.

Almost every time I have lost a drive over the years, heat was the problem, and during copy processes. Drives get massively hot during long copy sessions. And past failures happened on non-video systems as well (ie, bay of fans cranking away).

You also need to remember where I live. It's hot. We have AC, but it's hot. Today is in the 90s (F), the AC can only get into the highest 70s (otherwise it'll have problems, in addition to more energy costs). Add to that the state warning all residents to cut power use, and turn up the AC. I don't ever want to go through another rolling blackout, that sucked.

So, for me, due to experience, better safe than sorry. Losing a drive sucks. Losing a new drive because you pushed it too hard would suck more.

I thought Smurf Village was always cool. That's why you're all such a lovely blue! :P

So I have the chance to pick up two more 8TB My Books from Best Buy, which would have me all set for the next five years. But I'm still feeling a couple things out. One is trusting my entire digital library to just one brand and model. I know there's no need for brand concern, but it seems like a good, common sense stance to spread things across more than just one manufacturer. The other thing is that I'd like to support the one remaining manufacturer who isn't WD or Seagate, but Toshiba doesn't offer 8TB externals (that I've seen).

They do have these 4TB portables:

https://www.amazon.com/Toshiba-Canvi.../dp/B08JKCWRTD

But I'd have to deal with four of them running around instead of just two 8's. And they're portables. Though I have enough data that I can probably just load them with write-once, read rarely content (storing them, either locally or offsite, means they're being accessed just a few times a year). I can leave newer data for the My Books.

The thing about Toshiba is that their drives don't seem as widespread, nor their lines very ambitious. They've apparently stopped making desktop externals, B&H doesn't carry Toshiba externals of any kind, and Office Depot will only ship them. Amazon and NewEgg have them though.

As for capacity, I suppose there's also the possibility of purchasing two internals like X300 8TBs and enclosing them, but that hardly seems worth the effort (and maybe not the price). Either way I've never done internal/external swaps before (shucking et al).

It might just be easier with the portables, or easier still with all My Books.

As I posted above, Toshiba portables have the USB interface and port integrated into the mainboard like WD. I'm not sure if they can be modded to a SATA port like WDs if/when the interface or port breaks. May not be an issue, but IMO, why take the risk, trouble and cost when Seagate portables are regular SATA with a detachable interface.

IMO, you're overthinking and over complicating things. No matter how carefully you plan things, Murphy's Law has a way of happening. A month ago I was creating a second backup of one my drives. I wasn't in the room and somehow a folding table fell and hit the enclosure I had the drives in. The drive I was writing to went from 0 bad sectors to 7700+ in an instant. Somehow, the three other drives are okay.

I highly recommend you visit Datahoarder. Start by reading this from the Wiki: https://www.reddit.com/r/DataHoarder/wiki/index, do a search for backup strategies and browse at least a couple of weeks of posts because your questions have been asked and answered numerous times there.

You'll find much of what I've posted here, repeated there (especially by me <GRIN>). I'm confident you'll find that while specific recommendations may vary, everyone agrees the most important thing is backup, backup, backup! :congrats:

Here's a couple of searches to help:
https://www.reddit.com/r/DataHoarder...&restrict_sr=1

https://www.reddit.com/r/DataHoarder...&restrict_sr=1

Best guess is Seagate and WD have been in a price war for years and Toshiba chose not to compete. Toshiba a far third in sales behind Seagate and WD. https://www.forbes.com/sites/tomcoug...h=1002e9b37aae

Because of the low price of externals and the probability of their being mostly binned drives, Toshiba doesn't have enough volume to make it profitable at the prices Seagate and WD are selling their externals.

Edit: If you're wondering what Toshiba does with their binned drives if they're not selling them as externals, they probably sell them to companies like White Label and Water Panther who sell them as unknown manufacturer refurbs or sold for use as OEM drives in PC builds where there's no guarantee of drive specs or type of drive. They may also be used for RMA, which is usually a refurb.

FYI, after years of speculation that not much profit is made off externals, I finally got confirmation from the BackBlaze employee who posts at Datahoarder, that they're able to buy Enterprise drives for less than external. But, that's only because they're finally able to commit to buy 10K+ drives a month.

In their blog, Backblaze has stated they've shucked externals in the past because they were cheaper than their distributor's price. But this was then they were buying in lots of 100's or low 1000's.

In 2011/2012 when the Thailand WD factory flood caused drive prices to skyrocket, Backblaze experimented with buying 40 or so 8TB Seagate Archive externals because they were cheaper than what their cost for internals were. The experiment was quickly ended as the drives failed quickly because it was outside of their intended design.

Note that a Backblaze failure isn't what a consumer considers a failure, which is generally when a drive isn't recognized or files show signs of corruption. Backblaze considers a drive failure based on their very strict custom SMART stats that predicts when a drive will possibly fail. This is one of the reasons I take their published stats with a huge grain of salt. Thankfully, they state that their stats are only for their particular usage.

Thanks again, lingyi. I think you're right. I'm overthinking it. I'll probably try out a pair of 8TB Seagate Backup Pluses. Thinking back on it, I've been reluctant to run with Seagate because their portable enclosures seemed cheaply built, and they're commonly mentioned as failing sooner for more people. Just gave me a bad vibe. Maybe the desktop externals are a different experience (I have an old 500GB FreeAgent from about 2009 that still runs nice and quiet last I checked).

In terms of confidence in the full drive, my routine has been to run a complete diagnostic using HD Tune Pro's Error Scan. Obviously that doesn't write to the entire drive, but it does read every sector and verifies whether it's "ok" or "damaged." Is that not good enough?

Apologies if this is searchable on Datahoarder, but I wanted to respond to your quote directly and ask because I've thought about doing what you do, but it just seemed like a lot of work with no reasoning I could see that it's advantageous over simply scanning each sector. I guess what I told myself was, "if all the sectors are 'ok' I should be able to fill the entire drive successfully. Great. So I do that and then what? The drive being full doesn't seem like valuable proof over the Error Scan's 100% ok." But maybe it is?

Edit: These are good questions. Please don't hesitate to ask them. However, as I've said, don't overthink and overanalyze it. You can only do so much to try and ensure your data's integrity. Do the best you can and move on. Plan for a five year retention, but be prepared to perform upkeep and revisions at regular intervals. And weigh the cost and importance of your data against other things in life. You may find that paying for and stressing about your data may not be as important in the grand scheme of things than you think it is right now.

As lordsmurf points out in his post above, writing to a hard drive generates heat and stresses the drive. Better to stress and test a new drive for a few days/weeks, when it's more likely for any electronics to fail than. This is my part of my logic for not using new drives for archive. If it's going to fail, I'd rather have it happen during the warranty period rather than storing it away and finding out it went bad after the warranty.

I don't know the technical aspects of it, but to my limited understanding, only by writing and reading data to a block/sector can it be found to to be bad. I remember watching with fascination, Spinrite writing and reading each block/sector numerous times before it was finally declared uncorrectable.

Here's what others at Datahoarder say: https://www.reddit.com/r/DataHoarder...&restrict_sr=1

Edit: I've had several discussions at Datahoarder about the difference between readable vs bit accurate data. Along the hard drive data read chain, there's built in error correction. https://www.google.com/search?q=hard...hrome&ie=UTF-8

The end user result is that the read/copied data is identical to the original. So what's the issue? The issue is that the error correction may not be be 100% accurate all the time. A few missing bits or bytes may not make a difference in the playback of a movie or audio file, but it may be critical if it happens in a legal document or videogame, causing game to glitch or freeze at a critical moment.

My realization of the importance of having a bit for bit accurate copy of the data on your hard drive, optical disc or tape came when I first used EAC (Exact Audio Copy) to rip my CDs. In Accuraterip mode, it can be set to compensate for errors in the CD reader and will make multiple attempts to make a exact bit for bit copy of a track. https://www.exactaudiocopy.de/en/ind.../accurate-rip/

Is the difference made by Accuraterip noticeable by anyone but Golden Ears? Probably not. But like Golden Eyes like lordsmurf can see and identify video artifacts that I just can't see, the data accuracy and integrity is there for those who who can appreciate it.

Great point. :cool:

This makes sense. With new drives, I'm intending to load them right away so much of the writing will be happening during the warranty for sure. I'm at 12TB total right now, which means putting about 6 on each 8TB drive. So maybe I'll fill them with a redundant 2, then delete after. Or maybe I won't sweat it if I reevaluate how important this is, like you suggest. All my data to date can be restored from the older drives being retired anyway.

I haven't dipped into software to verify data accuracy (though I do use EAC for CD rips). Mostly because I take the time to check the bytes "by hand," checking Properties on PC. See attachment. It's a tedious extra step I suppose, but if I've done drive diagnostics and the OS always confirms bytes are identical after each and every transfer, I'm satisfied and stop worrying there.

Thanks again for the Datahoarder links. I'll take a look at those too.

A file size compare, especially on large files like videos can be deceptive. I've compared videos of the exact same file size and one is unplayable.

The proper way to compare files is to run a CRC (Cyclic Redundancy Check) and optionally save the HASH for future reference. https://computer.howstuffworks.com/encryption7.htm

I use Teracopy with verify on when copying files, It will show any files that didn't match exactly, then you can redo the copy of those files. I then use ViceVersa to compare the directories/folders to see if the files sizes match exactly. If I want to compare files that have already been copied, ViceVersa with a CRC and generate a HASH.

BTW, I never directly move important files, I always copy, then delete after the copy is done. This minimizes the possibility that something (like power loss) can interrupt the move and leave the original file and copy corrupted.

Will the HDD get mold?
Can I put the HDD in a paper bag in a box? Will that help? :laugh:

Just to add on this, yes, I can see most every flaw...

... but I'm a pragmatist, and will ignore things that don't interrupt enjoyable watching. It's errors that are distracting that need to go -- such as timing wiggles, lost color data, tearing, tracking, audio skew due to dropped frames, etc -- that must be fixed, using a mix of hardware correction before capture, restoration post-capture). Video can never be perfect, but it should be excellent.

I often make comments like "mask it and move on", and yet some folks get some ridiculous idea that I'm a video purist. Nope! Never have been, never will be. There is always an acceptable % of video errors, and an unacceptable %. There's actually few in-between errors. There are people at VH who want to filter video in Avisynth, but the source looks/sounds perfectly fine already -- they're nuts (extreme OCD), and too much time on their hands.

Now then...

When my data was recovered from my failed drives all those years ago, I got back 99% of it. Some files were not 100%, but also 99%, and had tiny video errors. Not too different from digital broadcast transport stream drops. These blips annoy me, because I knew it was 100% long ago, and I've replaced what I could (some is rarer video), but I can still enjoy the other 99%.

Missing data isn't really the same as video errors (or audio errors). With some files, any 1 or 0 that is missing renders it worthless. So testing is imperative.

I also use Teracopy, have for years. :)

Must...resist...temptation...I've been good! :P

I've often not been in a good mood, for months now, but you often cheer me up, at least for a few moments. :D

I've been figuring out a testing method for the new drives, and looked over material from Datahoarders to balance thoroughness with not going overboard. A lot of folks there are talking internals and RAID arrays, neither of which applies to me. Full reads and writes come up frequently though. A user named HittingSmoke suggests "Smart for testing the disk surface, fio for testing the mechanics." I think he means a long Smart test for checking the disk surface, which can take hours or even a full day. I don't know about Fios; never heard of it, it seems intense and it looks like it's only for Linux (or at least there's no GUI version for PC).

So I'm thinking:
1. CrystalDiskInfo - to check Power On Hours to confirm it's a new drive.
2. HD Tune - for Error Scans, et al. What I'm used to and trust, for what it's worth.
3. DiskCheckup - for a long Smart test. Maybe.
4. Then filling the drive for a full read/write.

My questions, which don't seem to be on Datahoarder, are:
1. Is a long Smart test necessary with the above, or is HD Tune plus a full read/write good enough?

2. Regarding heat, is it a problem testing an external for up to a full day?

3. lingyi, you mentioned you do a full read/write, then format to write again. I saw someone else mention this too. Why reformat?

Thanks again for indulging me. Once I have a good MO going, I won't need to ask again. :D