Universal compression scheme for data for archiving ?
 

I have been burning my raw data; photos & movie files from my digicam onto Taiyo Yuden DVD-R discs, but as my data increases, the amount of discs increases, I keep the discs at an office in my town as offsite storage, this has worked the best for now.

I could go to double layer discs, but they cost more per disc, & not as economical as DVD-R's. I have looked at blue ray, but I do not have a blue ray burner, & there does not seem to be any confirmation for me that blue ray discs are as archival or as good as a proven disc like Taiyo Yuden discs.

I have something called stuffit for mac compression software, & have seen zip files, that is alot of what downloads come in.

Is there a universal compression software to compress my data without losing quality, is cross plat form, & could be opened in the future? I am asking about this as a way to get more of my data on each DVD-R disc.

I have heard of having a drive offsite, but this is not practical, as I work from home, & have a family member keep my backup discs at their office, & having them tote a hard drive back & firth for incremental backups is not practical.

There are archival concerns about the second layer's longevity. Single-layer media is best, when burned, for truly archival needs.

Correct. The physical structure of a Blu-ray disc, which is essentially an upside-down CD (but with a micro-thin polycarbonate on the base), makes for a poor archival media, based on our media testing criteria.

Don't do it.
While the vanilla "zip" compression is probably safe -- and arguably so is a "rar" or a "tarball" (tar/tag.gz) -- you run a risk of further data loss. Compress archives add another layer of loss potential. If an archived file is corrupted, everything inside can be lost. Individual files, on the other hand, would not necessarily be destroyed if a neighbor file went bad (due to a bad spot on the media, be it disc or drive).

Then do incremental backups. In this scenario, you would copy your files (from your hard drive) to another drive once every 6-12 months or so, then discard the incremental DVDs. Between the drive clones, you use these discs. Do you understand?

The safest way is to also use two hard drives, and rotate them. One stays local until it's time to clone. Give it to the family member, and they bring the other one back, which you'll wipe and use for the next incremental in a few months. Again, between hard drive rotations, you'll send incremental discs, which are destroyed (put through a shredder) when the archive drive also has those files.

Again, does that make sense? (It's pretty straight forward.)

I use this method for my secondary disaster drive, which is only cloned once per year, and stored many 100's of miles away. It's a duplicate off-site backup, because I want to be double safe about backups.

between drive clones, could I use DVD-RW's? since they are not going to be permanent, buying top notch DVD-R's to burn to only to shred them later is a bit too expensive for me now.

Also, for a drive to be cloned & sent with family, could it be a portable External hard drive? such as one of these? http://eshop.macsales.com/shop/firewire/on-the-go & if so, which model? do I need firewire? or would usb work? do I need 72rpm? or would 5400 work? what would be the minimum capacity to get be?

the drive being sent with family would need to be not too heavy, & tough, as it will be carried in their tote bag that they take along with their lunch & books to work every day. Having them tote a full sized drive might be a little risky & heavy for them.

Yes. It will be slower, however, at 4x max.
Do NOT use DVD+RW -- only DVD-RW, as the DVD+RW have higher fail rates.
These are not archival media, but should last 6 months just fine.
Use good ones.

I like OWC, and have at least one of their enclosures for special needs (photography RAID-1), but for your purpose, those drives are overkill and massively overpriced.

We've been using a lot of portable drives lately for studio deliverables, transit back and forth of digital source and final encodes to distribution formats. I've been quite fond of the Seagate GoFlex drives (the silver ones are my favorite).

Those are $70 with free shipping, for 500MB drives: http://www.amazon.com/gp/product/B00...SIN=B003ELOSFK

The black 500MB are the same price.

Here's the full list of the GoFlex drives on Amazon: http://www.amazon.com/gp/redirect.ht...reative=390957
The black ones come in several sizes, up to 1.5TB for the small size version, and 3TB for the larger one.

That's at least 30-40% cheaper than the OWC drives -- and it's the same thing.

I did research not 7 days ago, and bought some more of the silvers from Amazon, because they were the best price.
The order arrived in 2 days (and I didn't pay for express shipping). One of them was filled and delivered Thursday already.

Yes. Backup drives don't need speed.

You may want to invest in one of these: Case Logic Compact Portable Hard Drive Case (Black)
$12 from Amazon: http://www.amazon.com/gp/product/B00...SIN=B000HDJT4S
I need to get myself one of these, too.

apologies if I am a little slow, so the new backup scheme would consist of something like a portable external hard drive such as the seagate pro, some DVD-RW's & my main full sized desktop external hard drive at home on my desk. My main desktop external would be in a OWC enclosure, at 1TB. So is it good to have my main full sized desktop external at home in a full sized enclosure? & the other drive to send with family member be a small portable?

So as I make data, I would back it up to the seagate, & send it with family member to keep at their office, it would stay there for about 6 months, & in the meantime, as I will probably create more data, I would make the incremental backups to the DVD-RW's & send them to the office as well with the seagate.

Then in about 6 months, have family member would bring back the seagate & DVD-RW discs back home for me to add the new content to the portable, wipe clean the DVD-RW's & start over again?

What would be the best size for the portable seagate drive? what would be the minimum size for me to go for? will just usb work? does it need to be 1TB 72rpm like my main desktop drive? if it will be a clone of my desktop hard drive at home, should it be the same size? in my case 1TB? & what if I move to a 2TB drive for my main external at home? I may eventually do this as the cost of storage goes down.

would the portable hard drive at the office last? it is moving parts.

Yes.

Yes.

The ONLY flaw in this plan is you make an off-site an "on-site" again temporarily. This is fine, if funds are limited. Just don't procrastinate and let all drives be at one place for longer than a few hours (overnight).

Ideally, you'd have TWO 1TB drives, one sits in a drawer at the off-site, one sits in a drawer on-site. At some point, you swap them. Send the new backup to the off-site. Then the old off-site comes back home. You destroy the purpose of two drives if you bring the old one back before the new one is in place. (Three drives on-site temporarily.) This may or may not be overkill, but it's the IDEAL solution, for proper backup policy.

You answered your own question already:
... meaning the next question...
... is also a 'yes'.
You could split to more drives, but given the price of drives, two 500MB is most costly than one 1TB drive.

Yes.
Copy files overnight, and consider a file copy tool that will ignore stupid errors. On Windows, that's the free copy software 'Ycopy'. On Mac, not sure -- never needed one. Nothing is more aggravating that starting a copy, going to sleep, and having an question on screen (OK, Cancel) with only 2% completed. Copy tools can be set to ignore those trouble files, and copy everything else.

Don't forget the option of online backup services as well. They work good for "off site" backup in a pinch. This is only viable if you have a reasonably fast internet connection though.

... and a small amount of data. :thumb:

It takes the better part of a day just to upload 1GB, with about a 2mbit upload speed (which is MUCH better than average). Also, note that I said UPLOAD speed (2mbit), not the DOWNLOAD speed (20mbit). I cannot even begin to imagine trying to push up dozens or 100's of GB's. By the time it finishes, it will already be past due for the second backup!

There's also the concern for privacy. How safe are your files from hackers or prying eyes? At least in a desk drawer, I know who has access.

can anybody suggest some good 2TB hard drives that I could use as a main & the other as offsite backup? I would put these in a;uminum mercury elite enclosures.

& if I use DVD-RW's what brand or type should I use? I was just told to use good ones.

For external USB2 2TB drives, the Fantom Green Drives are excellent for $100:
http://www.amazon.com/gp/product/B00...SIN=B002VLYK0M
I know that wasn't your exact question, but they're worth mentioning. I'll be buying 2-4 more very soon.
I bought two back in November of December, and they've turned out to be exceptional quality drives.

For SATA drives, to be used in an enclosure, get the Western Digitals "WD20EARS" drives for $80:
http://www.amazon.com/gp/product/B00...SIN=B002ZCXK0I

For SATA drives, used internally on a Windows XP system, get the "WD20EADS" drives for $100:
http://www.amazon.com/gp/product/B00...SIN=B001UE8LRE

EADS vs EARS.
EARS is new tech, only works on Vista and above, or Mac.
EADS is older tech, works on XP and earlier.

Buy these: http://www.meritline.com/verbatim-6x-dvd-rw-blank-media-silver-branded

Thanks for the links, is there a big performance difference between 32mb cache/buffer & 64 mb cache/buffer? I would be using this drive with firewire 800 on my enclosure with my macbook pro. The drive will have photo & video files from other digicams & .doc files & eventually .NEF RAW files & .mov files from my nikon D7000. Will the cache/buffers make a critical difference for me?

It seems alot of the drives have the 32mb cache/buffer, will this be enough to work with firewire or usb 2.0?

also, does windows xp "see" or recognize a drive connected via firewire? or only usb 2.0? I ask because I run virtulization software paralells desktop 5.0 on my macbook pro that enables me to run windows, I have windows xp & windows 7 on my macbook pro as well. But I have & use the hard drive via a firewire port on my macbook pro.

For certain tasks, yes.
For most tasks, no.

Difference? Yes.
Noticeable differences? Probably not. Debatable.
Critical difference? No.

At the risk of being wrong, I actually believe Firewire 400 and USB2 are too slow to really use the full buffer at full speed anyway. Those larger buffers are best used on new drives in new computers, where everything is fast (SATA II, eSATA, Firewire 800, USB3). For older drives, it makes little to no difference that I'm readily aware of.

Windows computers? Yes, if it's formatted to FAT32 or NTFS. Or if MacDrive is installed, and it's a Mac formatted drive.

Windows inside a Mac is different. Mac OS X controls the hardware, and Windows gets access via hardware virtualization (abstraction layers). Even if the drive is formatted to Mac HFS+, the drive is still usable by the VM Windows, because it's a Mac VM. In basic language, the virtual Windows can use anything the Mac sees.

So for my purposes that I listed earlier, a 2 TB hard drive with a 32mb buffer will be adequate? I would be using it with a firewire 800 port on my mac, since I am using firewire 800 & sometimes usb 2.0, is it worth the extra cost for me to get a hard drive with a 64mb cache?

What's your cost difference? If it's $5, sure -- if it's more, pass. That's what I would do. I took a quick gander online, and many others say the same thing -- there is NO DIFFERENCE between them. At most, people will choose 64 MB vs 32 MB cache buffers if the cost is $0-10, but never more. It just doesn't make much of a realistic difference.

Given that you're a student, save your money. Get the 32MB if it's lower price.

for a pair of rotating 1 or 2TB external hard drives in the backup scheme I have been advised on, what would be the best formatting system for them? I have preferred to use fat32 as this is cross platform right away, I dont need to have software installed on any mac or pc.

Is it safe? I have used fat32 on my 1TB drives without any problems that I have noticed.

FAT32 has a limitation on max file size.
A file cannot be larger than 3.99GB, so that is an issue for large videos, ISO images of discs, etc.

I'd use HFS+ if Mac is your main OS, and then install the $37 MacDrive on the Windows system.

cloning a drive with partitons to a blank drive
 

If I have a hard drive with a fat32 & say, a NTFS partition with content on it, & want to clone it, how would I do it? could I just format the blank drive to fat 32 & clone the other with the partitions & content? will they clone over?

You don't really need to "clone" the drive for what you're planning to do. Just copy files are drive A to drive B. That's all you really need to do. I don't actually "clone" drives very often, as it's unnecessary. Just copying files is fine.

"Cloning" a drive means you make a perfect image of it, all the way down to the file system (i.e., FAT32, HFS+, NTFS, etc).

If you want to copy files off NTFS to FAT32, that's fine, as long as the files are not larger than 3.99GB, as FAT32 cannot have files of 4GB or larger. Personally, I'd only use NTFS or HFS+ in 2011. There are also issues with the max size drive that can be formatted as FAT32.

What are enterprise drives? & would one of these be good for my backup scheme? are they practical? or would I be better off with a consumer drive?

That most likely refers to drives that are intended for high end servers, such as SAS (serial attached SCSI) in a RAID configuration. No, that's not something you'd need, want or be able to properly use. Get a "general use" type of drive for a home/office desktop setup.

I found this drive; http://www.amazon.com/gp/product/B00...SIN=B003S6ID20

is this a good one?

also, would it be safe to get the $90.00 one from amazon; http://www.amazon.com/gp/redirect.ht...reative=390957 they are used, but they have a fulfillment by amazon, or should I just get the new ones?

I have been using a 1TB hitichi deskstar with no problems yet.

and using it with or as the backup or swap drive with this one that I was suggested; http://www.amazon.com/gp/product/B00...SIN=B001UE8LRE

I read that some people prefer to use a drive one each from different makes/brands, is this a good idea? or should I stick with 2 of the same brand and model? does it matter?

I like the look of the western digital green, but I got nervous when I read about what seemed to be too few instances of failure on newegg; http://www.newegg.com/Product/Product.aspx?Item=N82E16822136344

It's not a "bad" one. I would simply suggest Western Digital has proven itself to be a better drive. And the WD drive is only $99 + $4 shipping. But if you want a Hitachi, go for it. It's a good drive, too. I have some Hitachi drives, and these have been fine.

I would buy the new drive, since this is for archival needs.

Same here. 500GB, 640GB and 1TB have all been fine.

That sounds like a good idea.

Generally speaking, you reduce the margin of "duplicated errors" by using two different drives, from two completely different manufacturers. Should one drive prove to have some kind of fatal flaw, the other drive is unlikely to suffer from that same fate.

You never know what boneheaded thing somebody may have done, so remember to NEVER TRUST user reviews left on shopping cart sites. While it can be good for identifying specific trends, it's also a hotbed for morons to demonstrate their lack of knowledge, skill and competence. Many, many negative reviews are left because a person has not followed instructions, or has mishandled the item. Finally, merchants and devices are often blamed for shipping-related errors (exposed to radiation, x-rays, extreme temperatures, etc). That, of course, should have no bearing on how good a store or product actually is, yet those are what the reviews blame.

As the old saying goes, "take it with a grain of salt". ;)

various shipping circumstances have me thinking, are there any things I should do to check for functional problems when I get my hard drive before using? if it was exposed to extreme temps while in transfer, (it is still winter), although I am in California, but if I order one from say the midwest or east in the US, should I leave it at room temp for a bit? or run disk checks on my mac? etc this sort of thing?

Always. Changes in temperature and/or humidity can cause condensation to form. And while it's not good for a device, regardless of on/off status, it's best to leave the device OFF while the condensation evaporates. Hard drives have small holes in them -- they are not air-tight or even water/liquid proofed. (Excluding a few specia;/expensive "tough" drives that are on the market -- mostly for gimmick, as far as I'm concerned.)

Aside from that, no, nothing you can really test. Turn on the drive, see if it works.

how long would the for a bit be? should I leave it in the box or shrink wrap? or take the bare drive out & let it sit?

You can open it, sure. Just be careful not to subject it to static electricity, put it on bare carpet, etc. I open drives and set them on a wood desk surface for several hours.

If I use hard drives, or mirrored hard drives as a backup, say for instance I have a hard drive stored somewhere for a few to several years & it is not connected or started up at a computer in that time frame, will it's components have deteriorated? will it rot out?

like a car, does it need to be started up periodically?

I have some older 250 gb PATA IDE Iomega external hard drives, will they rot out?

A car needs to be started periodically because it has chemicals in it that can go bad: gas, oil, grease, fluids, etc. Those can change consistency over time, the non-elementals can break down and separate, etc. A hard drive is mechanical. No, it cannot "rot" in any sense of the word. By "a few" we're only talking 5 years max, too -- not decades. After a few decades, yes, drives may have issues from decomposition of the metal alloys. It's possible.

If you have read something to the contrary, you're being fed misinformation. Here's a great example of utter bollocks (BS):
(Note: I don't link to crap. You'll have to copy/paste it to your browser if you want to read the article.) "Magnetic fading" is a made-up term, as used here, as quoted from some anonymous source. Somebody is confused. I would have to suggest that Mr. LJ is gullible to have believed that garbage. It's rather sad when people spread this sort of garbage online. This is the same sort of horsefeathers that is spouted by amateur video conversion services that insist "your tapes will fade" if you don't (wink, wink, nod, nod) get them converted right away.

Tape-based magnetic media has a lifespan of "only" a few decades, but that's because of glues, binders, chemicals, etc. In other words, it's like the car. Chemical changes. Non-chemical hard drive platters have no such affected systems. Hard disks give out because of wear to the motors, the arms, needles, etc -- basically hard drives wears from use, not inactivity.

The only reason hard disks are not tested for this length of time is because the tech obsolesces so quickly, within 5-10 years. Newer drives use different tech, and are most often completely incompatible with the previous generation of hardware. Even industrial computing items like ATMs, which often have last decade's hardware and software powering them, tend to be refreshed every 10 or so years. Data is easily moved to a new disc, because unlike tape "formats", the disks just hold data. The data is not formatted in a special way that forever binds it to the media, like VHS. There's no incentive or reason to extend the life of hard disks forever, by replacing motors and arms. Hence a complete lack of research. Which in turns allows for asinine claims about "disk fade" because nobody has wasted their time on contrary (more factual) research.

are there any special conditions that a hard drive must be stored in? if it is not in use in a computer or as an external? can components oxidize?

it seems then that the connector types on a hard drive might be an issue, if I have one in an enclosure with usb & firewire, I know no one can see in the future but, will I be able to connect it in several years after being stored? will there be adapters if the connectors become obsolete? will SATA be obsolete & will there be adapters for old tech?

Indoors.
Normal temperatures (60-80 F), normal humidity (not a green room) -- just a normal human living/office condition.

Adapters for old drives? Not usually, no. At least not historically.
IDE/SATA was an exception. That one is bidirectional!
USB 1.1 to USB 2.0 to USB 3.0 for external, same basic plug/comm setup, was an exception. Also bidirectional.
Firewire 400 works on Firewire 800, with plug shape adapter.
Reverse compatibility has gotten better with time.