Aligning your Corporate Backup Policy with Veeam

[tsightler]

We've recently started using Veeam backups to backup almost 100% of our VM infrastructure, however, we've hit one very big issue, how to meet our corporate backup policies requiring tape backups while using Veeam.

We have about 5TB of data to backup and our previous backups solutions performed full backups every 14 days on the weekends and incrementals nightly the rest of the time. This used a large number of tapes on the weekend, but nightly backups were only a few hundred MB's typically only needing 1 or 2 tapes each night, easily manageable.

Now with Veeam we have about 3TB of vbk files, but these files change every night. Our tape drives currently use LTO3 tapes so that's about 8 tapes every night. We have third-party agreements that require us to keep offline backups so our tape count has gone up significantly and the time involved in getting all of this data to tape is unworkable.

Has anyone else run into this issue where corporate requirements to put data on tape seem to work against Veeam? Is there a reasonable way to address it? The easiest thing would be if Veeam actually allowed incrementals and then simply consolidated those incrementals on a given schedule (perhaps once a week) but that's not going to happen.

We've considered using a large mirrored drive and simply swapping half of the mirrored drives every night, but we haven't tried it yet. Are people simply giving up tape completely? How do you address policies requiring "offline" backups. Success in getting them changed?

[Gostev]

From the feedback I've heard, instead of doing nightly backup of the VBK files to tape, customers are doing weekly backups instead. The VBK and the synthetic backups that stays on disk will give you the fast restore that you need (which is 90% of the restores requests usually), while the tape backups provide the long-term/archival of data in the more rare case that you have to restore something older than a week or two. The weekly tape backup job can then be scheduled on days that you have a larger window to work with, so you don't have to worry about getting everything done on days w/smaller backup windows.

[Gostev]

that's not going to happen.

I would not say so we are definitely open to suggestions and are willing to implement anything that our customers need. We are seeing that tape is being phased out more and more and are unlikely to change the main backup engine from synthetic. But, at the same time we are still ready to implement working workarounds (no pun intended) for customers who are stuck with tapes and certain policies, especially when multiple customers are in need of such (which seems to be the case). We will brainstorm this internally, and if you have any ideas, please do share them with us. The simpler the workaround will be - the faster we will be able to implement it.

[Gostev]

Finally, concerning your question about giving up the tapes completely, and addressing the policies requiring "offline" backups, I am hoping to hear some feedback from community on that. For instance, I would love to hear from Matt (mdornfeld), because one of his previous comments was inline with this.

Just my 2 cents. The company I work for would have very little current interest in this feature. We would definately like to see your developement efforts go towards getting the VMware backup to disk perfected.

Also, there was another interesting comment from Brett (bwestover) that I've picked up on LinkedIn discussion:

Personally, I have never slept better knowing my data was on tape. If you've ever seen a tape archive fail, you'll know what I mean.

Writing to magnetic tape is of course just a copy, but that copy has no redundancy, and worse typically carries a heavy dependency. In a 2 or 3 tape set, a single tape failure renders the entire backup set useless. Or even in a single tape, a single severe error on the tape, will render the entire tape useless.

Of course this is true of hard disks as well, but there are robust and redundant technologies that significantly mitigate the effects. Disk Arrays with interchangeable, redundant parts seem a lot more stable to me.

If I require a high level of protection, for my money, I'll always backup to disk and then replicate that data to a remote disk array for offsite storage.

I can also tell you how we are doing this (although we are not very big company, under 200 people). We have 2 hard drives that we copy our backups to, and we simply swap them taking the other one off-site. May be not the best solution if you have many terabytes of data to backup, but pretty much ideal for SMB. And it sounds like many customers are doing the same thing, we even had automation script posted for this procedure recently by one of our community members.

[tsightler]

From the feedback I've hear, instead of doing nightly backup of the VBK files to tape, customers are doing weekly backups instead. The VBK and the synthetic backups that stays on disk will give you the fast restore that you need (which is 90% of the restores requests usually), while the tape backups provide the long-term/archival of data in the more rare case that you have to restore something older than a week or two. The weekly tape backup job can then be scheduled on days that you have a larger window to work with, so you don't have to worry about getting everything done on days w/smaller backup windows.

The problem with this is that it's simply an unacceptable solution from a true "disaster recovery" scenario, a loss of everything in the datacenter on Friday would loose far too much data. Our policy requires, "nightly offsite, offline" backups of all data. Incremental backups to tape make this easy. vRanger backups were also easy since they write change files, but of course you have to run a full every so often.

Out tape backup software uses the concept of "consolidated incrementals". You take a full backup, and from that point you never take another full backup, you take daily incrementals and, on the weekend, you run a "consolidate" job which takes the last full, consolidates all the changes from the incrementals, and makes a new full, without actually having to do a full backup.

We will brainstorm this internally, and if you have any ideas, please do share them with us.

Unfortunately, without knowing how your product works, it's difficult to know how difficult to think of workarounds. We've currently been brainstorming our own solution that involves running Veeam (or at least the Veaam storage) within a virtual machine. We could then take a VMware snapshot prior to the Veeam run each night. When the Veeam backups ran, all changes would actually be in the "-delta" files of the VMware snapshot and we could just backup the -delta file. We would make a new snapshot every night until the end of the week and then delete all of the snapshots, which would consolidate all of the changes into the vmdk file, run a full backup of the vmdk, and start the cycle again.

We are already replicating our data offsite, but it's the "offline" part that is difficult to deal with. This part is critical however. People who simply mirror data seem to take the risk that they'll never be hacked or have data logically damaged by a virus or other attack. Offline storage addresses that requirement. Logical mirrors are not backups. They are great for redundancy, but they are not offline and thus are subject to being destroyed, whether by a hacker, a virus, act of nature, or simply a disgruntled employee. While those chances are certainly small, I cannot afford to overlook them at my company. A good backup policy should cover your companies data even in the event that and employee intentionally attempts to destroy all of the data. At my employer I could easily destroy every bit of data currently online on every storage array in the entire company, including the offsite mirrors, but I can't get those offline tapes. They won't even be retrieved from the vault without management sign-off.

If there's a single employee in your company that has the ability to delete both all of your live system data, and all of your disk backups, you're company is at risk. My employers policy virtually eliminates that possibility.

[dlove]

If there's a single employee in your company that has the ability to delete both all of your live system data, and all of your disk backups, you're company is at risk. My employers policy virtually eliminates that possibility.

security is a whole different topic and usually security is based on budget and not best practices or company sla's (ethically wrong but done everyday). i don't know what hardware you are using but offloading snapshots to an array will take care of corruption or the rogue IT employee who runs ape wild in the datacenter deleting files and unplugging network cords I guess separating out job duties would help too but it seems like what you describe is the mentality of a smaller company with few people doing everything which is why you would have access to everything in the first place.

Tape might be a necessity for your sla's but it's not a reasonable approach to restores on a mass scale unless you have several tape libraries and many drives working in tandem or you have little data in the first place. By the time you restore everything, you would most likely be out of business or pretty close to it. Tape should be a last resort and a last resort only! Replication to arrays/servers and implementing snapshots (hardware/software) based are the only way to really ensure you can get back up and running quickly. If you are truly worried about someone sabotaging the veeam backups then you should address those first.

That being said... I ran into the same problem as you having to offload to tape and we changed up the routines to get the job done. We snapshot lun's presented to the veeam proxy then present those snapshots to the netbackup server which in turn gets backed up to tape and vaulted. We also mirror the data (asynchronously) to a similar array and snapshot that as well since we have a ton of storage to toy with. Doing that gives me a few options...

1) revert from snapshots on the target veeam luns on primary array.
2) revert from snapshots on DR array and present that lun to a VM with veeam installed.
3) revert from VSS (we are using Win2k8) but haven't set it up for VSS.
4) revert from tape (last resort) files can be local on netbackup server disk and then duplicated to tape so same day restores would come from disk.

Of course there are a lot of other options to use that we haven't looked into because we just recently implemented the product and ran away from vizioncore and their "promises". We could look at software based snapshots through netbackup and FT fiber transport. I personally prefer block based replication built into the arrays but we could do robocopy or something along those lines to keep multiple copies in different locations.

[tsightler]

security is a whole different topic and usually security is based on budget and not best practices or company sla's (ethically wrong but done everyday). i don't know what hardware you are using but offloading snapshots to an array will take care of corruption or the rogue IT employee who runs ape wild in the datacenter deleting files and unplugging network cords I guess separating out job duties would help too but it seems like what you describe is the mentality of a smaller company with few people doing everything which is why you would have access to everything in the first place.

We're reasonable small, a 1000+ employee manufacturer with plants in the US and Europe. Separating out jobs duties is difficult when there are only 5 people total. Most of us have access to the datacenter. I'm pretty much the only one that has full access to everything, but many other employees could do serious damage. But the reality they don't have to do much damage to caused a lot of greif, simply delete everything they have access to and steal three drives from the "backup array" and they've wiped out the ability to restore. Yes it's far fetched and unlikely to happen, but easily accomplished. This isn't possible with tapes.

Tape might be a necessity for your sla's but it's not a reasonable approach to restores on a mass scale unless you have several tape libraries and many drives working in tandem or you have little data in the first place. By the time you restore everything, you would most likely be out of business or pretty close to it. Tape should be a last resort and a last resort only! Replication to arrays/servers and implementing snapshots (hardware/software) based are the only way to really ensure you can get back up and running quickly. If you are truly worried about someone sabotaging the veeam backups then you should address those first.

I think your assumption that tapes are not fast enough and you'd be "out of business or pretty close to it" before you get it restored is pretty far fetched. You seem to assume that everything we backup is critical to be restored in some extremely fast window, or that the data that has to be on tape due to SLA's is required to be available 24x7. That's simply not true. The data has to be backed up and 100% safe to prevent loss in the event of a disaster, the data is not required to be online 100% of the time, heck we could live without it for several days without a major impact. The critical systems could be restored from tape in a matter of hours. We've actually had to restore our ERP database from tape after a failed maintenance. It took about 2 hours. We have disaster plans on the plant floor to keep critical manufacturing flowing for 24 hours or even longer if the event is planned/known (for example hurricanes where major disruptions are possible/likely). We could easily restore every system from tape in 24 hours, critical systems to support manufacturing and sales would be back in 2-4 hours. Ancillary systems to support engineering, R&D, marketing, etc would take quite a bit longer, but wouldn't affect our manufacturing and thus are not "business critical". I can promise we would not be out of business in that time.

Now, I'm not arguing that tape shouldn't be the last resort. That's certainly true, we make heavy use of snapshots and replication, and have for years. The point is, nightly backups to tape with offline, offsite transport are far safer in the case of a fire or major catastrophe in a geographic area and I could restore the critical data from them fast enough. A restore from tape with Veeam is actually much longer because it's a two step process.

We've looked at many of the options you mentioned, and kicked around many other options. We're trying to come up with way to provide the same level of physical safety with disk that we do with tape but are finding that difficult. A lot of people discuss offsite replication but there's just know way of doing that quickly unless you have really fast links to some remote site. With our previous backup solution a nights worth of incrementals fit on one tape, we'd duplicate the tape, send a copy to the offsite vault, and have a copy we could take with us. Now with Veeam even if we figure out how to spool them off we'll have half a dozen tapes every night at a minimum. Our storage costs are going to go way up. The lack of a true incremental that can easily be pushed to tape is killing us and if we can't come up with a working option may force us either back to Vizioncore (as much as we'd hate that) or to another product completely, relegating Veeam to the smaller VM's where we can afford to push the whole VBK to tape every night.

[Gostev]

Just wanted to update that we have done some further discussions and planning around this feature (regular incremental backup) with the team, and we are now planning to make this feature generally available as a minor release by the end of this year.

[tsightler]

This is great news for us. We are truly having great success with Veeam and this was really the only thing that keeps Veeam from being perfect for us (well, and index of restorable files would be nice, but difficult with an image level backup tool). The requirement to get the data to offsite tape daily is pretty challenging with the current implementation but we'll make do for the next 6-9 months while you guys work on something.

Thanks for listening to client requirements!! You guys may be the most responsive vendor I've ever worked with.

[fredbloggs]

The requirement to get the data to offsite tape daily is pretty challenging with the current implementation but we'll make do for the next 6-9 months while you guys work on something.

Just wondering, how do you guys work around this at present?

[tsightler]

Well, I really wouldn't say that we have a work around. We are simply living with having to spool the entire VBK/VBR files to tape every day. We split our jobs up so that they are typically not too big (~500GB) and the job runs a command when finished to start the tape job. That way some systems are still running Veeam backups while other are spooling to tape. This works OK for most of our jobs, but our biggest servers (one server is over 1TB) takes nearly 11 hours to backup with Veeam and then 8 hours to spool to tape, so that's 19 hours total. Obviously if this system became much bigger we wouldn't have enough time. This is obviously eating tapes (LTO3, 400MB/tape) like crazy but we only have to keep daily backups for 7 days, weeklys for 4 weeks, and monthly's for a year, so it won't be horrible.

We're going to try to tough it out this way until Veeam includes the incremental option.

[akafritz]

We implemented Veeam several months ago and our backup administrator informed me of the same issue recently. Luckily we have enough tapes in rotation that none have had to be purchased - yet. If they did, we'd also be eating up an extra dozen tapes a week to facilitate the lack of traditional incrementals.

Since I just just beginning to devote some time to this issue before it requires spend, its great to see that a plan was created to facilitate the paradigm. So, I second the need for it, and am curious as to the progress of a release by years end. Until then, we'll be evaluating the suggestions voiced here within our infrastructure, and welcome any others.

Thanks

[dpenrod]

I am evaluating Veeam for VM backup. I have the same issue with Tape backup. I need to be able to take the daily data changes, and save them to tape or replicate them offsite. The single Large VBK with no daily incrementals presents a problem peeling off these daily changes.

I am happy with the quick restore from disk, thats the main feature. But my Exec teams #1 question is: what happens if a fire burns the datacenter down. With no offsite backup, were done. Want to go tapeless, but that requires offsite copies of backups, and multiple offsite versions for recovery from data corruption. I dont see how to do that without high monthly recurring costs of a high speed WAN connection just for replicating backups. This can price the solution out of reason when compared to alternatives. So using the existing tape capability is a lower cost solution compared to WAN recurring costs. Spending $3k on Veeam, and then $18K per year for the WAN to enable it does not work for me.

Possible solutions: ?

Can the VBK files be replicated using some type of changed block software, so only the changes in the VBK need be replicated over my single T1? WIn2k8 SRV DFS replication? Anyone doing this with third party replication software, sending only VBK changes?

Can the changed blocks be left in existance seperate from the main VBK, as an option in addition to them being incorporated into the VKB? Then the changes could be offloaded to tape. Would need a tool to rebuild them back into a VBK in the event of a total loss and restore.

Could de-duplication be built into the backups, similar to VMware DR, so that the amount of data to replicate to an offsite Veeam target is doable without a large pipe incurring monthly recurring communication costs.

Need
The Offsite replication target needs to maintain multiple previous backups (limitied by available disk space) so that if data corruption gets replicated to the offsite target, there are previous versions from which to recover. I am looking for a complete backup and recovery solution, not just a partial solution.

[Gostev]

You can use DFS-R or RSYNC to transport the VBK changed blocks to DR site.
Some customers are doing this today: Replicating .vbk files using DFS-R.

[cryptonym]

Tape backup support is a waste of the developer's time. Tape is obsolete, and you couldn't find a much more unreliable medium for backup unless you implemented punched paper tapes.

I'd rather see the core product functionality improved.

As to off-site backups, simply have your remote site connect in over a T3 line or something and replicate the SAN volumes holding the backups as a snapshot. This doesn't have to be active all the time, and can be initiated from the remote site (maybe require physical access to that system i.e. "locked in the vault").

Just my 2 cents.

[tsightler]

I don't think anyone here is asking for Veeam to integrate tape support into their product (maybe a few did, I didn't re-read the whole thread), but integrating tape backup into the product, and making you disk backups easy to archive to tape are two completely different things. Actually, I suspect that a mode which simply copies the changes to a separate file would not only help the tape case, but also cases like Data Domain archivers where the current Veeam paradigm causes performance issues.

It would also help replication for those of us that use rsync to copy the files. Right now rsync processing time is high because the main file has to be processed after each backup, but with a simple incremental strategy this wouldn't be the case.

I could also make arguments for tape being the only reasonably cost-effective and manageable archive media, for those of us that need to keep backups for fixed periods of time that exceed a year, but that's really beside the point. There are good reasons for having this functionality for quite a number of users, even excluding the tape argument, not the least of which is that their competitors all have modes which support this quite easily and thus, some customers that have corporate policies that require regular tape archiving HAVE to choose another product due to the lack of this capability. The issue has come up enough on the forums over the past year that it's obvious this is a problem for quite a few people.

[Gostev]

Tom is correct, we are not talking about adding native tape support. I've talked a lot with many customers about this one past year, and most people (even those actually using tape) all agreed we should not spend time on this huge task now when the tape backup is dying, but rather focus on perfecting disk based backups.

Here we are talking about adding regular inc/diff backups (without dropping synthetic) to give our customers more flexibility and better address some special scenarios, like those Tom described above. This is one of the features we are working on adding right now.

[cryptonym]

That's marginally better. But I would hope that these inc/difs would be generated as requested (adhoc on the fly) from the synthetics, and would not require any duplication in the nearline storage system. Maybe a utility to simply regurgitate the synthetics into a delta file and ability to write it to a cifs or nfs share would suffice? I certainly wouldn't want to be writing the same backups in different forms, chewing up bandwidth, performance and capacity.

[tsightler]

That's marginally better. But I would hope that these inc/difs would be generated as requested (adhoc on the fly) from the synthetics, and would not require any duplication in the nearline storage system. Maybe a utility to simply regurgitate the synthetics into a delta file and ability to write it to a cifs or nfs share would suffice?[/quopte]

If you don't want or need the inc/diffs, you'll just keep doing what you do now, synthetic fulls every night. The new features are not changing any existing functionality.

I wouldn't have expected the "chewing up bandwidth/capacity" argument from a person who solution in the last post was replicating all your data over an expensive T3. Not saying that's not a fine idea, but not everyone has the luxury of a T3 between two locations or the bandwidth to spare for replicating all their data. Heck, a lot of people don't even have a "failover" or "remote" site to copy the data too, they just need their backups archived offsite.

[Gostev]

Warren: I am not sure I understood your question, but you will be able to choose backup type for each job. Synthetic backup type will still remain default backup type.

[cryptonym]

My earlier post was to say "where there's a will there's a way". Whereas, my latter post was to say that the needs of a few (tape) should not outweigh those of the many (economical, performant, nearline, multiple backups suffice).

As to recovery, if someone doesn't have a failover site, then they probably also don't have the failover equipment in line either. So what are they going to restore their tapes with or onto? Acquiring outdated hardware can be a major hurdle to overcome in an emergency. Who's going to setup the new data center? Two men and a truck?

If someone is just handing off tapes for offsiting, and not testing restorations, then failure is in their future too. This is no matter the media used - processes fail. Even with 2 identical tape drives on each side, what if drive A is -3% of tolerance, and drive B is +3% of tolerance and the tape is readable over a 5% variance. Congratulations, no data to be found. How many times have you heard "We have nightly backups". Then when needed, heard "Well maybe the other tape will be good". Followed by the requisite "Glad I keep my resume up to date" as the last tape fails to restore. More times then I care to remember.

Which is more likely to fail (lower MTBF), tape or disk? That's why tape is nearly dead. And, we're nearing a time when the MTBF on hard disks is such that even they can not be trusted to recover the large volumes of data we need to store. In the not to distant future, I think we'll be commonly seeing RAID based backups as the minimum configuration.

[Gostev]

Very interesting. I learned a few things from this post. Thanks!

[tsightler]

As to recovery, if someone doesn't have a failover site, then they probably also don't have the failover equipment in line either. So what are they going to restore their tapes with or onto? Acquiring outdated hardware can be a major hurdle to overcome in an emergency. Who's going to setup the new data center? Two men and a truck?

Once again, you seem to completely ignore offsite archiving. This is a REQUIREMENT for many people. Some of this can be offset with disk based archiving, but then again, in most cases the disk archiver also needs to be backed up and and those backups may need to be taken offsite, and many archiving solutions are VERY EXPENSIVE. As far as acquiring equipment, many companies don't have requirements to be up and running in an hour. A small or medium business who facility is damaged in a hurricane may not even be able to occupy the building for a week or more, and in that time you'll be able to locate equipment, get the tapes, and get data restored, it's more important to have the data safe and available than to have it back up and running quickly. I've worked with many small and medium manufacturers in this scenario to restore their operations after hurricanes or, in one case, a tornado. If they had been bigger, global organizations it might have been critical to have failover and they would have had the budget for it, but for a small organization the offsite tapes were safe and effective.

If someone is just handing off tapes for offsiting, and not testing restorations, then failure is in their future too. This is no matter the media used - processes fail. Even with 2 identical tape drives on each side, what if drive A is -3% of tolerance, and drive B is +3% of tolerance and the tape is readable over a 5% variance. Congratulations, no data to be found. How many times have you heard "We have nightly backups". Then when needed, heard "Well maybe the other tape will be good". Followed by the requisite "Glad I keep my resume up to date" as the last tape fails to restore. More times then I care to remember.

Wow, I'm amazed you've heard this that many times. I've been doing this 20+ years, and I've heard it exactly "zero" times. I've restored data from tapes that were two and three years old from archives at non-original sites, and I can't even remember the last time I had a tape failure during a restore, I think it was on an old Novell 3.x server back in the 80's. We preform a restore from tape test on our ERP system every year and we always use a 6-month old tape from the off-site vault to do it, and it's preformed at our remote site on different tape drive hardware.

Which is more likely to fail (lower MTBF), tape or disk? That's why tape is nearly dead. And, we're nearing a time when the MTBF on hard disks is such that even they can not be trusted to recover the large volumes of data we need to store. In the not to distant future, I think we'll be commonly seeing RAID based backups as the minimum configuration.

Modern LTO drives have MTBF that are in the same ball park as disks (250,000 hours at 100% duty cycle). Yes many hard disks will claim longer, but real world data puts those numbers in the same ballpark. I might agree that tape is still "dying", but I think it's death watch will be much longer than you do. Tape still has so many advantages, and all the "big boys" understand it. http://h20338.www2.hp.com/enterprise/us ... orage.html

[Gostev]

Fact 5 has quite impressive numbers... I did not realize. Although this type of equipment and actual tapes probably cost a fortune?

[cryptonym]

Once again, you seem to completely ignore offsite archiving. This is a REQUIREMENT for many people. Some of this can be offset with disk based archiving, but then again, in most cases the disk archiver also needs to be backed up and and those backups may need to be taken offsite, and many archiving solutions are VERY EXPENSIVE. As far as acquiring equipment, many companies don't have requirements to be up and running in an hour. A small or medium business who facility is damaged in a hurricane may not even be able to occupy the building for a week or more, and in that time you'll be able to locate equipment, get the tapes, and get data restored, it's more important to have the data safe and available than to have it back up and running quickly. I've worked with many small and medium manufacturers in this scenario to restore their operations after hurricanes or, in one case, a tornado. If they had been bigger, global organizations it might have been critical to have failover and they would have had the budget for it, but for a small organization the offsite tapes were safe and effective.

I'm not ignoring offsite. The DR site is by definition offsite.

If you are trying to differentiate an offline archive from an online replica, I completely agree that a synchronized online (remote or not) replica is not a backup. But a remote site, with copies made through a batch process is not a synchronized online clone, it is performing in the role of a backup. In our case, it is generally done as follows. The primary SAN and the remote SAN are replicated. Batch jobs at the remote site snapshot the LUNs to other offline LUNs in another disk group. This gives quick DR (move to the remote site) and also provides offline, point in time, recovery images of the LUNs (from the snapshots). As long as the snapshots do not initiate in the backup windows, the snapshots have valid backup images in them. And no, this is not a SMB solution. In this case, the snapshots are replacing the tape, and are offsite from the primary site.

I'm familiar with SMB's too, but not all are so fortunate to rely on their systems in such a minor fashion as yours have. I've consulted for a company making just-in-time parts for auto manufacturer's where the penalty for delivering parts to the line was more than $100,000 per hour. I've watched as they literally carried a armful of parts to a helicopter landed in the parking lot and flew the parts to the out of state plant to stay on schedule. For them, losing plant automations would have literally put them under in well under the time frames you seem so comfortable with. I've worked with some SMB's long enough to realize that as things become more automated, and the "knowledge" moved into systems, the users and management (due to nothing more than normal promotions, hiring, attrition, etc) became less able to "remember" how or why processes were performed. The paper-based alternatives were long ago sent to the shredder, literally. This in itself forces the SMB's to constantly re-evaluate the importance of their data systems.

Wow, I'm amazed you've heard this that many times. I've been doing this 20+ years, and I've heard it exactly "zero" times. I've restored data from tapes that were two and three years old from archives at non-original sites, and I can't even remember the last time I had a tape failure during a restore, I think it was on an old Novell 3.x server back in the 80's. We preform a restore from tape test on our ERP system every year and we always use a 6-month old tape from the off-site vault to do it, and it's preformed at our remote site on different tape drive hardware.

I'm glad it all works so well for you, but the fact is that whole companies exist to deal with this supposed non-problem. See http://www.ontrackdatarecovery.com/unde ... data-loss/ and http://www.ontrackdatarecovery.com/tape-recovery/ .

Modern LTO drives have MTBF that are in the same ball park as disks (250,000 hours at 100% duty cycle). Yes many hard disks will claim longer, but real world data puts those numbers in the same ballpark. I might agree that tape is still "dying", but I think it's death watch will be much longer than you do. Tape still has so many advantages, and all the "big boys" understand it. http://h20338.www2.hp.com/enterprise/us ... orage.html

MTBF is just that, a mean, and we all know reality always seems to put us on the low side, but a MTBF of 1.2 million hours on a disk vs. 250,000 on LTO can't be dismissed, it's a 5:1 difference - so it's a little league vs. majors ball park . Especially when the same LTO's that say 250,000 hrs MTBF also say the head life is 60,000 hours. Seems to me like the head is a fairly important component to remain operable. Yet in a tape it is always being sanded down and grooved, whereas in a disk it should be blissfully flying on a cushion of air.

From your link, HP would have no financial interest in selling you tapes (at about $50 a pop), that last about 200 passes (see http://en.wikipedia.org/wiki/Linear_Tape-Open), right? Consumables are a vendors dream.

How much of a backup is lost if a tape fails? And, how much is lost if a disk in a disk library fails? The answer to the latter, is zero - they are usually RAID'd sets.

To address the the lowest end of the SMB range, there are small external removable disk units, that take hard disks mounted in trays. These can be swapped out and stored off site, and cost about twice what an LTO does for similar capacity, yet have a vastly superior MTBF. I'd be much more confident in this type of setup than in tape. And even these can be configured in RAID sets to maintain data integrity should one drive fail.

Tapes will live on, just as the floppy disk has.

In closing and trying to return to what matters for Veeam, I believe that their synthetic backups represent an elegant solution to the issues of handling the volume of data we must contend with today. I'd hate to see any back peddling done to make an obsolete paradigm more workable. To that end, I believe they should stay the course on the synthetic backups and if any concession is made, should simply produce a process that has the means to produce an full/incremental/differential from the synthetics and present the result to a user, upon demand. I'd propose that they not ever write full/incremental/differentials to their backup repositories. This leaves the actual backup processes used now unaltered, while providing the utility that users seem to desire.

[tsightler]

I'm familiar with SMB's too, but not all are so fortunate to rely on their systems in such a minor fashion as yours have. I've consulted for a company making just-in-time parts for auto manufacturer's where the penalty for delivering parts to the line was more than $100,000 per hour. I've watched as they literally carried a armful of parts to a helicopter landed in the parking lot and flew the parts to the out of state plant to stay on schedule. For them, losing plant automations would have literally put them under in well under the time frames you seem so comfortable with. I've worked with some SMB's long enough to realize that as things become more automated, and the "knowledge" moved into systems, the users and management (due to nothing more than normal promotions, hiring, attrition, etc) became less able to "remember" how or why processes were performed. The paper-based alternatives were long ago sent to the shredder, literally. This in itself forces the SMB's to constantly re-evaluate the importance of their data systems.

But many small manufacutures, or other businesses for that matter, are local. They're still dependent on their computer systems and they need an economical way to provide, but if a major catastrophe damages their facility, it's not critical that everything be running again 5 minutes later. Take hurricane Katrina. Many local business were affected, restaurants, dentist offices, schools, small manufacturers, body shops, I could go on and on. Many of these business, including the smaller manufacturers, only have a single facility. If their power is out it doesn't really matter if their automation systems are running or not, they can't make anything because they don't have electic service. If they had a $100,000/hr penalty for a customer they either need "loss of capacity" insurance or they need to source an entire, equipped facility.

Just a single example, we had a medium sized private school that had the roof ripped off the facility, completely destroying the servers due to rain damage. There was really no hurry to recover that facility because the damage was significant enough that it wasn't going to be occupied right away, but it was important that the data was safe. The site had a simple offsite tape rotation and we had the last two retrieved and restored the data in a temporary office they rented about 4 days later. There are literally 10's of thousands of companies that use this method today. Now, am I saying that tape is the only solution here? You could probably do something with online storage, but you have to worry about restore times as well. Perhaps not a big deal but still has to be considered. With tape we just brought the media and drives with us to the new location, and started our backup procedure right where we left off.

I'm glad it all works so well for you, but the fact is that whole companies exist to deal with this supposed non-problem. See http://www.ontrackdatarecovery.com/unde ... data-loss/ and http://www.ontrackdatarecovery.com/tape-recovery/ .

To be fair here, Ontrack has been around a LONG time and is far more known for it disk recovery services than tape recovery. Heck I've used their services for a client who did not have a tape backup. They've been doing disk recovery for years. Also, I did not said a tape NEVER failed, I'm sure there are cases, just like RAID arrays. I've experienced data loss events on three RAID arrays, twice on an EMC array that turned out to be a firmware bug, and once on a Adaptec RAID appliance where a drive failed in a strange way and started rebuilding only to eat all the data. Guess what was storage on that Adaptec RAID appliance, our disk backups.

MTBF is just that, a mean, and we all know reality always seems to put us on the low side, but a MTBF of 1.2 million hours on a disk vs. 250,000 on LTO can't be dismissed, it's a 5:1 difference - so it's a little league vs. majors ball park . Especially when the same LTO's that say 250,000 hrs MTBF also say the head life is 60,000 hours. Seems to me like the head is a fairly important component to remain operable. Yet in a tape it is always being sanded down and grooved, whereas in a disk it should be blissfully flying on a cushion of air.

The reason I said, "in the same ballpark" is because, in general, you need many more disks to backup the same amount of data as required by tape. The 1.2 million hours is for a "single drive", but a raid array is almost always going to have multiple drives. With 6 disk drives in the array (not uncommon) you're in the same ball park as a single tape drive (you'll achieve 1.2 million hours of drive time in only 200,000 hours because you have 8 drives). Also, tapes MTBF are accrued only during use, so the 250,000 hours assumes 100% duty cycle. In addition, the stated MTBF's are generally longer than the useful live of the technology anyway. Think about it, even if I only use the 60,000 hours of head life, how long is that? Assuming 100% duty cycle, that's almost 7 years. In seven years I'll probably be moving to new technology, but even so, it's extremely unlikely I'll have 100% duty cycle so the life is likely longer (we have a number of LTO 1 drives that are nearing 10 years of service without replacement).

From your link, HP would have no financial interest in selling you tapes (at about $50 a pop), that last about 200 passes (see http://en.wikipedia.org/wiki/Linear_Tape-Open), right? Consumables are a vendors dream.

Of course, but that's true of disk drive vendors, storage vendors, software vendors. Disk vendors write most of the papers on tape backup to disks, so this is mostly a wash. Services for storage arrays, WAN links, software, and offsite services are those vendors dreams.

How much of a backup is lost if a tape fails? And, how much is lost if a disk in a disk library fails? The answer to the latter, is zero - they are usually RAID'd sets.

If setup correctly, no more than one set of backups. We lost all of our backups when our Adaptec RAID storage failed miserably. Of course, we had tape backups of the disk backups (same think we're trying to accomplish with Veeam now).

To address the the lowest end of the SMB range, there are small external removable disk units, that take hard disks mounted in trays. These can be swapped out and stored off site, and cost about twice what an LTO does for similar capacity, yet have a vastly superior MTBF. I'd be much more confident in this type of setup than in tape. And even these can be configured in RAID sets to maintain data integrity should one drive fail.

BINGO!! This is a case that doesn't work well with Veeam now either, because you'd basically have to run a full every night to accomplish this, or backup to a local disk and then copy it to an external media. Some people do this now, but an incremental would make this a much faster operations for those smaller shops that still have a lot of data.

Tapes will live on, just as the floppy disk has.

Hehe, good one! I should point out that floppies did live a very long time after major industry players proclaimed their death though.

In closing and trying to return to what matters for Veeam, I believe that their synthetic backups represent an elegant solution to the issues of handling the volume of data we must contend with today. I'd hate to see any back peddling done to make an obsolete paradigm more workable. To that end, I believe they should stay the course on the synthetic backups and if any concession is made, should simply produce a process that has the means to produce an full/incremental/differential from the synthetics and present the result to a user, upon demand. I'd propose that they not ever write full/incremental/differentials to their backup repositories. This leaves the actual backup processes used now unaltered, while providing the utility that users seem to desire.

Well, we aren't far apart here. I would never want Veeam to change their Synthetic backup either. I believe Veeam is implementing this so that, even if you use incremental/differential, that you still get "synthetic" backup as well. Basically you take a "Full", then say a weeks worth of "incrementals" then for the next "full" the changes from the "incrementals" are consolidated into the full, and a "synthetic full" takes place. The incrementals become rollbacks for the new full, and rinse/repeat (I don't know that Veeam is doing it this way, but it's my understanding that this was the goal). This has the advantage of still giving synthetic fulls, preforming much less I/O on the target storage during nightly runs (since only changed block have to be written to the incremental file, rather than read old data from VBK, write old data to VBR, write new data to VBK that happens with a synthetic full), and making it friendlier to replicate nightly changes to a remote site or to tape. To me this is a win for everybody, and if you don't want/need the feature, just keep doing what you're doing.

I've been using "synthetic" backup with my traditional backup software for 5+ years, it was just implemented slightly different than Veeam. They called in "incremental forever" and basically they simply took the incrementals from the week and consolidated them into a new full.

[dpenrod]

We have the same offsite requirement. I engineered my own offsite backup setup using off the shelf components.

Not knowing powershell, I wrote a dos batch file to copy the veeam backups folder to an external hard drive. I have 2 physical servers that backup exec writes a disk2disk backup up to a folder on same drive with veeam Backups. My DOS DRCOPY script saves them to external drive as well.
I put an esata controller in my veeam backup server, and attached an external 2TB drive in an external Esata docking bay.

http://www.istarusa.com/storage/docking/xagen99us.aspx

They also have a 2 drive hot dock available as well.
The controller, cable and docking bay was less than $100. I rotate thru the external drives, taking one offsite each nite. Takes about 4 hours to load 1TB of data onto the external drive, faster than tape, and quicker to restore. I run the DRCOPY script around 9:00am, its usually done right after lunch. 1TB drives are less than $100, and 2TB drives are less than $200. Near that same cost as equivalent LTO2 tape storage.

I had some issues with drive letters being lost when hot swapping the external drives, so I ended up using folder mount points, and mounted each drive to to its own folder. Now when I hot swap them, the drive appears under the correct folder automatically, after I mounted them the frst time. Initialinzing the disk first time adds a registry entry that seems to remember the Drive ID, and the corresponding folder mount point.

I use:
d:/hotswap/HD1
d:/hotswap/HD2
d:/hotswap/HD3

My DOS script accepts a Parm of Drive ID, (HD1, etc) so it knows where to write the copies.

I had to download a free program for my Silicon Image esata controller, called hotswap. Its puts another "safely remove" icon in the tray, which lets me shut down the esata drive before I hotswap it, flushing all writes to the drive 1st.

I have one more tweak to do to the DOS bat file, to select only the most recent .VBK files.
The bat file also copies my DR recovery software onto the offsite HD each time, consisting of the win2k3 srv ISO, backup exec install software, and veeam install software. Everything I need to rebuild a recovery server on a physical machine in the event of a total disaster.

I will probably increase my drives from 3 to 5, one for each day of the week. Then I could schedule the DRcopys, one HD assigned to each day mon-fri.

If you needed more offsite storage, you could use the 2 bay hot dock, use two 2TB drives, and write to them both simultaneously.
Should be able to backup 4TB to 2 disks in about 8hrs, with this setup.

I can share the DOS bat file if anyone wishes, its nothing fancy tho.

P.S. for Veeam onsite backup purposes, I put a 4 bay trayless sata enclosure ($220) attached to an HP P600 SAS/SATA raid controller in my Veeam server. Four 2TB drives in raid 5. Single SFF8044(?) cable connects HP P600 to external enclosure. This let me add 5.5TB usable storage for a veeam backup target. P600 was used from ebay, so the whole deal only cost $1200.

Veeam uses Vstorage API over fiber to suck data right out of the SAN, and write it to local attached disk. No Network load, and its fast.

[H in OH]

We have the same offsite requirement. I engineered my own offsite backup setup using off the shelf components.
...
I put an esata controller in my veeam backup server, and attached an external 2TB drive in an external Esata docking bay.
...
I had some issues with drive letters being lost when hot swapping the external drives, so I ended up using folder mount points, and mounted each drive to to its own folder. Now when I hot swap them, the drive appears under the correct folder automatically, after I mounted them the frst time. Initialinzing the disk first time adds a registry entry that seems to remember the Drive ID, and the corresponding folder mount point.

This is what gets you with externally attached disk solutions. If you are working on site, or there is at least a semi-capable Windows admin on site, then you could get away with using removable disks. The problem is when you have to set something up to be run by completely non-technical on site personnel. Anyone can replace a tape, and some tape backup solutions include an automated email with the correct tape name for the next backup.