Comprehensive data protection for all workloads
Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jun 19, 2009 2:06 pm

You don't need previous nights tape with previous version of VBK in such scenario. Inconsistent VBK does not mean you cannot perform restore. Obviously you cannot restore from VBK alone because backup cycle did not complete (and VBK represents last backup cycle which had failed), but you can always restore using rollback data - restore process will automatically leverage rollback file to reconstruct consistent VBK to the state of last successful backup (in other words, same as "previous nights tape" content).

Just to clarify this further, before the incoming incremental data block is injected into VBK file, the data block that is being replaced has to be successfully stored in the reversed increment (or "rollback") file (*.VRB) first. This is why data cannot be lost even in case of system crash happening during VBK data block commit.

As for writing to a separate VBK file - this would not work just because there is no separate VBK file, there is only one, with some of its blocks being updated with changed blocks brought over from source. Also, this algorithm would require 2x of backup storage space during backup.

Tom - I am not sure I understand the issue with having multiple jobs running. Have you read this feedback from another customer which has about the same amount of data that needs to be backed up? He is doing this and it works well for him on ESX 3.5.

tsightler
VP, Product Management
Posts: 5305
Liked: 2160 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Wishlist / Issues

Post by tsightler » Jun 19, 2009 11:57 pm

Gostev wrote: Just to clarify this further, before the incoming incremental data block is injected into VBK file, the data block that is being replaced has to be successfully stored in the reversed increment (or "rollback") file (*.VRB) first. This is why data cannot be lost even in case of system crash happening during VBK data block commit.
Well, there are a couple of problems with this:

1. I've found that after a crash backup there's probably a 10-15% chance that the .vbk file will simply not be recoverable, in our testing of "hard crash" scenarios we've had 4 vbk files get into this unrecoverable state. True disaster scenarios can come in all types, a friend of had their IBM Shark storage literally catch fire and pretty much burn up before fire suppression managed to put out the fire. The storage array and data were pretty much lost but he had other arrays on which to restore his data. Now let's say he was using Veeam backup and this happend while the backups were running. Would he be able to restore that data? I'm sure some would, but based on my testing it's likely that at least one .vbk file would be corrupt.

2. Recovering a damaged .vbk file can take quite a while. One of our host is 520GB, when we aborted the backup and tested a restore it took almost an hour to "roll back" the damaged file and perform a restore.
Gostev wrote: As for writing to a separate VBK file - this would not work just because there is no separate VBK file, there is only one, with some of its blocks being updated with changed blocks brought over from source. Also, this algorithm would require 2x of backup storage space during backup.
Um, I didn't say to write a new VBK I said to write the "changed blocks" to a separate file and then commit those into the VBK file at the end. You already have the ability to take a vbk file and apply a roll-back (vbr) file, so how difficult could it possibly be to extend this concept to create a roll-forward (maybe a vbf) file which contains blocks to commit and then commit those block to the VBK file at the end, creating the new VBR as you go. This is not a difficult concept, it's how VMware snapshots already work. The different between rolling forward and rolling back is not really significant.
Gostev wrote: Tom - I am not sure I understand the issue with having multiple jobs running. Have you read this feedback from another customer which has about the same amount of data that needs to be backed up? He is doing this and it works well for him on ESX 3.5.
Yes, I read that. Do you notice how he had to use multiple servers and create 18 total jobs. If he added more machines or his storage grows he'll likely need even more and he'll have to manually create and manage even more jobs. We just migrated from vRanger to Veeam and currently have about 20 jobs scheduled to start every 5 minutes or so. If I add a new host I'll probably need add another job. Do you know how many jobs I had on vRanger? I had 2 jobs total, that's it. Each job backed up a Folder called "Image Backups" on a different ESX datacenter. It had a set for the maximum simultaneous jobs to run and we generally ran 4 jobs simultaneously from each datacenter as that did a pretty good job of loading the network/storage. So those 2 jobs would actually run 8 simultaneous backup tasks and always keep 8 running, when one machine would finish it would move on to the next machine in the folder. If we created a new machine that we wanted to get image backups we simply made sure it was in that folder and it would be run simultaneously with the other backups without us even touching the backup server.

I know that Veeam does support backing a VirtualCenter folder, but a backup of any given folder is run sequentially and that just doesn't provide the throughput to get our backups done within the required window. It's the one thing where vRanger was actually much better than Veeam. We can work with the current scheduler to get the behavior we want, but it's quite a bit more administrative load than simply creating the jobs and letting the backup system manage the load.

Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jun 20, 2009 7:25 pm

Hi Tom, this interesting discussion - thank you for your time.
tsightler wrote:I've found that after a crash backup there's probably a 10-15% chance that the .vbk file will simply not be recoverable, in our testing of "hard crash" scenarios we've had 4 vbk files get into this unrecoverable state. True disaster scenarios can come in all types, a friend of had their IBM Shark storage literally catch fire and pretty much burn up before fire suppression managed to put out the fire. The storage array and data were pretty much lost but he had other arrays on which to restore his data. Now let's say he was using Veeam backup and this happend while the backups were running. Would he be able to restore that data? I'm sure some would, but based on my testing it's likely that at least one .vbk file would be corrupt.
Have you been testing this with the latest release? We have addressed couple of scenarios which could lead to this when backup server crashes during backup in 3.0.1 and 3.1. I am not saying that the chance is 0% now - every crash is different in nature because crash moment is random, but this chance should be very low now. Also, keep in mind that in case when UI recovery errors out, we should still be able to do this with our support tool. Restore with UI may error due to corrupt backup meta data (that has much higher chance to get corrupted during crash because its update is not transactional currently), while all actual backup data can still be recovered.
tsightler wrote:Recovering a damaged .vbk file can take quite a while. One of our host is 520GB, when we aborted the backup and tested a restore it took almost an hour to "roll back" the damaged file and perform a restore.
At this time, this seems to be pretty fair time to process 520GB, because the process involves random disk reads/writes. But we are making adding some significant enhancements to backup storage in 4.0, so all operations with backup files should become faster.
tsightler wrote:Um, I didn't say to write a new VBK I said to write the "changed blocks" to a separate file and then commit those into the VBK file at the end. You already have the ability to take a vbk file and apply a roll-back (vbr) file, so how difficult could it possibly be to extend this concept to create a roll-forward (maybe a vbf) file which contains blocks to commit and then commit those block to the VBK file at the end, creating the new VBR as you go. This is not a difficult concept, it's how VMware snapshots already work. The different between rolling forward and rolling back is not really significant.
Yes, Veeam Backup 4.0 enhanced storage has similar concepts, although we are not implementing these enhancements for the same reason alone. We've taken a complex approach because backup storage seems to affect just about anything: full and incremental backup performance, repair time, crash protection, integration with 3rd party storage device (aka DataDomain CIFS integration issue), etc. And many of these requirements are typically conflicting with each other (speed vs. reliability for instance). So now that we have collected all the intelligence, we have designed the "best of all worlds" implementation.

Indeed, it was actually a big problem for us to design complete VBK protection from backup server crashes without affecting backup performance heavily (frankly, I thought it would be impossible). And my biggest concern was that if product evaluator does not see good backup performance - they just go away and we lose the customer - he or she simply would not get to the next step, evaluating our awesome backup reliability. Anyway, our devs seem to have made the impossible... yet again. By the way, it would be awesome if you could find some time to participate in closed beta when it is available and help us verify that we got it all covered, since from what you said above you clearly have great experience with VBK crash-tests ;)
tsightler wrote:We just migrated from vRanger to Veeam and currently have about 20 jobs scheduled to start every 5 minutes or so.
Tom, could you please clarify why do you need this many jobs. Based on feedback, I was under impression that in case of VCB SAN backup with FC4 SAN, backup storage speed is always a bottleneck. Meaning that it does not really help to run more than 1-2 jobs hitting the same backup storage. And when you want to help target backup storage and use High compression, even 1 job fully saturates Veeam Backup server CPU even on quad core CPU, so multiple jobs do not help again. Which one is your scenario, and what is performance differences for running 2 jobs versus multiple jobs?

I just got Google alert for new blog post from Steve Philip, our community member (sphilp). By the way, Steve - great post, I am adding it to the "Feedback from the Field" thread - will ask devs to investigate CIFS issue too). Anyway, Steve is also concluding that it is backup storage which defines how fast your VCB SAN backups can go. Here is the link to the blog post: http://sphilp.blogspot.com/2009/06/vmwa ... up-31.html

tsightler
VP, Product Management
Posts: 5305
Liked: 2160 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Wishlist / Issues

Post by tsightler » Jun 21, 2009 3:30 am

Gostov wrote:Hi Tom, this interesting discussion - thank you for your time.
No problem. I want to make it clear that we love Veeam, we think it is a great backup tool and is so much better than our previous solution and we understand that every environment is different, but this was a "Wishlist" thread so I threw ours out there.
Gostov wrote:Have you been testing this with the latest release? We have addressed couple of scenarios which could lead to this when backup server crashes during backup in 3.0.1 and 3.1. I am not saying that the chance is 0% now - every crash is different in nature because crash moment is random, but this chance should be very low now. Also, keep in mind that in case when UI recovery errors out, we should still be able to do this with our support tool. Restore with UI may error due to corrupt backup meta data (that has much higher chance to get corrupted during crash because its update is not transactional currently), while all actual backup data can still be recovered.
We're testing with 3.1 that we downloaded about two weeks ago. I'm pretty sure that's the latest. Our testing basically involves simple test like unplugging the network cable of the backup server, rebooting the server during a backup, etc.
Gostov wrote:At this time, this seems to be pretty fair time to process 520GB, because the process involves random disk reads/writes. But we are making adding some significant enhancements to backup storage in 4.0, so all operations with backup files should become faster.
I didn't say that this was slow, but it would be instant if the changes were backed up to a separate file first. Actually, we have other reasons we wish the VBK didn't change on every backup. We have almost 3TB of vbk files at the end of our backups, these must all then be spooled to tape every night. It would be much nicer if the VBK file only changed once a week, writing change files and then consolidating the changes once a week would eliminate that problem.
Gostov wrote:Yes, Veeam Backup 4.0 enhanced storage has similar concepts, although we are not implementing these enhancements for the same reason alone. We've taken a complex approach because backup storage seems to affect just about anything: full and incremental backup performance, repair time, crash protection, integration with 3rd party storage device (aka DataDomain CIFS integration issue), etc. And many of these requirements are typically conflicting with each other (speed vs. reliability for instance). So now that we have collected all the intelligence, we have designed the "best of all worlds" implementation.

By the way, it would be awesome if you could find some time to participate in closed beta when it is available and help us verify that we got it all covered, since from what you said above you clearly have great experience with VBK crash-tests ;)
Veeam 4.0 sounds great, I can hardly wait. We'd love to test the beta when you get to that point. I'll be honest, one of the reasons we switched to Veeam Backup was because of the strives you've made in the product in the last year. We looked at the product at the 1.0 version and it just wasn't able to meet our requirements, then after living a year with vRanger and not being very happy we took another look a few weeks ago and couldn't believe how far the product had progressed, still not perfect, but unbelievably improved. We expect to see the same great things from 4.0.
tsightler wrote: Tom, could you please clarify why do you need this many jobs. Based on feedback, I was under impression that in case of VCB SAN backup with FC4 SAN, backup storage speed is always a bottleneck. Meaning that it does not really help to run more than 1-2 jobs hitting the same backup storage. And when you want to help target backup storage and use High compression, even 1 job fully saturates Veeam Backup server CPU even on quad core CPU, so multiple jobs do not help again. Which one is your scenario, and what is performance differences for running 2 jobs versus multiple jobs?
OK, I'll try to describe our environment. We an all iSCSI shop, no FC. We use Dell/Equallogic storage arrays and currently have 3 of them, they can easily run at 180MB/sec each for a total of over 540MB/sec. Our backup server has 4iSCSI HBA's and thus tops out at a little over 480MB/sec. For some reason though VM backups seem to be limited to 35-40MB/sec each, this happens no matter what mode we use, network backups, VCB SAN, etc. I'm not really sure why, but my suspicion is a fairly low queue depth combined with the added latency of iSCSI works against it's top speed. Anyway, the only way we've been able to get enough speed is to run more jobs. We've found that 4 jobs per array get's pretty close to maxing the capacity while leaving a little headroom for normal operations and running 10-12 jobs gives us 350-450MB/sec total performance. We use a mix of network backups and VCB SAN since VCB backups seem to be limited to 8 simultaneous jobs and network backups work well for offloading some of the read load.

We also have a lot of jobs because most of our VM's are not very similar and many are quite large (500GB+). We use many jobs to keep the VM count down and the backup sizes reasonable. We also backup production instances to individual backups because we spool them to tape and we need the fastest possible restore time. We're not currently using best compression, we just don't have enough hardware to run that many jobs with best compression although it seems to mainly be only on the initial full.

Overall we're pretty happy with the current schedule the only problem is that we have to manually spread out the jobs to try to balance keeping 10-12 running, vRanger did this automatically. Not a real big deal, just a little extra admin work.

Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jun 21, 2009 9:03 am

tsightler wrote:Our backup server has 4iSCSI HBA's and thus tops out at a little over 480MB/sec. For some reason though VM backups seem to be limited to 35-40MB/sec each, this happens no matter what mode we use, network backups, VCB SAN, etc. I'm not really sure why, but my suspicion is a fairly low queue depth combined with the added latency of iSCSI works against it's top speed.
I believe the reason for this is that VCB does not like multipathing, which you probably have set up since you have 4 HBA's. Many of our customers have reported that their backup speed going up a few times from 30-40MB/s after they disable multipathing.

tsightler
VP, Product Management
Posts: 5305
Liked: 2160 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Wishlist / Issues

Post by tsightler » Jun 21, 2009 4:21 pm

Gostev wrote: I believe the reason for this is that VCB does not like multipathing, which you probably have set up since you have 4 HBA's. Many of our customers have reported that their backup speed going up a few times from 30-40MB/s after they disable multipathing.
I read those reports, and we've tried both completely without multipath (we thought one link per array would be OK) and with different multipath modes (failover, least queue depth, and round robin) and in our case this has absolutely zero impact on the backup performance. We also see the same speed with "direct to target" mode using the network backup mode, both agent and agentless. While I have no proof, I still think it's the latency/queue depth/minimal read-ahead combination that works against iSCSI configurations. With some tweaking we've managed to get to around 50MB/sec per backup using VCB SAN on incrementals so running 6-8 jobs (2-3 per array/link) should get really close to saturating our links/SAN.

We're also not a big fan of backing up many VM's to a single VBK. If we do have a damaged VBK we loose all of the backups in it and you can't restore from a VBK while a backup is running. I know you stated that your support should be able to recover data from a damaged VBK but having to contact support to restore data is not high on our wish list. :wink:

I should note that most of our VM's contain lots of data so we don't get the performance advantage of VM's with a lot of zero'd space. We do have a few that are pretty empty and those VM's show speeds as high as 90MB/sec.

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Veeam Wishlist / Issues

Post by lohelle » Jun 21, 2009 7:04 pm

I would like to see an option to create one vbk per VM actually. In my small 50 VM environment this makes me more happy. It is much easier to move those files off to other storage later on then.

Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jun 29, 2009 2:38 pm

Gostev wrote:Pause/unpause scheduled runs (internally, we call this functionality "temporary disable job") is actually in the list of low priority features for 4.0. Meaning, we are planning to add this feature if time allows to do it without slipping the release. So chances are good! Thank you.
Just wanted to update that this feature has made it into Veeam Backup 4.0...

Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jun 29, 2009 2:41 pm

lohelle wrote:I would like to see an option to create one vbk per VM actually. In my small 50 VM environment this makes me more happy. It is much easier to move those files off to other storage later on then.
Lars, I think I missed this post. Would it be OK if instead of backing up to individual files Veeam Backup still backs up all the data to single VBK, but then provides option automatic action to extract all or only required VMs to the designated share or folder? This is more realistic functionality which would not require completely revamping the product's architecture.

dlove
Influencer
Posts: 18
Liked: never
Joined: May 15, 2009 1:51 pm
Full Name: darren
Contact:

Re: Veeam Wishlist / Issues

Post by dlove » Jul 22, 2009 11:50 pm

I'd personally like to be able to backup via VCB (SAN) and then on failure be able to retry using LAN. Every night I have odd ball servers that fail when using VCB and then work fine when doing network backups. It's not always the same servers but it's annoying to go in and change the job temporarily to network then retry the job which finishes successfully and then change back the job to VCB. Vizioncore had this feature on the 3.x vranger that we gladly left behind us :D

Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jul 23, 2009 10:19 am

Yes, we have this feature in vStorage API backup mode of Veeam Backup 4.0.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Veeam Wishlist / Issues

Post by withanh » Jul 23, 2009 3:06 pm

Sounds like Veeam v4 has a lot of nice new features that I would like to use as well.

What is the expected release time frame for v4?
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24092
Liked: 3278 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Wishlist / Issues

Post by Gostev » Jul 23, 2009 3:14 pm

We are looking to RTM the code in the end of Q3.

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Veeam Wishlist / Issues

Post by lohelle » Jul 30, 2009 7:59 pm

I would like the possibility to put the diff-files on a separate location. This would be nice when the target storage is often almost (and completely) full.

tsightler
VP, Product Management
Posts: 5305
Liked: 2160 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Wishlist / Issues

Post by tsightler » Aug 11, 2009 5:21 pm

So, after using Veeam for a few months I've come up with a few more things to add to the "Wishlist":

1. A fully functional Linux file level restore. The current solution is OK, but has a lot of drawbacks. It crashes when attempting to restore files with names that include characters unsupported in Windows. It looses all file attributes, including ctime, atime, etc.

2. A command line utility to mount a VBK/VBR backup on linux directly. This would pretty much solve the linux file level restore problem for me since you could simply mount the VBK/VBR file on the linux server and copy the files with native tools.

3. A faster file level restore when restoring a moderate to large number of files. We recently needed to restore a directory that contained ~35,000 files but the total size was only about 10GB. This took about 90 minutes with Veeam. Since this system was also still being backed up with our legacy tape backup solution we decided to compare how long the same restore would have taken from tape via this method and it took only 40 minutes.

4. A really nice feature would be a tool that could index the files within the images similar to legacy file based systems. I understand that this might be unrealistic for an image based backup solution, but I could envision a feature where, after the backup was complete, the backup images were mounted and indexed to allow files to be searchable, offering the best of image and file level backups. This is really important for sites like us where we may need to restore a file from a year ago. With our current solution we can find the file and it will tell us to go get the archive tape from 10 months ago (the most recent file that contains the tape). Even now that we are archiving Veeam backups we no longer have an index. If someone needed a file from a year ago we'd have to grab multiple tapes and start restoring files looking for the vbk file that might have the file we needed.

I understand that #4 might not be reasonable, but it is a wishlist afterall.

Locked

Who is online

Users browsing this forum: Bing [Bot], foggy, jmmarton, kyle.shuberg and 30 guests