Veeam Wishlist / Issues

[mdornfeld]

I think I came into Veeam expecting too much. Maybe Veeam sales just did a really good job, or maybe I just have unrealistic expectations for backing up my VM environment, probably a good mix of the two. Either way, I've been collecting a list of things that I would like to see improved, and most of them Veeam I'm sure is well aware of, but just in case, I wanted to list them all. As I've seen in the past, for some of them, Veeam will point directly at VMware and say, "there's nothing we can do, it's what we have to work with", and while that's understandable, I don't think that's the right approach. I guess my expectation as a customer is that Veeam will use their relationship with VMware to get these issues fixed on VMware's side as well as Veeam's side.

Backup & Restore
-Handle the restore of Virtual RDM's correctly
-Skip over reading blank space when doing backups
-Fix the issue where simultaneous backup jobs of highly transactional servers causes backups to fail.
-Removing a server from a Veeam backup job doesn't shrink the backup VBK file appropriately
Explaination: I removed a rather large server from our backup job, and the backup file stayed at 165 GB well past the number of rollbacks to keep. After I blew away the full backup, the next time it created a backup file, it was only 38 GB. In my mind, it should have auto-shrunk to the correct size on it's own.
-Increase speed of both backup and especially restoration.
-Become MS Exchange and MS SQL aware so it can take smarter/faster incrementals.

Scheduling
-Clean up scheduling engine (let me be able to specify exact multiple times per day that I want it to run a job - like windows scheduler)
-Enable me to do a grandfather, father, son rotation without scheduling 3 different jobs and having 3 different huge fulls hanging around.
-Enable the tape engine backup software to send/receive the following commands
1. Resume a Backup Job Schedule
2. Stop the scheduler for a particular backup job
3. See if a Veeam backup job is still in progress.

Other Issues
-The Veeam console acts flaky/crashes when opening it up in multiple sessions or closing it when a job is in progress.
-Veeam Support Logs - Everytime I seem to need the logs because a job failed/hung/whatever, they don't seem to be created in "Support Information" area. It's like they're deleted when the job fails or something.
-Stopping a Veeam job is like trying to stop a freight train. You can ask it to stop, but it's going to keep doing whatever it wants to do.

I hope Veeam takes this as constructive critisism. We obviously still chose to use Veeam over it's competitors, but we customers never focus on what's working right, do we

Gustov, is it possible to get a status on the aforementioned items in the list?

Thank you

[Gostev]

Matt, thank you for your feedback. Yes, there is always place for improvement

On the other hand, we've made something in 1.5 years that our competitors still do not have, having 1.0 version of their solutions released 3-4 years ago. So you can count on us - we will always be a few steps ahead. We have the best team in the world!

Good to hear our sales do good job

Yes, you may hear sometimes from support that "it is VMware problem", but they merely indicate this fact - it does not mean we are not planning on addressing this issue from our side. This is why I like these community forums - they give me visibility to those replies which make our customers not very happy. On the other hand, the problem we are seeing right now is VMware restricting all vendors to the use of public APIs only. And there is very little you can do to address VMware issues by using their own public APIs (which obviously all suffer from said issues too).

I've copied your bullet list below, see my answer in red.

Backup & Restore
-Handle the restore of Virtual RDM's correctly
If you are talking about produced VMDK naming issue, I believe it will be possible to resolve even with fix for current version. I can discuss this with our project lead when he returns from vacation.
-Skip over reading blank space when doing backups
For incremental backup: 4.0 (requires ESX 4.0 or later, will read changed blocks only).
For initial full backup: planned for 5.0
-Fix the issue where simultaneous backup jobs of highly transactional servers causes backups to fail.
Need more information on this one, but we did change some default freeze timeouts in 3.1 to address this issue.
-Removing a server from a Veeam backup job doesn't shrink the backup VBK file appropriately
Explaination: I removed a rather large server from our backup job, and the backup file stayed at 165 GB well past the number of rollbacks to keep. After I blew away the full backup, the next time it created a backup file, it was only 38 GB. In my mind, it should have auto-shrunk to the correct size on it's own.
4.0 via perform full backup command
-Increase speed of both backup and especially restoration.
Backup: significant improvements in 4.0 with ESX 4.0 hosts
Restore: planned for 4.5 via direct-to-SAN restores
-Become MS Exchange and MS SQL aware so it can take smarter/faster incrementals.
Not sure what you mean here. But again, 4.0 will have extremely fast incrementals with ESX 4.0 hosts (less than a minute).

Scheduling
-Clean up scheduling engine (let me be able to specify exact multiple times per day that I want it to run a job - like windows scheduler)
Not currently planned: based on feedback, such schedules are rarely used, and can be still implemented today by leveraging Windows scheduler and making it start Veeam Backup job from command line.
-Enable me to do a grandfather, father, son rotation without scheduling 3 different jobs and having 3 different huge fulls hanging around.
Please clarify?
-Enable the tape engine backup software to send/receive the following commands
1. Resume a Backup Job Schedule
2. Stop the scheduler for a particular backup job
3. See if a Veeam backup job is still in progress.
Can you clarify why this does not work for you? http://www.veeam.com/forums/viewtopic.php?p=5230#p5230
We are also considering native tape integration, and it is very high on our priorities list.

Other Issues
-The Veeam console acts flaky/crashes when opening it up in multiple sessions or closing it when a job is in progress.
Multi-user job management UI is a part of Veeam Backup 4.0
-Veeam Support Logs - Everytime I seem to need the logs because a job failed/hung/whatever, they don't seem to be created in "Support Information" area. It's like they're deleted when the job fails or something.
This issue with logs was never reported before. Logs are always created for each job, but they are never deleted in case of a failure of crash. If you can reproduce this behavior, we would be interested to investigate.
-Stopping a Veeam job is like trying to stop a freight train. You can ask it to stop, but it's going to keep doing whatever it wants to do.
Yes, it takes time to commit all VBK transactions, shutdows all agents and/or VCB processes, and especially to remove created snapshot. If we don't perform these operation and exit dirty, next job runs will still be failing, and there will be snapshot left - this would be very bad for us to do.

[mdornfeld]

-Enable the tape engine backup software to send/receive the following commands
1. Resume a Backup Job Schedule
2. Stop the scheduler for a particular backup job
3. See if a Veeam backup job is still in progress.
Can you clarify why this does not work for you? http://www.veeam.com/forums/viewtopic.php?p=5230#p5230

Let me back up off this request, and make a much simpler request that would have more universal appeal. We're troubleshooting some issues right now, and the only way I have to stop a scheduled Veeam job from running is to go into the properties, write down the current schedule, reschedule it to not run, and then when I want to start it's regular schedule again, I have to go back in and reconfigure the whole schedule. Why isn't there "pause/unpause schedule" functionality, (much akin to Windows Scheduler "Enabled" check box).

Now it might be fair to ask me that if I like the functionality of windows scheduler so much, why don't I just use that? Here's the reason. I, personally, want a nice, integrated interface, that can tell me everything I want to know specifically about my VMware backup jobs in one place. This is a bit tongue-in-cheek, but If I didn't want that, I could have just as easily used a competitor's product. This is the same reason I don't want to use my tape engine to do all the scheduling (besides, my tape engine has enough problems running it's own backup jobs).

In all seriousness though, if I'm the only one who wants the ability to pause/unpause schedule ability, I'd say toss it out the window. But I guess my assumption is that other people would be interested in it as well. I'm sure you have ways (including these forums) of determining what your customers want.

Thanks as always for your help Gustov. I will try to clarify some other items when I have time.

[Gostev]

Pause/unpause scheduled runs (internally, we call this functionality "temporary disable job") is actually in the list of low priority features for 4.0. Meaning, we are planning to add this feature if time allows to do it without slipping the release. So chances are good! Thank you.

[mdornfeld]

-Become MS Exchange and MS SQL aware so it can take smarter/faster incrementals.
Not sure what you mean here. But again, 4.0 will have extremely fast incrementals with ESX 4.0 hosts (less than a minute).

Two things I mean here:
1. My *assumption* is that if Veeam sees an exchange or SQL database file change by one byte, it thinks it has to back up the entire file all over again, not just the changes that were made within that database file.
2. Have Veeam truncate trans logs for both SQL/Exchange.

[mdornfeld]

-Enable me to do a grandfather, father, son rotation without scheduling 3 different jobs and having 3 different huge fulls hanging around.
Please clarify?

Example: Currently I can hang onto x amount of rollbacks. But lets say for instance I have a server backing up every 2 hours. I would like to retain the last 12 backups online (the last full day), then one backup per day for the last 7 days, and then one backup per a week for the last 4 weeks.

To do this currently with your software, I would have to have 3 jobs:
-2 hr backup job with 12 rollbacks
-Daily backup job with 7 rollbacks
-Weekly backup job with 4 rollbacks
This equates to 3 separate fulls.

Right now we accomplish this with just going to tape and using our tape rotation, so it's definately lower priority, but in the future it will become more useful as we phase out tape more and more.

[mdornfeld]

We are also considering native tape integration, and it is very high on our priorities list.

Just my 2 cents. The company I work for would have very little current interest in this feature. We would definately like to see your developement efforts go towards getting the VMware backup to disk perfected.

[lohelle]

1.
I would like to be able to select VM's to be backed up in a job based on resource pools. Now we can select all VM's in a host and in a cluster (and individual), but not by resource pools / "folders".

Example:
Lets say we have 50 VM's in a Cluster. This cluster changing a lot. A lot of the VM's use local storage and others SAN storage. If we have two or more resource pools that separate these we could create 1 job for "local storage"-VM's and one for SAN storage. Without this we would need to update the job every time a VM is added to the cluster (or use service console backup for all VM's)

2.
When using service console backup:
Automaticly start one backup per host simultaneous (or x backups). This way we can create ONE job for all hosts, and still run many simultaneous backups at the same time. Simultaneous VCB backups could also be a nice feature, but then x jobs per SAN or target

3.
Failover to service console/agentless backup if VCB backup failes.

[Gostev]

1. My *assumption* is that if Veeam sees an exchange or SQL database file change by one byte, it thinks it has to back up the entire file all over again, not just the changes that were made within that database file.

No, actually Veeam does incremental backup based on VMDK block level (1024KB block). So with your example, if Exchange or SQL database file changes by one byte, only 1024KB block involving this byte is transferred. If more than one byte changes, all blocks that got "dirty" due to this are getting transferred. For example, if you apply 4.5MB of new data, it makes 5 blocks dirty, in that case transfer "overhead" is just 0.5MB.

2. Have Veeam truncate trans logs for both SQL/Exchange.

If you are using Veeam VSS intergration, this should happen automatically after backup is done, as a part of regular Exchange VSS Writer activities (this is not a part of Veeam Backup functionality though).

Example... [skipped] ...Right now we accomplish this with just going to tape and using our tape rotation, so it's definately lower priority, but in the future it will become more useful as we phase out tape more and more.

Thanks for clarification, I understand. Definitely makes sense. I see it as a part of advanced scheduler we have been talking about above.

Just my 2 cents. The company I work for would have very little current interest in this native tape integration. We would definately like to see your developement efforts go towards getting the VMware backup to disk perfected.

In fact, these are very valuable 2 cents.
We are also having big internal debates about this specific feature.

[Gostev]

I would like to be able to select VM's to be backed up in a job based on resource pools. Now we can select all VM's in a host and in a cluster (and individual), but not by resource pools / "folders".

I will check if we can add resource pools objects in selection in 4.0, this should be easy to do. But we already provide ability to create jobs by VM folders today. There is view mode selection above the UI control you use to pick the objects for backup.

When using service console backup: Automaticly start one backup per host simultaneous (or x backups). This way we can create ONE job for all hosts, and still run many simultaneous backups at the same time. Simultaneous VCB backups could also be a nice feature, but then x jobs per SAN or target

First, I wanted to note that at this time we did not have plans to enhance service console agent backup, because 2010 will be the last year for "fat" ESX. But we certainly have plans on implementing this functionality for remaining backup modes, and I see nothing preventing us from using this logic with network backup mode, since many restrictions do not really depend on backup mode (like ensuring that each LUN does not have more than one snapshot operation at a time, or preventing more than X jobs hitting the same target etc).

Failover to service console/agentless backup if VCB backup failes.

This is already implemented in the upcoming Veeam Backup 4.0, in vStorage API backup mode (VCB replacement).

[lohelle]

I will check if we can add resource pools objects in selection in 4.0, this should be easy to do. But we already provide ability to create jobs by VM folders today. There is view mode selection above the UI control you use to pick the objects for backup.

Ahh! I missed that one. Thank you!

[me123]

This is already implemented in the upcoming Veeam Backup 4.0, in vStorage API backup mode (VCB replacement).

Does this mean that we will be able to have a SAN mode backup without having to use VCB?

[Gostev]

That is correct, so no more VCB babysitting.

[tsightler]

I think my biggest wish would be the ability to set the number of simultaneous backups. We have about 40 VM's with about 7TB of data to be backed up. Since Veeam requires a full scan of every volume every night that means we pretty much have to read all 7TB of disk. Our typical backup speed is about 40-50MB/sec per backup, but we have about 500MB/sec available from the disk array. That's more than enough bandwidth to get the backups done in our overnight windows, but we have to run around 8-10 backups simultaneously to do it. That means trying to play with schedules to keep a reasonable number of jobs running. We end up using a mix of Network and VCB backups to get it done (network backups allow us to spread the load across many ESX servers and are great for "smallish" machines).

We considered simply grouping our backups into 8 or so jobs that are roughly the same size, but this leaves us with vbk files that are huge and, if there's a failure, the rollback can take and hour or more and the vbk is not usable for restores.

Actually, that's another big deal. A failed backup should not cause the vbk to be unusable. As it is now, what happens if disaster strikes during a backup. Your disk backups are no good and you'd likely be restoring your vbk's from your previous nights tape. I would be better if a new backup would write it's changes to a temporary file, and then commit the changes at the end. This would have a little more overhead, and wouldn't eliminate the possibility of a corrupt vbk (it could crash during the commit) but, in the more normal cases of a backup failing during it's processing, the recovery is just to delete the temp file, not to rollback transactions into the main file and hope for the best.

We came from vRanger and while we weren't very happy with it, mainly the pretty much useless file level restore, it was nice to be able to simply tell it what machines to backup, how many jobs to run simultaneously, and let it manage the details.

I realize this is probably going to be much less of an issue with vSphere 4 and the block change tracking, so maybe none of these issues are worth the effort, but I suspect many of us will be on ESX3.x for quite some time.

[mdornfeld]

Actually, that's another big deal. A failed backup should not cause the vbk to be unusable. As it is now, what happens if disaster strikes during a backup. Your disk backups are no good and you'd likely be restoring your vbk's from your previous nights tape. I would be better if a new backup would write it's changes to a temporary file, and then commit the changes at the end. This would have a little more overhead, and wouldn't eliminate the possibility of a corrupt vbk (it could crash during the commit) but, in the more normal cases of a backup failing during it's processing, the recovery is just to delete the temp file, not to rollback transactions into the main file and hope for the best.

I wholeheartedly agree with the above statement. Is Veeam looking into this?

[Gostev]

You don't need previous nights tape with previous version of VBK in such scenario. Inconsistent VBK does not mean you cannot perform restore. Obviously you cannot restore from VBK alone because backup cycle did not complete (and VBK represents last backup cycle which had failed), but you can always restore using rollback data - restore process will automatically leverage rollback file to reconstruct consistent VBK to the state of last successful backup (in other words, same as "previous nights tape" content).

Just to clarify this further, before the incoming incremental data block is injected into VBK file, the data block that is being replaced has to be successfully stored in the reversed increment (or "rollback") file (*.VRB) first. This is why data cannot be lost even in case of system crash happening during VBK data block commit.

As for writing to a separate VBK file - this would not work just because there is no separate VBK file, there is only one, with some of its blocks being updated with changed blocks brought over from source. Also, this algorithm would require 2x of backup storage space during backup.

Tom - I am not sure I understand the issue with having multiple jobs running. Have you read this feedback from another customer which has about the same amount of data that needs to be backed up? He is doing this and it works well for him on ESX 3.5.

[tsightler]

Just to clarify this further, before the incoming incremental data block is injected into VBK file, the data block that is being replaced has to be successfully stored in the reversed increment (or "rollback") file (*.VRB) first. This is why data cannot be lost even in case of system crash happening during VBK data block commit.

Well, there are a couple of problems with this:

1. I've found that after a crash backup there's probably a 10-15% chance that the .vbk file will simply not be recoverable, in our testing of "hard crash" scenarios we've had 4 vbk files get into this unrecoverable state. True disaster scenarios can come in all types, a friend of had their IBM Shark storage literally catch fire and pretty much burn up before fire suppression managed to put out the fire. The storage array and data were pretty much lost but he had other arrays on which to restore his data. Now let's say he was using Veeam backup and this happend while the backups were running. Would he be able to restore that data? I'm sure some would, but based on my testing it's likely that at least one .vbk file would be corrupt.

2. Recovering a damaged .vbk file can take quite a while. One of our host is 520GB, when we aborted the backup and tested a restore it took almost an hour to "roll back" the damaged file and perform a restore.

As for writing to a separate VBK file - this would not work just because there is no separate VBK file, there is only one, with some of its blocks being updated with changed blocks brought over from source. Also, this algorithm would require 2x of backup storage space during backup.

Um, I didn't say to write a new VBK I said to write the "changed blocks" to a separate file and then commit those into the VBK file at the end. You already have the ability to take a vbk file and apply a roll-back (vbr) file, so how difficult could it possibly be to extend this concept to create a roll-forward (maybe a vbf) file which contains blocks to commit and then commit those block to the VBK file at the end, creating the new VBR as you go. This is not a difficult concept, it's how VMware snapshots already work. The different between rolling forward and rolling back is not really significant.

Tom - I am not sure I understand the issue with having multiple jobs running. Have you read this feedback from another customer which has about the same amount of data that needs to be backed up? He is doing this and it works well for him on ESX 3.5.

Yes, I read that. Do you notice how he had to use multiple servers and create 18 total jobs. If he added more machines or his storage grows he'll likely need even more and he'll have to manually create and manage even more jobs. We just migrated from vRanger to Veeam and currently have about 20 jobs scheduled to start every 5 minutes or so. If I add a new host I'll probably need add another job. Do you know how many jobs I had on vRanger? I had 2 jobs total, that's it. Each job backed up a Folder called "Image Backups" on a different ESX datacenter. It had a set for the maximum simultaneous jobs to run and we generally ran 4 jobs simultaneously from each datacenter as that did a pretty good job of loading the network/storage. So those 2 jobs would actually run 8 simultaneous backup tasks and always keep 8 running, when one machine would finish it would move on to the next machine in the folder. If we created a new machine that we wanted to get image backups we simply made sure it was in that folder and it would be run simultaneously with the other backups without us even touching the backup server.

I know that Veeam does support backing a VirtualCenter folder, but a backup of any given folder is run sequentially and that just doesn't provide the throughput to get our backups done within the required window. It's the one thing where vRanger was actually much better than Veeam. We can work with the current scheduler to get the behavior we want, but it's quite a bit more administrative load than simply creating the jobs and letting the backup system manage the load.

[Gostev]

Hi Tom, this interesting discussion - thank you for your time.

I've found that after a crash backup there's probably a 10-15% chance that the .vbk file will simply not be recoverable, in our testing of "hard crash" scenarios we've had 4 vbk files get into this unrecoverable state. True disaster scenarios can come in all types, a friend of had their IBM Shark storage literally catch fire and pretty much burn up before fire suppression managed to put out the fire. The storage array and data were pretty much lost but he had other arrays on which to restore his data. Now let's say he was using Veeam backup and this happend while the backups were running. Would he be able to restore that data? I'm sure some would, but based on my testing it's likely that at least one .vbk file would be corrupt.

Have you been testing this with the latest release? We have addressed couple of scenarios which could lead to this when backup server crashes during backup in 3.0.1 and 3.1. I am not saying that the chance is 0% now - every crash is different in nature because crash moment is random, but this chance should be very low now. Also, keep in mind that in case when UI recovery errors out, we should still be able to do this with our support tool. Restore with UI may error due to corrupt backup meta data (that has much higher chance to get corrupted during crash because its update is not transactional currently), while all actual backup data can still be recovered.

Recovering a damaged .vbk file can take quite a while. One of our host is 520GB, when we aborted the backup and tested a restore it took almost an hour to "roll back" the damaged file and perform a restore.

At this time, this seems to be pretty fair time to process 520GB, because the process involves random disk reads/writes. But we are making adding some significant enhancements to backup storage in 4.0, so all operations with backup files should become faster.

Um, I didn't say to write a new VBK I said to write the "changed blocks" to a separate file and then commit those into the VBK file at the end. You already have the ability to take a vbk file and apply a roll-back (vbr) file, so how difficult could it possibly be to extend this concept to create a roll-forward (maybe a vbf) file which contains blocks to commit and then commit those block to the VBK file at the end, creating the new VBR as you go. This is not a difficult concept, it's how VMware snapshots already work. The different between rolling forward and rolling back is not really significant.

Yes, Veeam Backup 4.0 enhanced storage has similar concepts, although we are not implementing these enhancements for the same reason alone. We've taken a complex approach because backup storage seems to affect just about anything: full and incremental backup performance, repair time, crash protection, integration with 3rd party storage device (aka DataDomain CIFS integration issue), etc. And many of these requirements are typically conflicting with each other (speed vs. reliability for instance). So now that we have collected all the intelligence, we have designed the "best of all worlds" implementation.

Indeed, it was actually a big problem for us to design complete VBK protection from backup server crashes without affecting backup performance heavily (frankly, I thought it would be impossible). And my biggest concern was that if product evaluator does not see good backup performance - they just go away and we lose the customer - he or she simply would not get to the next step, evaluating our awesome backup reliability. Anyway, our devs seem to have made the impossible... yet again. By the way, it would be awesome if you could find some time to participate in closed beta when it is available and help us verify that we got it all covered, since from what you said above you clearly have great experience with VBK crash-tests

We just migrated from vRanger to Veeam and currently have about 20 jobs scheduled to start every 5 minutes or so.

Tom, could you please clarify why do you need this many jobs. Based on feedback, I was under impression that in case of VCB SAN backup with FC4 SAN, backup storage speed is always a bottleneck. Meaning that it does not really help to run more than 1-2 jobs hitting the same backup storage. And when you want to help target backup storage and use High compression, even 1 job fully saturates Veeam Backup server CPU even on quad core CPU, so multiple jobs do not help again. Which one is your scenario, and what is performance differences for running 2 jobs versus multiple jobs?

I just got Google alert for new blog post from Steve Philip, our community member (sphilp). By the way, Steve - great post, I am adding it to the "Feedback from the Field" thread - will ask devs to investigate CIFS issue too). Anyway, Steve is also concluding that it is backup storage which defines how fast your VCB SAN backups can go. Here is the link to the blog post: http://sphilp.blogspot.com/2009/06/vmwa ... up-31.html

[tsightler]

Hi Tom, this interesting discussion - thank you for your time.

No problem. I want to make it clear that we love Veeam, we think it is a great backup tool and is so much better than our previous solution and we understand that every environment is different, but this was a "Wishlist" thread so I threw ours out there.

Have you been testing this with the latest release? We have addressed couple of scenarios which could lead to this when backup server crashes during backup in 3.0.1 and 3.1. I am not saying that the chance is 0% now - every crash is different in nature because crash moment is random, but this chance should be very low now. Also, keep in mind that in case when UI recovery errors out, we should still be able to do this with our support tool. Restore with UI may error due to corrupt backup meta data (that has much higher chance to get corrupted during crash because its update is not transactional currently), while all actual backup data can still be recovered.

We're testing with 3.1 that we downloaded about two weeks ago. I'm pretty sure that's the latest. Our testing basically involves simple test like unplugging the network cable of the backup server, rebooting the server during a backup, etc.

At this time, this seems to be pretty fair time to process 520GB, because the process involves random disk reads/writes. But we are making adding some significant enhancements to backup storage in 4.0, so all operations with backup files should become faster.

I didn't say that this was slow, but it would be instant if the changes were backed up to a separate file first. Actually, we have other reasons we wish the VBK didn't change on every backup. We have almost 3TB of vbk files at the end of our backups, these must all then be spooled to tape every night. It would be much nicer if the VBK file only changed once a week, writing change files and then consolidating the changes once a week would eliminate that problem.

Yes, Veeam Backup 4.0 enhanced storage has similar concepts, although we are not implementing these enhancements for the same reason alone. We've taken a complex approach because backup storage seems to affect just about anything: full and incremental backup performance, repair time, crash protection, integration with 3rd party storage device (aka DataDomain CIFS integration issue), etc. And many of these requirements are typically conflicting with each other (speed vs. reliability for instance). So now that we have collected all the intelligence, we have designed the "best of all worlds" implementation.

By the way, it would be awesome if you could find some time to participate in closed beta when it is available and help us verify that we got it all covered, since from what you said above you clearly have great experience with VBK crash-tests

Veeam 4.0 sounds great, I can hardly wait. We'd love to test the beta when you get to that point. I'll be honest, one of the reasons we switched to Veeam Backup was because of the strives you've made in the product in the last year. We looked at the product at the 1.0 version and it just wasn't able to meet our requirements, then after living a year with vRanger and not being very happy we took another look a few weeks ago and couldn't believe how far the product had progressed, still not perfect, but unbelievably improved. We expect to see the same great things from 4.0.

Tom, could you please clarify why do you need this many jobs. Based on feedback, I was under impression that in case of VCB SAN backup with FC4 SAN, backup storage speed is always a bottleneck. Meaning that it does not really help to run more than 1-2 jobs hitting the same backup storage. And when you want to help target backup storage and use High compression, even 1 job fully saturates Veeam Backup server CPU even on quad core CPU, so multiple jobs do not help again. Which one is your scenario, and what is performance differences for running 2 jobs versus multiple jobs?

OK, I'll try to describe our environment. We an all iSCSI shop, no FC. We use Dell/Equallogic storage arrays and currently have 3 of them, they can easily run at 180MB/sec each for a total of over 540MB/sec. Our backup server has 4iSCSI HBA's and thus tops out at a little over 480MB/sec. For some reason though VM backups seem to be limited to 35-40MB/sec each, this happens no matter what mode we use, network backups, VCB SAN, etc. I'm not really sure why, but my suspicion is a fairly low queue depth combined with the added latency of iSCSI works against it's top speed. Anyway, the only way we've been able to get enough speed is to run more jobs. We've found that 4 jobs per array get's pretty close to maxing the capacity while leaving a little headroom for normal operations and running 10-12 jobs gives us 350-450MB/sec total performance. We use a mix of network backups and VCB SAN since VCB backups seem to be limited to 8 simultaneous jobs and network backups work well for offloading some of the read load.

We also have a lot of jobs because most of our VM's are not very similar and many are quite large (500GB+). We use many jobs to keep the VM count down and the backup sizes reasonable. We also backup production instances to individual backups because we spool them to tape and we need the fastest possible restore time. We're not currently using best compression, we just don't have enough hardware to run that many jobs with best compression although it seems to mainly be only on the initial full.

Overall we're pretty happy with the current schedule the only problem is that we have to manually spread out the jobs to try to balance keeping 10-12 running, vRanger did this automatically. Not a real big deal, just a little extra admin work.

[Gostev]

Our backup server has 4iSCSI HBA's and thus tops out at a little over 480MB/sec. For some reason though VM backups seem to be limited to 35-40MB/sec each, this happens no matter what mode we use, network backups, VCB SAN, etc. I'm not really sure why, but my suspicion is a fairly low queue depth combined with the added latency of iSCSI works against it's top speed.

I believe the reason for this is that VCB does not like multipathing, which you probably have set up since you have 4 HBA's. Many of our customers have reported that their backup speed going up a few times from 30-40MB/s after they disable multipathing.

[tsightler]

I believe the reason for this is that VCB does not like multipathing, which you probably have set up since you have 4 HBA's. Many of our customers have reported that their backup speed going up a few times from 30-40MB/s after they disable multipathing.

I read those reports, and we've tried both completely without multipath (we thought one link per array would be OK) and with different multipath modes (failover, least queue depth, and round robin) and in our case this has absolutely zero impact on the backup performance. We also see the same speed with "direct to target" mode using the network backup mode, both agent and agentless. While I have no proof, I still think it's the latency/queue depth/minimal read-ahead combination that works against iSCSI configurations. With some tweaking we've managed to get to around 50MB/sec per backup using VCB SAN on incrementals so running 6-8 jobs (2-3 per array/link) should get really close to saturating our links/SAN.

We're also not a big fan of backing up many VM's to a single VBK. If we do have a damaged VBK we loose all of the backups in it and you can't restore from a VBK while a backup is running. I know you stated that your support should be able to recover data from a damaged VBK but having to contact support to restore data is not high on our wish list.

I should note that most of our VM's contain lots of data so we don't get the performance advantage of VM's with a lot of zero'd space. We do have a few that are pretty empty and those VM's show speeds as high as 90MB/sec.

[lohelle]

I would like to see an option to create one vbk per VM actually. In my small 50 VM environment this makes me more happy. It is much easier to move those files off to other storage later on then.

[Gostev]

Pause/unpause scheduled runs (internally, we call this functionality "temporary disable job") is actually in the list of low priority features for 4.0. Meaning, we are planning to add this feature if time allows to do it without slipping the release. So chances are good! Thank you.

Just wanted to update that this feature has made it into Veeam Backup 4.0...

[Gostev]

I would like to see an option to create one vbk per VM actually. In my small 50 VM environment this makes me more happy. It is much easier to move those files off to other storage later on then.

Lars, I think I missed this post. Would it be OK if instead of backing up to individual files Veeam Backup still backs up all the data to single VBK, but then provides option automatic action to extract all or only required VMs to the designated share or folder? This is more realistic functionality which would not require completely revamping the product's architecture.

[dlove]

I'd personally like to be able to backup via VCB (SAN) and then on failure be able to retry using LAN. Every night I have odd ball servers that fail when using VCB and then work fine when doing network backups. It's not always the same servers but it's annoying to go in and change the job temporarily to network then retry the job which finishes successfully and then change back the job to VCB. Vizioncore had this feature on the 3.x vranger that we gladly left behind us

[Gostev]

Yes, we have this feature in vStorage API backup mode of Veeam Backup 4.0.

[withanh]

Sounds like Veeam v4 has a lot of nice new features that I would like to use as well.

What is the expected release time frame for v4?

[Gostev]

We are looking to RTM the code in the end of Q3.

[lohelle]

I would like the possibility to put the diff-files on a separate location. This would be nice when the target storage is often almost (and completely) full.

[tsightler]

So, after using Veeam for a few months I've come up with a few more things to add to the "Wishlist":

1. A fully functional Linux file level restore. The current solution is OK, but has a lot of drawbacks. It crashes when attempting to restore files with names that include characters unsupported in Windows. It looses all file attributes, including ctime, atime, etc.

2. A command line utility to mount a VBK/VBR backup on linux directly. This would pretty much solve the linux file level restore problem for me since you could simply mount the VBK/VBR file on the linux server and copy the files with native tools.

3. A faster file level restore when restoring a moderate to large number of files. We recently needed to restore a directory that contained ~35,000 files but the total size was only about 10GB. This took about 90 minutes with Veeam. Since this system was also still being backed up with our legacy tape backup solution we decided to compare how long the same restore would have taken from tape via this method and it took only 40 minutes.

4. A really nice feature would be a tool that could index the files within the images similar to legacy file based systems. I understand that this might be unrealistic for an image based backup solution, but I could envision a feature where, after the backup was complete, the backup images were mounted and indexed to allow files to be searchable, offering the best of image and file level backups. This is really important for sites like us where we may need to restore a file from a year ago. With our current solution we can find the file and it will tell us to go get the archive tape from 10 months ago (the most recent file that contains the tape). Even now that we are archiving Veeam backups we no longer have an index. If someone needed a file from a year ago we'd have to grab multiple tapes and start restoring files looking for the vbk file that might have the file we needed.

I understand that #4 might not be reasonable, but it is a wishlist afterall.