SureBackup : General Discussion

[rowdy]

No, we won't. I believe Q3 and I already opted in for the beta

[lohelle]

I hope direct-to-san restore is one of the new features. Restores at 3-400 MB/s would be nice!

[Gostev]

Does that mean that we don't get v5 on Monday?

It's been only 3 months since version 4.1 release... we are not THAT good

[withanh]

LoL sure you are!

[Gostev]

So now after you've seen the announcement and what the technology we are introducing allows to do (essentially, bring up any backup to life in a few minutes), my answers to "dream-only" comments about item-level restoration on the previous page of this topic should make much more sense for everyone.

[donikatz]

Will the webinar on Wed offer more info than the online video? Exciting stuff, can't wait to hear more!

[Gostev]

Yes, there will be much more information than in video, plus I will run Q&A chat. You can already pickup a lot of additional information from datasheet, technical blogs (for example here, here, here or here - we had a pre-briefing with some bloggers), plus there are bunch of other articles like this. I am sure there will be tons more to come - just search for #veeam or #surebackup on Twitter.

But you really don't have to wait for Wednesday - you can start asking right now (if you have specific questions), and I will answer here.

[Gostev]

Nice coverage from virtualization.info (this is the biggest review and analysis so far)
http://www.virtualization.info/2010/03/ ... ogies.html

Another big review from TechTarget:
http://searchdatabackup.techtarget.com/ ... 01,00.html

Comprehensive podcast about SureBackup
http://www.dabcc.com/media.aspx?id=936

[rowdy]

It's certainly very nice to always know your backups are 100% okay.

And it seems like I can just run VM's from the backup? The backup can be on any regular NAS then that's connected to an ESXi host? Because our backups are just being stored on normal NAS'ses with SMB and FTP access to it. They cannot be mounted as iSCSI devices in ESXi, naturally.
So how can ESXi run these virtual machines then?
Or do I need to store the backups on another ESXi server? VeeAm does provide the option to store on UNC SMB paths.

Moreover, if this is possible, can we also start a virtual machine from the backup and recover it in the background? It would be nice to be operational again in 2 minutes (at reduced performance) and have the restore run in the background

[donikatz]

Ok, I'll bite --

1) I guess my #1 question is how will Veeam configure the verification sandbox? (I get that it uses NFS to present to ESX.) Maybe the most-common thing folks will want to verify is AD + Exchange. How will Veam know dependencies, priorities, delay timings: built-in presets, or users will configure this (e.g. testing DCs are running & replicating before testing Exchange)? A specific example of ours would be an RHEL Oracle db server VM + RHEL WebLogic app server VM + RHEL Apache web server VM making up the complete web stack. We would configure dependencies and other options?

2) What would be a rough expected procedure for Exchange brick-level restores? If it's a sandboxed environment, how would we recover a specific message from backup mailbox to the user? Would there be an additional connector to the live instance somehow, or would we still have to manually copy from the backup sandbox to the live server? (One of the big advantages of an agent like Backup Exec is being able to point brick level restores directly to the live running instance).

Thanks!

PS -- rowdy's last question is an excellent one too!

[Gostev]

It's certainly very nice to always know your backups are 100% okay.

We think so to

And it seems like I can just run VM's from the backup? The backup can be on any regular NAS then that's connected to an ESXi host? Because our backups are just being stored on normal NAS'ses with SMB and FTP access to it. They cannot be mounted as iSCSI devices in ESXi, naturally.

It does not matter where backup is stored. It can be on any storage - local or remote to Veeam Backup server - DAS or SAN LUN, NAS w/CIFS or remote Linux box, DataDomain box or alike, and of course ESX w/VMFS. In othere words, anything Veeam Backup supports today to backup to.

So how can ESXi run these virtual machines then?

We will run NFS server on Veeam Backup server that will get the requested VM data blocks from whatever storage the corresponding backup file sits on. The NFS datastore on Veeam Backup server is what will be mounted to the ESX host you will select for the SureBackup lab.

Or do I need to store the backups on another ESXi server? VeeAm does provide the option to store on UNC SMB paths.

No need to change anything you are doing today to start leveraging this functionality. Not even location of your backup files. You will be able to verify your v4 backups first thing after upgrading to v5 - without making any changes to your backup infrastructure.

Moreover, if this is possible, can we also start a virtual machine from the backup and recover it in the background? It would be nice to be operational again in 2 minutes (at reduced performance) and have the restore run in the background

Sure. I don't see why you could not SVMotion the test VM. This is on my to do list to test before the release
Really the technology behind SureBackup provides truly endless possibilities, I am sure there are much more use cases then we are thinking about today.

In all cases, besides SVMotion, our replication, hot VM copy and file copy jobs will definitely be supported with SureBackup NFS to facilitate such instant VM recovery. This is essential because SVMotion is not available with some VMware licenses anyway.

[Gostev]

Doni, I think I have just answered your questions in the dedicated thread:
SureBackup : Granular Item Recovery

[donikatz]

Doni, I think I have just answered your questions in the dedicated thread:
SureBackup : Granular Item Recovery

Thanks, I'll follow up there.

[avlieshout]

A few more questions about Veeam SureBackup that came accross my mind.

#1: The solution provides a NFS server on the Veeam backup server to present the backup files to an ESX server. How will that impact the load on your backup server when all those VM's disk IO's are handled by the backup server?

#2: Can you still run backups and backup verification or restores at the same time regarding backup server load?

#3: As the NFS will be readonly, there must be differences files stored somewhere on your ESX box. Can you select a specific datastore for these differences files or is it selected randomly?

#4: I also have a question regarding the use of application group or vApps. We have several application groups with dependent servers on different vlans. for instance the AD server is on a different vlan than the application server or database server. How will these servers be able to communicate to eachother without the ip subnets on the vlans being routed?

#5: Are we able to pre-define the sandbox networking environment and connect it using uplinks to external switches so you are able to use more ESX hosts for backup verification or restore purposes?

Thanks!

Arnim

[avlieshout]

Oh and another one.

#6: will it be compatible with VMware vShield zones?

[Gostev]

Hello Arnim,

#1: The impact is of course negative, however you should not (and cannot) run verification while backup job is running. You will run this after backup job completes. But more generally speaking, I expect that verification will be done outside backup windows by the most customers. Since it only puts load on backup storage and dedicated resource pool selected for SureBackup lab, you can easily run it during production hours. And if you will use DR site or test lab hardware for verification jobs, then it is not a concern at all.

In fact, if you have DR site I recommend to consider syncing backup files to DR site and running verification there, to both include DR site's hardware in the testing loop (ESX, network, differences storage), and make sure backup files did not get corrupted during the sync process.

#2: You cannot run verification job on the backup file being processed by backup job. Other than that, you can run backup and verification at the same time (for different backups), but again there is little sense (see #1).

#3: Correct, you specify that as a part of defining SureBackup lab (which is one time process, unless you need multiple labs).

#4: We will auto-configure basic SureBackup vLab with multiple isolated networks. Theoretically, this basic configuration can be manually extended to include external routers, switches etc. In practices - we will need to test it down the road to have the answer for "sure" before the release

#5: We will not support complex SureBackup labs like that (which span multiple ESX) in v5. We don't believe that it is critical functionality to have in the very first release of this functionality, given that you can have multiple SureBackup labs and load-balance the recovery verification and universal item-level restore activities between them easily.

#6: I don't see any reason why it won't be compatible. Test VMs are regular VMs, and test VLANs are regular VLANs.

[rowdy]

Moreover, if this is possible, can we also start a virtual machine from the backup and recover it in the background? It would be nice to be operational again in 2 minutes (at reduced performance) and have the restore run in the background

Sure. I don't see why you could not SVMotion the test VM. This is on my to do list to test before the release
Really the technology behind SureBackup provides truly endless possibilities, I am sure there are much more use cases then we are thinking about today.

I just limited the quote, but thanks for all your extensive answers, even to questions that must sound stupid to you. It's just that I'm really a Windows sysadmin with some (very) basic linux knowledge and I'm learning more and more about VMWare. I just managed to install and enable SSH and FTP on my ESXi box (I know that's not recommended and why it's not) and I must say FTP speeds to ESXi are just great compared to the horrible speeds of the DataStore Browser and FastSCP. But that's another topic, just wanted to put things in perspective and thank you for answering my 'dumb' questions

But more to the point:
I don't think VMotion is possible: when running of the backup, aren't all changes discarded afterwards? If we have Exchange online and running again in 2 minutes, users will start receiving mail and that shouldn't be lost. Furthermore, we need to be able to select to do a 'live restore' to a server we choose and with a working network configured and everything. That might need an additional 'perform live restore' option in the interface somewhere?
And then there's always the 'problem' vMotion is only supported in the most expensive versions of VMware so a different kind of solution would be welcome, but it would be understandable if it's not possible, of course. It might be just asking too much for too little money, I guess.

[Gostev]

We are actually talking about SVMotion (Storage VMotion), not regular VMotion. These are different things.

But again, we still have plenty of time to test everything before the release - both ourselves and with help of customers during closed beta. I really don't want to guess and possibly set wrong expectations on any application of this technology that involves 3rd party functionality - even if it is VMware. I would like us to test everything first.

[Gostev]

By the way, what other application do you see for "on demand production replica" this technology provides? Do you see yourself using this for patch testing, investigating potential fixes to current problems in production, or something like this?

I myself think it's pretty cool that you can spin-off your Exchange server from the latest backup and run ESEUTIL and ISINTEG on test VM - rather than on production VM. Now if these tools shows no issues on test VM, then you do not need to perform the same test on your production environment and affect it. And if they do show issue - you can try to fix them on test VM and see if this helps - before doing this on production environment.

Previously, it would take you hours to create such lab - and obviously you would rarely bother to do this, and instead in most cases you would skip the lab part and do this on production host directly. But now, when you can have it running in less than 5 mins - it may change everything!

[Gostev]

Comprehensive podcast about SureBackup
http://www.dabcc.com/media.aspx?id=936

Good questions, good answers.
Highly recommended.

[jeremystw]

Rather then running these backup files just in an isolated environment to retrieve files etc..., can you run them on your network. Even if it is a degraded performance because of slower storage etc.. it still may make sense for a cheep DR scenario. Also I would love to see some physical to virtual component backup component included.

[Alan_Eden]

For example, what other application do you see for "on demand production replica" this technology provides? Do you see yourself using this for patch testing, investigating potential fixes to current problems in production, or something like this?
...
Previously, it would take you hours to create such lab - and obviously you would rarely bother to do this, and instead in most cases you would skip the lab part and do this on production host directly. But now, when you can have it running in less than 5 mins - it may change everything!

Hi Anton, this stuff seems pretty cool.

Our main use case at the moment is "on demand replication" to produce reliable Dev environments, with pressure to produce these environments as quickly as possible. Deploying Veeam has already cut the roll out time by 50%, and this looks like it will reduce it further.

Can I run the following scenarion past you to make sure I understand the possibilities correctly:

1) Locate the backup of the environment (can be split across multiple Veeam backup jobs).
2) Make the backed up environment visible in the ESX cluster and reconfigure the VMs as required (change VLAN settings etc)
3) Boot up the environment and handover to the developers
4) Use Storage vMotion to migrate environment onto dev cluster.

Have I understood correctly? Will this be possible with SureBackup?

Thanks,
Alan

[Gostev]

Alan, yes your understand is correct, this will be possible with SureBackup. We have not actually tested Storage vMotion part (4) yet, but I believe this should work. Even if this does not work for some reason, we will be able to provide working alternative down the road to enable your scenario.

[tsightler]

OK, I'm a little confused. Based on some comments above the "run from your Backup" option brings the systems up on an isolated network, not the original network. For testing backups, this is perfect. However, there are also comments that imply that, in the event of a disaster, you could simply power on you VM's and run from the backups, then potentially SVmotion the systems back but in this case the VM's would need to be on the production network to be of any use. The "backup verification" is interesting, but it's the "run from backups" that I would think is groundbreaking.

Just this past week we had a disaster in our datacenter. Basically, we had a simple drive failure on a RAID storage array, which should have been no big deal, however, from what we currently believe is a firmware bug in the hardware, the while the array was in degraded mode, it was silently corruption blocks of various storage, slowly destroying the underlying volumes. By the time we managed to get the drive swapped it had seriously corrupted almost 6 TB of volumes, almost 4TB of VM's. We had to restore all of these VM's from Veeam backup which took almost 24 hours.

If we had been able to power our VM's on from the "backups", even this major outage would have been almost completely unnoticed by the user community. As it was, it took almost 4 hours before the first VM's were restored, and while we were able to get the critical systems restored prior to the full business day, it was still quite an interruption. Being able to run from your production system from the backups while you restore your infrastructure would be truly innovative.

[Gostev]

Hello Tom, I have guessed that something might have happened because I have not seen you much on the forum lately... glad to hear you are fully recovered, 4TB of restored VMs is quite impressive I must say, sounds like large scale outage indeed.

For the SVMotion scenario you would need to tell our engine not to reconfigure VM networks to isolated, and instead keep current networks intact. This is actually the very first technology use case we are describing in our patent application. The only reason we go with "backup verification" application of this technology first, is because we would like to polish the "run from backup" engine including our NFS server, and see how well will this work in real-world environments.

For backup verification scenario, our NFS server's performance obviously does not matter too much (just a few VMs are run in parallel, and no users accessing those). However, for instant "restore" scenario, we are talking about running production workload (with hundreds of users accessing it), and a lot more servers to run in case of disaster like you have just experienced. Even if you are only going to use this capability for the most critical VMs, this would still require really well polished engine, and good understanding of scalability per our NFS server in case of production workload... not something I would expect we will have in "1.0" release of this technology. Anyhow, as our MVP, you will have a chance to put your hands on this technology among the very first, and experiment with this as much as you want.

So on another note, did we manage to impress you with this stuff, or do you see us among those

Products rarely seem to live up to such hyped-up annoncement

[Gostev]

By the way, as you may have already guessed my response to your "killer feature" request will actually work in v5. I know you wanted something nicer then this command line... but please appreciate the fact that this feature was implemented even before you have requested it

[tsightler]

I really just haven't had much time lately, that's the only reason I haven't been posting much. I have been keeping up with the forums though.

The outage this week was the single worst outage I've experienced with systems that I personally admin in my 20+ years of managing servers/networks. I have experienced worse outages for clients that called me in after a disaster (one client experienced a major flood in the datacenter, and another had a small fire, including a melted UPS that blew about a dozen power supplies in the equipment, those were both pretty bad). In this case it appears that the storage array basically corrupted every volume that it hosted. It's a mid-tier iSCSI array from a pretty big name vendor, but I don't want to implicate their technology at this time since we haven't completed our post-mortem, but it's quite obvious something seriously wrong happened with this array as we lost 4TB of VMFS volumes hosting VMDK files, and 2 TB of Linux ext3 filesystems from a physical host. Every one of the volumes had unrecoverable corruption.

I'm sure we would have picked only a few strategic VM's to power on. Many of our most critical VM's were replicated anyway, but this case there were a few application type servers that were not replicated that would have been nice to have up in short order. They do very little IO so the storage performance probably wouldn't have been a major issue.

I don't know if you've impressed me yet or not, I think a lot will depend on the implementation. You could alread get very close to this with Veeam replication, but of course you lost the compression/dedupe. Still, assuming the implementation is strong, and considering the technology you're talking about wrapping around it (like web based portal for restore), if you get all of that working in a reasonable way, then yes, I'd be impressed. In other words, ideas are the easy part, it's the implementation that will impress me or not. That's why I wasn't too happy with the hype, it was the hype of a "technology announcement", not a real product. I've seen many great technology solutions that sound cool but are so poorly implemented they're not usable. So, in short, do screw it up and then you'll impress me.

[Gostev]

OK, sounds fair One thing you can be sure about Veeam is we do not market vaporware - we only start talking about new features once they are actually implemented and hit the testing, and after release date becomes more or less predictable.

[tsightler]

BTW, just want to make it clear that I understand the marketing reasons why companies do the whole "technology announcement" marketing things, I just don't really like it. Veeam does have a pretty good track record so far though, so I'll wait anxiously for the first beta!

[Gostev]

Consolidated SureBackup feedback so far:
SureBackup – 1 week later

Links to all reviews, blogs, articles.