New Install : Best Setup ?

[andybh]

Hi,

Although we've had a licence for Veeam for a while now we are only now looking seriously at putting it live. I have a number of options available to me and am trying to work out the best combination. We have Equallogic iSCSI SAN devices as the primary SAN holding VM files on VMFS (ESX 3.5) - Four ESX Hosts. The intended store for local Backups is a 24TB OpenFiler box. These backups are intended to replicate to a matching OpenFiler box (using its own replication functionality). I hope to backup every VM every day (hopefully without too much impact on production performance - we have no "backup window"). I have a spare Win2003 physical server which I can use to run VEEAM.

Questions......

I can carve the Backup storage as either an SMB share, a Windows formatted iSCSI volume presented to the Veeam server, or a VMFS formatted datastore presented to the ESX hosts. Which is the ideal ? If one of the iSCSI options would performance be better if the Veeam server had a QLogic iSCSI HBA fitted (I have one 'spare')

Is VCB mode likely to be quicker / less intrusive ?

"backup" is faster than "replication" ? The fact that backup allows compression is not a huge issue to me - my backup storage has more space than my production environment. Our production SAN (equallogic) has volume replication to our DR site and will have automated snapshots hourly to allow quick rollback if needed. What I'm after is a long term backup that we can eventually take off to tape and hold in archive in case ever needed.

Most important factor to me is the speed of the incremental backup - I don't mind the intial backup taking hours but once up & running I ideally want even our Exchange server (500GB ish) to be backed up every night.

If anyone can give me some very basic pointers then I can make a start and fill in more details as I go along !

My first attempts at installing Veeam a week or two ago (network backup/replication to VMFS store) worked OK but I could not see the second or third pass as being any faster than the initial job, even on a machine with very little data change.


Thanks !

[tsightler]

With ESX3 you won't see the incremental passes be much faster than the first pass no matter what you do (I suppose a mostly empty VM with very little changes might be faster). That's because Veeam has to read the entire VMDK file each and every time to determine what blocks have changed. The incrementals might be a little faster simply because less data has to be written to the backup volume, but assuming your target is fast enough to handle the load, incrementals are pretty much the same as fulls from a backup perspective.

We are also an Equallogic iSCSI shop. We started off using Qlogic iSCSI HBA's along with VCB, but we found that we simply weren't able to push the full bandwidth of our array with this method. We had to run 6-8 VCB jobs simultaneously and, to be honest, VCB is a little flaky once you get multiple jobs running in parallel. With ESX3 the only thing we could do to get all of our backups completed at night was to use Veeams network backup mode with optimal compression. This only works with full ESX (not ESXi) but pushes a small Veeam Agent to each ESX host. We found that we could easily run 10-12 backups in parallel and really max out our storage array bandwidth.

Unfortunately, with Veeam 3 the Exchange backups incrementals were always very slow. I believe this is because Exchange involves so many small changes across the entire disk, increasing overhead on the target disk image. Veeam 4 seems to handle this case much better, with incrementals running basically at the same speed as fulls. We were never able to get more than about 40MB/sec per each backup (except for VM's with lots of unused space), but running 10-12 jobs in parallel got us to 400-450MB/sec, pretty much topping out our three EQL arrays, and we could just barely get our backups done overnight (we really don't have a "window" either, being a manufacturer, but most systems are at least a little quieter at night).

Since you're using OpenFiler, I'd suggest not using iSCSI or CIFS file sharing, assuming the hardware is decent, I'd simply add the box as a Linux target and use the network Agent on the ESX console. In that scenario you'll get direct to target backups (lowest network overhead, especially for incrementals) and you want need to setup and shares. The Veeam server at that point is basically just a console and job queuing interface. I'd highly suggest using the just released Veeam 4, as it includes several fixes that significantly improve this setup, specifically the speed of file level restores, and the performance of Exchange backups.

Of course the "Holy Grail" would be to move to ESX4 in combination with Veeam 4 using vStorage API and the OpenFiler as a Linux target. That's what I'm setting up right now (well, not using OpenFiler, just a RHEL5 box, but same concept). With the vStorage/ESX4/Veeam4 combo you get "block change tracking" which means that Veeam no longer has to read the entire VMDK file to determine what blocks have changed, VMware will simply tell it what blocks have changed and it will only read those blocks. For fairly static systems like application servers, the wins are amazing. Many of our servers back up in 2-3 minutes using this setup, and most of that time is take and removing the snapshot. The servers with more changes are still slower, our ~350GB Exchange server takes about 2 hours, but that's infinitely faster than the 12-14 hours it took with ESX3/Veeam3. We expect that we can get all of our backups done in 2-3 hours a night, with much less of a performance impact than we had previously since we only need to run 3-4 jobs in parallel (and that's mostly to separate Windows and Linux jobs -- not a requirement, just a desire -- and to backup two different sites).

Good luck!!
Tom

[andybh]

Tom,

An excellent and very useful reply - thanks

Summary I'm taking from this is that until I move to ESX4 I may as well keep it as simple as possible.

If Veeam server is "just a console job and queuing interface" - i.e. no local storage, no iSCSI connections does it follow that the VEEAM server might just as well be a VM itself ?

As I am going to start from scratch over the next few days I will install Veeam4 and see how I get on. To be honest I could probably live with a full backup of my Exchange server taking 10 hours or more as long as (a) its only once a week as EQL snapshots/replicas will suffice as first level rollback and (b) the Exchange server stays available to the users during this time.

As with any solution I guess theres a certain amount of "suck it and see" and each users mileage will differ depending on their own environment.

What your reply has confirmed is that (with our current setup - including ESX 3.5) the ideal goal of being able to run incremental backups in minutes rather than hours is not achievable so I can give up chasing it !

Until recently we were still running BackupExec accross all VMs which apart from being a huge expense in licensing had a high impact on useability. We had got to a point where backups were still running when users wanted to work and the whole LAN performance was crippled. I really want to get to a point where I can be confident we don't need to run any of these "traditional" backup methods anymore. My hand was slightly forced on making that decision when our last tape autoloaded packed up.

Currently we are not backing up to tape at all - but I'd hardly say I feel comfortable/confident !

Thanks,

Andy

[tsightler]

If Veeam server is "just a console job and queuing interface" - i.e. no local storage, no iSCSI connections does it follow that the VEEAM server might just as well be a VM itself ?

The only real disadvantage to running Veeam in a VM that I've found is the fact the the File Level Restore appliance will not run in a VM since it is itself a VM. This isn't a problem if you're only backing up Windows systems since the Veeam Backup can restore Windows files natively straight from the management GUI, but if you're backing up in Linux systems, or others systems that require the File Level Restore appliance, then this might be a concern. There's actually some unsupported hacks that allow the VM to run inside a VM, and it does work, but it's pretty slow.

I should have probably also pointed out one possible negative to using the OpenFiler as a direct Linux target. When using the "Network Backup" mode you should pretty much only use Optimal compression (or, with Veeam 4, "Low" compression). Since the agents run in the ESX console there's not a lot of CPU horsepower. The Optimal (Veeam 3)/Low (Veeam 4) modes are designed to provide very low overhead for running on the ESX host itself with minimal impact. It appears to mainly just do simple dedupe and empty space compression. Like us you have plenty of backup space so this is probably no big deal for you.

What your reply has confirmed is that (with our current setup - including ESX 3.5) the ideal goal of being able to run incremental backups in minutes rather than hours is not achievable so I can give up chasing it !

Yep, simply won't happen with Veeam and ESX 3.x, but once you get to ESX 4, well, things will be looking up. We actually rushed our ESX 4 upgrade just because of this. Heck, it was practically the only reason we upgraded to ESX 4 as none of the other features were really that compelling in our environment (nice, but no real need).

Currently we are not backing up to tape at all - but I'd hardly say I feel comfortable/confident !

We still spool to tape at least once a week. We're fortunate enough to have a site about 10 miles away that's connected via a 1Gb Fiber connection so we actually have our Linux backup target at that site. Then we spool the Veeam backups back to a tape drive here in our data center and rotate them offsite. That gives us offsite disk backups, and offsite tape backups. Once Veeam implements true incremental backups, which will make nightly replication of backups to secondary disks and tape much easier, we may well be in backup nirvana.

BTW, if you use Veeam 4 I doubt your Exchange backups will take 10 hours. That happened with Veeam 3, but as we were testing Veeam 4 beta the nightly incrementals were only about 2 hours even when using ESX 3.5.

[Gostev]

Thread stickied, Tom - thank you very much for taking time to write your recommendations.

If Veeam server is "just a console job and queuing interface" - i.e. no local storage, no iSCSI connections does it follow that the VEEAM server might just as well be a VM itself ?

Just wanted to stress here that Tom is giving you recommendation to use ESX service console agent based "Network" backup mode, which is indeed fastest way to backup ESX 3.5 hosts with local storage. In this mode, when backing up to Linux or another ESX host, backups are "direct-to-target", and actual Veeam Backup server is not even involved. Temporary backup agents are automatically deployed and started on source ESX and target host, and communicate directly to each other. You even get network traffic compression with this mode, unlike vStorage API or VCB network modes. Traffic gets compressed with "Low" compression algorithm, and if the job is configured to use Optimal/Best, actual additional compression gets offloaded to the target host - so again Veeam Server is not involved anyhow, other than controlling and reporting on job execution.

The main drawback of "Network" mode is that backup puts load on your production ESX host (since the service console agent runs on it). Unlike in case of vStorage/VCB "SAN" modes, where all backup activities are completely offloaded to the storage layer and backup activities do not load or even touch production ESX - which is a good thing for those more heavily loaded 24/7 ESX hosts - which is why this is VMware-recommended approach.

Additionally, you need to be careful pointing multiple jobs to the same target and running them all in parallel. This may cause uncontrolled target saturation (this results in random write errors and may reduce backup success ratio significantly).

The latter is something that will never happen with "proxy" approach used in case of vStorage API and VCB backup modes. Now, Veeam Backup server is no longer just console and UI. In these modes, all data goes through the Veeam Backup server, which then writes data to target storage. So unless you are using "Low" compression, you have to make sure it has powerful CPU (modern, multi-core).

The same stands true when backing up to CIFS share (in any backup mode). Since we cannot deploy agent on share, Veeam Backup does all the processing and then just writes backup data there. So using v4 Optimal/Best compression will actually increases backup performance in many cases when backing up to network targets (instead of locally attached), because much less data will need to travel over the network. And you will get smaller backups, too. So this is something which needs to be considered.

You can install Veeam Backup in a VM in any mode, however again, depending on compression level you will want to give the Veeam Backup VM 4 vCPUs.

If you want to go with virtual Veeam Backup server, but still get direct storage access and no agents on ESX hosts, then Veeam Backup 4.0 features another cool new mode for this, so-called "Virtual Appliance" mode. This mode is being discussed in the following thread: vStorage API in Virtual Applicance mode.