New implementation of VMware B2D2T

[Phorward]

I'm going to suggest to a customer a new implementation of Veeam B&R v7 replacing their current backup solution.
What their current solution is isn't really important, what's important is the physical structure of their environment and how to best implement Veeam.

This is what their infrastructure looks like:
Two ESXi 5.1 servers connected to a SAS shared disk enclosure.
A physical server running Windows Server 2008 x64 with vCenter 5.1.
A Tape Library connected to the Windows Server with SCSI.
A NAS with iSCSI, NFS, CIFS/SMB abilities.

Two physically separate networks, one for production data and one for backup data. The ESXi's and Windows Server have connections to both, the NAS only to the network for backup data.

I have noticed that I had much better performance from the NAS when connecting it to the ESXi's using NFS rather than iSCSI.

The Windows Server, Tape Library and NAS is in a different location from the ESXi's.

This is what I'm thinking of doing: Install Veeam B&R on the physical Windows server. Install a virtual Windows server to act as Veeam Proxy. Create a CIFS share on the NAS to use as backup repository. Create a backup job to use the Proxy with Hot Add function to backup to the NAS. Backup the repository to tape using the physical Windows server preferably as a secondary target in the same job.

Will this work or is there a better solution for this infrastructure?
I want to keep the backup data running on the separate network intended for backups. When doing the backup to tape will the traffic go directly from the NAS through the physical Windows server to tape or will it have to be re-routed through the virtual Veeam Proxy?

After implementing Veeam then I will probably upgrade vSphere to 5.5, will that be a problem for Veeam?
Edit: just read that Veeam is not ready for 5.5 yet but that support is coming in November.

[veremin]

Hi, Hans,

From my perspective, you should (most of these steps you’ve already mentioned):

1) Install VB&R on physical Windows machine that tape library connected to.
2) Assign proxy role to any Windows-based Virtual Machine you have in your environment in order to use it as a Hot-Add proxy.
3) Among three types of connection (ISCSI, CIFS,NFS), CIFS is the least preferable. So, if I were you, I would either connect the said device directly to physical machine via ISCSI or mount it to the Linux box (NFS) and add it later as a backup repository.
4) Create a share disk, map it to your physical instance of VB&R and use as a target repository for configuration backup. Should any disaster situation happen (physical server goes down, for instance), all you would need to do is to install Veeam on any other virtual machine you have and import previously backed up configuration.
5) Don't rush with the upgrade to vSphere 5.5, since it’s not supported by VB&R yet.

Thanks.

[Phorward]

Hi Vladimir,

Thanks for your reply and suggestions.

I would either connect the said device directly to physical machine via ISCSI or mount it to the Linux box (NFS) and add it later as a backup repository.

So if I connect the NAS to the physical server with iSCSI or NFS and create a repository, how should I connect the virtual Hot-Add proxy to that repository? With iSCSI or NFS from inside Windows? Can two different Windows servers connect to the same iSCSI or NFS target without being in a cluster?

[veremin]

Actually, you don't have to connect proxy to repository. In the scenario that I've proposed the traffic flow will look like the following: Datastore -> to Proxy (via Hot-Add) -> to Backup Management server (via network) -> to NAS/repository (via ISCSI)

Or you're after "networkless" scenario?

[Phorward]

OK, I see. I thought the proxy had to have a direct connection to the storage.
It's alright to transfer from the Proxy to the physical backup server over the network since it will be using the separate network intended for backup data.

Thanks! I hope I get to try this.

[veremin]

Actual tests should give you the final answer, indeed. Kindly, keep us updated about the results you get. Thanks.

[foggy]

Also, depending on your hardware, it might be possible to use Direct SAN transport mode with SAS storage, please look at this topic.

[Phorward]

I have now done the installation and taken the first backup.
This is what I ended up doing.
Installed B&R on the physical Windows server and attached the NAS directly to a free NIC port on the server using iSCSI.
Installed the Proxy on a virtual Windows 8 workstation.
So the disks of the virtual machines gets hot added to the proxy and then the data is sent over the network to the physical Windows server who stores it on the NAS. Then the data is copied from the NAS to the directly attached tape library. Nothing too complicated.

I have two B2D jobs and two subsequent D2T jobs.
The first B2D job is a Incremental job that runs every day with synthetic fulls once a week with the option to transform previous full backup chains into rollbacks. Active full backups are run three times a year. I have chosen to keep 365 restore points at the moment. I have a tape job as secondary job.
The second B2D runs a full backup once a year and then writes that backup to tape for archiving purposes.

The two B2D jobs write to different repositories on the same disk.

In total there's 11 virtual machines backed up, 2,3TB data totally. The NAS holds 11TB. The tape media pool for daily backups holds 11 LTO4 tapes and the archive pool holds 9 LTO4 tapes. I have three free slots that I can assign as needed. The daily backup pool is set to get overwritten as needed, I guess I have to watch out so that the full backup doesn't get overwritten before a new full backup gets done?

What I noticed is that the first full backup took a very long time. Processing those 11 machines with 2,3TB took over 65 hours. The incremental backup the next day was fast.
The full backup started out at over 80 MB/s but as it progressed it kept dropping constantly and in the end it was only 5 MB/s resulting in an average of 23 MB/s. These are the bottlenecks: Source 7% > Proxy 26% > Network 21% > Target 88%.
The feeling I got is that the B&R server isn't powerful enough. It's a HP ProLiant DL380 G5 with two dual core CPU's at 2GHz and 16GB memory (memory shouldn't be the issue). Looking at the task manager the first CPU core is at about 80% and the other three at 50% during backups. The process VeeamAgent.exe is the the one taking most of the power at about 25%, does those 25% mean something like maxing out one core?
The backup server also have vCenter on it and the SQL databases for both vCenter and Veeam.
Should I have a more powerful machine as backup server?

Also, depending on your hardware, it might be possible to use Direct SAN transport mode with SAS storage, please look at this topic.

This was not an option since the backup server is in another location. I have read that the first Optical SAS products have appeared but we're not there yet.

[foggy]

Should I have a more powerful machine as backup server?

No, your bottleneck stats (Target 88%) indicates that the write speed to the target storage is the weakest point in the entire backup process.

[Phorward]

No, your bottleneck stats (Target 88%) indicates that the write speed to the target storage is the weakest point in the entire backup process.

Since the speed was good to begin with but dropped I suppose that it started out with sequential writes and later on the IOPS increased?
If so then what is the standard things to look out for regarding getting better performance in that case?
I know that when I had the NAS (It's a Synology RackStation) connected to a VMware host with iSCSI the performance was lousy and got much better when I connected it using NFS (a well known problem with Synology). But I don't think NFS is an option with Windows?

[foggy]

Normal full backup is sequential writes only. What if you try to back up to the local disk instead?

Have you monitored memory usage on the proxy server during the backup?

[Phorward]

Thanks for the suggestions. Unfortunately I have only small disks attached locally, I will have to see if I can find some disks to put in the server. I'll have a look at the memory of the Proxy. I have another full backup that runs tomorrow that I can monitor.

I also have a another problem that I will post in a new thread.

[Phorward]

Have you monitored memory usage on the proxy server during the backup?

I assigned more memory and two more cores to the proxy but the speed still drops.
I ran the SysInternals Cacheset utility on both the proxy and the backup server but the cache is far from full on both. Besides, according to that thread the caching problem should have been fixed since B&R v6.1?
I noticed that most of those having this problem were using Windows Server 2008 SP2, the same as me. Might be worth trying to upgrade to 2008 R2 but that's a lot of work since vCenter is running on the same machine.

[foggy]

Then it is worth testing with some other storage and contacting support for more thorough investigation.

[Phorward]

I connected the NAS to the backup proxy instead and now the backup runs without problem. It seems using Windows Server 2008 as a backup repository might not work so well.