PS6100 source, MD3000i target

[PaulNSW]

Does anyone have any tips on how best to backup VM's running on ESXi4.1 with an EQL PS6100 as the shared datastore. I would like to back it all up to our old MD3000i iSCSI SAN.

The MD3000i, PS6100 and ESX hosts all have a dedicated switch for the iSCSI traffic

At the moment I only have 1 virutal veeam server running everything, backing up to a QNAP NAS. It's using the hot-add methor at the moment.

Should I create a windows or linux backup proxy and connect it to the MD3000i? Should I start a new Windows VM and use the MS iSCSI initiator to connect to both the PS6100 and MD3000i?

[foggy]

Should I create a windows or linux backup proxy and connect it to the MD3000i? Should I start a new Windows VM and use the MS iSCSI initiator to connect to both the PS6100 and MD3000i?

You already have this VM (with Veeam B&R installed) and could create an NTFS LUN on MD3000i and present it to this VM as local disk.

[PaulNSW]

I have around 30TB of space on this MD3000i, what's the best way to present that. Could I present it directly as one huge GPT disk? I take it this would be using the MS software iSCSI as VMware is limited to 2TB VMFS?

[dellock6]

Using the iscsi initiator directly inside the VM will bypass VMFS3 limits, so you can use GPT and format directly the MD3000i. You would also need a virtual machine type portgroup in the vswitch connetcted to the iscsi network, and assign a new nic to the Veeam Server on this network. In this way your backups will run lan-free.

[PaulNSW]

ok, so I got my VM running Veeam hooked up to the MD3000i via iSCSI

The backup is being performed by hot-add.

Is the first backup a lot slower that subsequent ones? I'm currently only getting 10MB/s processing rate, with the bottleneck being the target.

Looking at Performance Monitor, disk access is around 34Mb/s combined R&W, and network activity (to MD3000i iSCSI SAN) is around 50 Mbps. However if I do a straight copy from a file stored on the main SAN (PS6100) to the MD3000i, I'm getting around 110MB/s disk access and 350Mbps network

[dellock6]

Which backup method are you using? If it's reverse incremental for example, you cannot get full speed of the storage since per every bit you save, the storage has to do 3 I/O operations, thus reducing the effective speed.
A good test with Veeam to see the speed of the Storage can be the first run of a forward incremental backup or an active full backup. Both of these methods use 1 I/O per saved bit.

Luca.

[Vitaliy S.]

Paul, subsequent backup job runs will have much better performance since you will be leveraging changed block tracking.

Luca, it does not matter what mode has been configured, since Paul is running his first full backup pass, which is essentially the same for forward and incremental backup modes. Difference in performance rates will only be observed on the incremental job runs.

[PaulNSW]

500GB over the first 24hrs... is that really the performance I should expect the first time?

At that rate it will take 30 days to take the first backup!!

[foggy]

MD3000i could be the issue. Bottleneck target shows that the job is waiting for the target storage to write the data most of the time, which results in overall low processing performance. Btw, what are the full bottleneck stats numbers for this job?

Could you try to backup to some local disk to compare the speeds?

[Gostev]

500GB over the first 24hrs

This gives 6 MB/s. When even a single consumer grade hard drive can do over 100 MB/s on sequential writes (and full backup is sequential writes).
Something is clearly wrong with the storage, as I am sure normally MD3000i can do way more than that - just because of having multiple spindles alone.

[PaulNSW]

I'm not sure what's going on! I added another disk to the Veeam VM, this one is on the PS6100. Processing rate was around 30MB/s with the bottleneck as the source.

If I try a straight forward copy from Windows, I get a speed of 34MB/s from a disk hosted on the PS6100 to a disk hosted on the MD3000i

[PaulNSW]

I changed the disk from GPT to MBR, and the speed is now 16MB/s

I then changed it back to GPT, but kept the 2TB disk size, the speed is also 16MB/s

We are currently backing up to a QNAP TS-459U+, and it's getting 36MB/s over a 1gigiabit LAN connection SMB share. If I create an iSCSI target on the QNAP and use it, I get 36MB/sec too - this is on our production LAN with all our other traffic too!

the MD3000i is on its own switch purely for iSCSI. Maybe there's something up with the networking between them, but then why is Windows copy performance a lot higher than using Veeam?

[dellock6]

Maybe is a stupid idea, but have you enabled jumbo frames on the Equallogic? I've seen sometimes little problems with them in some situations, maybe it's worth a try (if you can disable them without blocking the storage...)

[PaulNSW]

Not such a stupid idea after all! Disabling jumbo frames on the MD (and vswitch and guest nics) bumps the processing speed up to 36MB/s. With VSS and indexing enabled, the bottleneck shifts to the source and drops to 27MB/s

[dellock6]

the drop is reasonable, VSS and indexing adds some load on the source since you need to read those infro from the storage. 36 MB/s is quite a gain, can you also try to disable jumbo frames also on the Equallogic?

[PaulNSW]

hmmm, I think I'll leave the EqualLogic alone - it contains all our production VMs and I dont want to go messing around with that

The bottle neck has shifted back to the target now, but still chugging along at 26MB/s
Source: 68% (backing up exchange 2010 currently - hot add VMFS disks on PS6100)
Proxy: 50%
Network 32%
Target: 80% (MD3000i)

[Gostev]

If this a full backup? If yes, than one word: Slow
I also find it very strange and unexpected why no number in the bottleneck stat is 99%.

[PaulNSW]

Yeah, that was a full backup, with VSS and Indexing enabled

[Gostev]

VSS and indexing has no effect on actual data processing performance.

[jorgeblat]

In some post on the net somebody had problems with MD3200i and sugested the best MTU to be 6000 (not typical jumbo frame). I am using that value now and it seems faster than jumbo frame or standard MTU size.

[PaulNSW]

I might give that a go. I've left my backup running over the weekend - 15TB worth. It started off pretty well, the first VM/hard disk at 32MB/s. Then onto another VM, 1st disk (OS) at 77MB/s, then 25, 12, 7, 8, 6, 4, 4MB/s - gradually getting slower and slower for each subsequent disk.

Think I may start pestering Dell support about this one!

[Gostev]

The gradual slowdown is actually the known issue > Full backup speed decreases as it runs

[PaulNSW]

Roll on version 6.1!!

Just upgraded to ESXi5 and presenting the MD3000i via RDM. I'll see what performance difference there is in the meantime

[pcmmitham]

be careful with MD3000i we got rid of ours because of a known dell issue...no alerts when cache batteries no longer charge! it was a serious enough issue for us to dump the md and go with a set of Equallogics! for now we have relegated our Md3000i as a shelf to hold other physical servers that dont have rails!!

[PaulNSW]

well 6.1 made no difference - chugging along at at an average 4MB/s

taken 13 hrs to backup 105GB (read) of a 256GB disk containing WSUS files.

I've tried using MS ISCSI initiatior, RDM with LSI SCSI, RDM with paravirtual adapter, regular frames, jumbo frames - and still the network attached QNAP using SMB beats the MD3000i over iSCSI!


This is on one of the two paths in use on the iSCSI switch, the spike is me achieving a 50MB/s copy speed via windows, while Veeam is backing up

[PaulNSW]

I just tried backing up (first full backup) on a new LUN on the same EqualLogic SAN

It starts out a lot faster than the MD3000i, natch, but then it gradually gets slower and slower

[Gostev]

Please let me know the support case ID for this, and I will ask our devs to take a look. I've gotten an idea about a test tool we could build and run against your environment.

[PaulNSW]

So I tried a little work-around...

I backed up, adding an extra disk each time. So starting of with the system disk, then the next run was the system disk and 1 data drive, the next run had 2 data drives, and so on.

Although a pain, this meant each iteration, the disk receiving its first full backup, was read on around 40-70MB/s, with the previously backed up disks only needing a few GB.

[Gostev]

Paul, small update... we could not reproduce this issue internally under any circumstances, so this looks to be environment or storage specific issue. As the next step, we are planning to prepare a special data mover agent with extended performance counters logging, and collect the extended logs from your environment. Thanks!

[PaulNSW]

Thanks for letting me know Gostev