Discussions specific to the VMware vSphere hypervisor
Post Reply
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

PS6100 source, MD3000i target

Post by PaulNSW »

Does anyone have any tips on how best to backup VM's running on ESXi4.1 with an EQL PS6100 as the shared datastore. I would like to back it all up to our old MD3000i iSCSI SAN.

The MD3000i, PS6100 and ESX hosts all have a dedicated switch for the iSCSI traffic

At the moment I only have 1 virutal veeam server running everything, backing up to a QNAP NAS. It's using the hot-add methor at the moment.

Should I create a windows or linux backup proxy and connect it to the MD3000i? Should I start a new Windows VM and use the MS iSCSI initiator to connect to both the PS6100 and MD3000i?
foggy
Veeam Software
Posts: 21035
Liked: 2106 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: PS6100 source, MD3000i target

Post by foggy »

PaulNSW wrote:Should I create a windows or linux backup proxy and connect it to the MD3000i? Should I start a new Windows VM and use the MS iSCSI initiator to connect to both the PS6100 and MD3000i?
You already have this VM (with Veeam B&R installed) and could create an NTFS LUN on MD3000i and present it to this VM as local disk.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

I have around 30TB of space on this MD3000i, what's the best way to present that. Could I present it directly as one huge GPT disk? I take it this would be using the MS software iSCSI as VMware is limited to 2TB VMFS?
dellock6
Veeam Software
Posts: 6121
Liked: 1916 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: PS6100 source, MD3000i target

Post by dellock6 » 2 people like this post

Using the iscsi initiator directly inside the VM will bypass VMFS3 limits, so you can use GPT and format directly the MD3000i. You would also need a virtual machine type portgroup in the vswitch connetcted to the iscsi network, and assign a new nic to the Veeam Server on this network. In this way your backups will run lan-free.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

ok, so I got my VM running Veeam hooked up to the MD3000i via iSCSI

The backup is being performed by hot-add.

Is the first backup a lot slower that subsequent ones? I'm currently only getting 10MB/s processing rate, with the bottleneck being the target.

Looking at Performance Monitor, disk access is around 34Mb/s combined R&W, and network activity (to MD3000i iSCSI SAN) is around 50 Mbps. However if I do a straight copy from a file stored on the main SAN (PS6100) to the MD3000i, I'm getting around 110MB/s disk access and 350Mbps network
dellock6
Veeam Software
Posts: 6121
Liked: 1916 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: PS6100 source, MD3000i target

Post by dellock6 »

Which backup method are you using? If it's reverse incremental for example, you cannot get full speed of the storage since per every bit you save, the storage has to do 3 I/O operations, thus reducing the effective speed.
A good test with Veeam to see the speed of the Storage can be the first run of a forward incremental backup or an active full backup. Both of these methods use 1 I/O per saved bit.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Vitaliy S.
VP, Product Management
Posts: 26796
Liked: 2655 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: PS6100 source, MD3000i target

Post by Vitaliy S. »

Paul, subsequent backup job runs will have much better performance since you will be leveraging changed block tracking.

Luca, it does not matter what mode has been configured, since Paul is running his first full backup pass, which is essentially the same for forward and incremental backup modes. Difference in performance rates will only be observed on the incremental job runs.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

500GB over the first 24hrs... is that really the performance I should expect the first time?

At that rate it will take 30 days to take the first backup!!
foggy
Veeam Software
Posts: 21035
Liked: 2106 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: PS6100 source, MD3000i target

Post by foggy »

MD3000i could be the issue. Bottleneck target shows that the job is waiting for the target storage to write the data most of the time, which results in overall low processing performance. Btw, what are the full bottleneck stats numbers for this job?

Could you try to backup to some local disk to compare the speeds?
Gostev
Chief Product Officer
Posts: 31006
Liked: 6418 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: PS6100 source, MD3000i target

Post by Gostev »

PaulNSW wrote:500GB over the first 24hrs
This gives 6 MB/s. When even a single consumer grade hard drive can do over 100 MB/s on sequential writes (and full backup is sequential writes).
Something is clearly wrong with the storage, as I am sure normally MD3000i can do way more than that - just because of having multiple spindles alone.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

I'm not sure what's going on! I added another disk to the Veeam VM, this one is on the PS6100. Processing rate was around 30MB/s with the bottleneck as the source.

If I try a straight forward copy from Windows, I get a speed of 34MB/s from a disk hosted on the PS6100 to a disk hosted on the MD3000i
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

I changed the disk from GPT to MBR, and the speed is now 16MB/s

I then changed it back to GPT, but kept the 2TB disk size, the speed is also 16MB/s

We are currently backing up to a QNAP TS-459U+, and it's getting 36MB/s over a 1gigiabit LAN connection SMB share. If I create an iSCSI target on the QNAP and use it, I get 36MB/sec too - this is on our production LAN with all our other traffic too!

the MD3000i is on its own switch purely for iSCSI. Maybe there's something up with the networking between them, but then why is Windows copy performance a lot higher than using Veeam?
dellock6
Veeam Software
Posts: 6121
Liked: 1916 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: PS6100 source, MD3000i target

Post by dellock6 » 1 person likes this post

Maybe is a stupid idea, but have you enabled jumbo frames on the Equallogic? I've seen sometimes little problems with them in some situations, maybe it's worth a try (if you can disable them without blocking the storage...)
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

Not such a stupid idea after all! Disabling jumbo frames on the MD (and vswitch and guest nics) bumps the processing speed up to 36MB/s. With VSS and indexing enabled, the bottleneck shifts to the source and drops to 27MB/s
dellock6
Veeam Software
Posts: 6121
Liked: 1916 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: PS6100 source, MD3000i target

Post by dellock6 »

the drop is reasonable, VSS and indexing adds some load on the source since you need to read those infro from the storage. 36 MB/s is quite a gain, can you also try to disable jumbo frames also on the Equallogic?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

hmmm, I think I'll leave the EqualLogic alone - it contains all our production VMs and I dont want to go messing around with that :)

The bottle neck has shifted back to the target now, but still chugging along at 26MB/s
Source: 68% (backing up exchange 2010 currently - hot add VMFS disks on PS6100)
Proxy: 50%
Network 32%
Target: 80% (MD3000i)
Gostev
Chief Product Officer
Posts: 31006
Liked: 6418 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: PS6100 source, MD3000i target

Post by Gostev »

If this a full backup? If yes, than one word: Slow
I also find it very strange and unexpected why no number in the bottleneck stat is 99%.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

Yeah, that was a full backup, with VSS and Indexing enabled
Gostev
Chief Product Officer
Posts: 31006
Liked: 6418 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: PS6100 source, MD3000i target

Post by Gostev »

VSS and indexing has no effect on actual data processing performance.
jorgeblat
Lurker
Posts: 1
Liked: never
Joined: May 28, 2012 3:22 am
Full Name: Jorge Blat Palacios
Contact:

Re: PS6100 source, MD3000i target

Post by jorgeblat »

In some post on the net somebody had problems with MD3200i and sugested the best MTU to be 6000 (not typical jumbo frame). I am using that value now and it seems faster than jumbo frame or standard MTU size.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

I might give that a go. I've left my backup running over the weekend - 15TB worth. It started off pretty well, the first VM/hard disk at 32MB/s. Then onto another VM, 1st disk (OS) at 77MB/s, then 25, 12, 7, 8, 6, 4, 4MB/s - gradually getting slower and slower for each subsequent disk.

Think I may start pestering Dell support about this one!
Gostev
Chief Product Officer
Posts: 31006
Liked: 6418 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: PS6100 source, MD3000i target

Post by Gostev »

The gradual slowdown is actually the known issue > Full backup speed decreases as it runs
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

Roll on version 6.1!! :D

Just upgraded to ESXi5 and presenting the MD3000i via RDM. I'll see what performance difference there is in the meantime
pcmmitham
Influencer
Posts: 18
Liked: never
Joined: Jan 17, 2012 6:18 pm
Full Name: Pierre Mitham
Contact:

Re: PS6100 source, MD3000i target

Post by pcmmitham »

be careful with MD3000i we got rid of ours because of a known dell issue...no alerts when cache batteries no longer charge! it was a serious enough issue for us to dump the md and go with a set of Equallogics! for now we have relegated our Md3000i as a shelf to hold other physical servers that dont have rails!!
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

well 6.1 made no difference - chugging along at at an average 4MB/s

taken 13 hrs to backup 105GB (read) of a 256GB disk containing WSUS files.

I've tried using MS ISCSI initiatior, RDM with LSI SCSI, RDM with paravirtual adapter, regular frames, jumbo frames - and still the network attached QNAP using SMB beats the MD3000i over iSCSI!

Image
This is on one of the two paths in use on the iSCSI switch, the spike is me achieving a 50MB/s copy speed via windows, while Veeam is backing up
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

I just tried backing up (first full backup) on a new LUN on the same EqualLogic SAN

It starts out a lot faster than the MD3000i, natch, but then it gradually gets slower and slower

Image
Gostev
Chief Product Officer
Posts: 31006
Liked: 6418 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: PS6100 source, MD3000i target

Post by Gostev »

Please let me know the support case ID for this, and I will ask our devs to take a look. I've gotten an idea about a test tool we could build and run against your environment.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

So I tried a little work-around...

I backed up, adding an extra disk each time. So starting of with the system disk, then the next run was the system disk and 1 data drive, the next run had 2 data drives, and so on.

Although a pain, this meant each iteration, the disk receiving its first full backup, was read on around 40-70MB/s, with the previously backed up disks only needing a few GB.
Gostev
Chief Product Officer
Posts: 31006
Liked: 6418 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: PS6100 source, MD3000i target

Post by Gostev »

Paul, small update... we could not reproduce this issue internally under any circumstances, so this looks to be environment or storage specific issue. As the next step, we are planning to prepare a special data mover agent with extended performance counters logging, and collect the extended logs from your environment. Thanks!
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

Re: PS6100 source, MD3000i target

Post by PaulNSW »

Thanks for letting me know Gostev :)
Post Reply

Who is online

Users browsing this forum: Google Feedfetcher and 58 guests