Fast then slow Backup

[JohnGG]

gday everyone, i started a backup job of 2 servers and one begin a mail server (1.5 terabytes in size) and the other is a file server (around 3.5 terabytes in size).
Now the mailserver was fast and ran at roughly 80 mbps, now the file server has been crawling along at around 5 to 10 mbps and has been running for 48 hours and only sits at %34.

Any reason why this started so slow all of a sudden? Its all going to a 6 terabyte western digital USB drive, all other servers have been running super fast backup times its just once it started the file server it started crawling along at a snails pace.

Note: backup server is physical and usb plugged directly in.

[Shestakov]

Hello John,
Is that statistics related to a single job run or it`s an average values across multiple job runs?
Could you also provide the job`s full bottleneck statistics? What transport method is used?
Thanks!

[JohnGG]

The bottle neck goes from source to target and vice versa.
Settings are Optimized for Local target, compression is optimal.
All other 6 servers have been flying but it just bogged down with the file server which i have now taken out of that backup job and will try it separate.

I am just getting some backups (We do replications every night a well to another ESXi box) for off site before christmas break (Cyclone season) so i have 1 BU job with 6 servers, that runs fast at around 80 mbps.
The next one i created were the bigger servers IE(Mail and File) mail moved at good speed 70-80 mbps but the file server just seemed to bog down.
All servers on the same datastores/raid and appear to have the same block size using the commend (fsutil fsinfo ntfsinfo C:).

[Shestakov]

Thanks for the reply, John.

The bottle neck goes from source to target and vice versa.

I meant bottleneck statistics you see in the "Action" section of the console. It shows Load: Source __% > Proxy__% > Network__% > Target__%
And when you click on VM, the "Action" section displays which proxy and transport mode were used:
i.e. Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]

Please provide the stats and check if same transport mode is used for those VMs.

Thanks!

[Vitaliy S.]

In addition to Nikita's post, just wanted to highlight that different processing rate for different VMs can be expected, see this thread for more info > Interesting speed difference

[JohnGG]

Sorry for late reply have eben busy.
This is the stats.

17/12/2015 7:45:55 AM :: Load: Source 87% > Proxy 17% > Network 39% > Target 44%

i ended up removing the file server from the back up and started it by ists elf and its running at a round 50 mbps, i will let you know when its done and shows the the details of load.

[Shestakov]

John,
Since your bottleneck is source, I need to ask once again what your transport mode is

when you click on VM, the "Action" section displays which proxy and transport mode were used:
i.e. Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]

By the way, 50MBps doesn`t look slow.

[JohnGG]

When you say transport mode, do you mean "Optimized for Lan target"? because thats what its set to.
These backups are coming of a SAN storage over a 1 GIG Ethernet, also now the large file server has come to a complete crawl at around 5 mbps so im going to have to cancel this as it will not be finished before the Christmas break
It appears that Veeam cannot handle servers over 4 terabyte is this right because it starts crawling after 1 terabytre. Could it be the USB (The amil server run fine and thats over 1 terabyte in size)?

Everything was running smooth untill it hit 1 terabyte in size, anyway here is a screen shot.


http://imgur.com/p93xeiM

Thanks for the reply as well mate.
Can i aslo ask why my other forum thread has been removed with out an answer?never mind its been merged in vsphere thread

[Shestakov]

By transport mode I meant the method of VMs data retrieval and how it is sent to proxy.
From your screenshot can be seen that you use the slowest one, called Network Mode.
That`s the general reason why bottleneck is the source.

Having SAN infrastructure, the best solution for you is to leverage Direct SAN mode. Please read the requirements.

4TB VMs and USB should work fine. Just a note that processing is shown in "MB per sec", not "Mb per sec".

[JohnGG]

mm ok, i thought the SAN access was only for HP storage.
Where do i find the transport mode as i cant remember ever seeing that when i set the backup jobs or replication (Ill do a search for that settings).
The funny thing is the USB backup jobs all ran fine until it hits about a terabyte in size then it crawls along.

EDIT:Ok we cant use direct SAN access as i dont have a HBA connected to SAN as the only HBA i have is for tape backups.
The san has 2 controllers with 2 LSI logic cards on each controller but both are used for fail over so network mode it is.

[Shestakov]

In simple words, transport mode depends on how backup proxy is connected to a datastore. You can check which proxy is chosen for each job in the job settings(wizard). By default it is set to automatic selection which chooses fastest mode automatically.
As I wrote, to find out which mode is used in the particular job run, you may take a look at the "Actions" pane.
By the way, if you can`t leverage Direct SAN, try Virtual Appliance mode.

[Vitaliy S.]

The funny thing is the USB backup jobs all ran fine until it hits about a terabyte in size then it crawls along.

Does this also happen when you start writing data to this USB drive via Windows Explorer or any other tool?

[JohnGG]

Well copying a large file now is slow (Around 20 MBps) as the USB drive is being used by Veeam but generally its around 70 when copying files.
image again

http://imgur.com/hBG0Nv1

[Shestakov]

The funny thing is the USB backup jobs all ran fine until it hits about a terabyte in size then it crawls along.

That is a strange thing. Do you observe it all the time or it happened once or twice?
Could you also answer Vitaliy`s question please?

Thanks!

[JohnGG]

Yeh i did, when using Explorer its usually copying at around 70 ( but generally its around 70 when copying files.).
i have only just started doing backups for disaster recovery in case we get a cyclone over the break, normally we replicate every night anyway but paranoid as i am i wanted yet another copy off site along with a full LTO6 tape lol.
All the other servers i have backed up, some around 1.6 terabyte all ran fast and finished with a reasonable time frame (Mail is around 1.2 terabyte and finished in 10 hours).
Only thing i can think of is that the physical proxy server doesnt have enough RAM and is now sitting at %95 Physically used. Veeam agent sitting around 1 gig of RAM.

Either if it doesnt finish i will have to kill the back up before we close tomorrow, i have all the other servers and will have all the files on LTO6 if this building burns down.

[JohnGG]

I also cant use virtual appliance mode because the backup server is a physical and from i just read it has to be a virtual on the same SAN as the ones you are backing up.

[veremin]

The backup server itself doesn't have to be a virtual in this case. You can assign a role of hot-add proxy to any Windows-based Virtual Machine that runs on same host datastore as VMs to backup do. Thanks.

[winnt]

What is the OS on the backup proxy? If it is 2008, then upgrade to at least 2008 R2.

[Neilxu]

Hi All

We have very wierd situation: within the same network two hosts have very different backup performance.

host01 has backup rate of 40MB/s
host02 has backup rate of only 5MB/s (backup analysis indicating a bottleneck of source)

two hosts are in the same network, similar hardware specification. Can anyone advise what may make the difference ???

thanks very much !!!

Neil

[Shestakov]

Hi Neil,
As was mentioned above in the thread, speed difference can take place because of number of factors such as amount of changed blocks, their compressibility, inline deduplication etc.
What is the full bottleneck statistics for both jobs? Are you on vSphere or Hyper-V? What transport mode was used?
Thanks!

[Delo123]

@John Ram could definitely be the issue here. You say file copies run at 70MB/s (Megabytes correct?) which is quite much for a single usb disk. Anyway, probably windows file cache is helping you here. I assume when you copy a really big file speed will go down...
Anyway, the same windows file caching feature is probably killing your backup performance also, since especially 2008 didn't have a good way of releasing memory, so probably you are working on your swap file. Updating to 2012R2 or at least 2008r2 would be good. Also there are some tools to tweak windows caching on 2008. But if possible try to increase ram on your backup machine / proxy...

[Neilxu]

Hi Shestakov

thanks for your replay. on the slower host the bottleneck stats are source 99% proxy 53% network 0% target 0%, and the stats is for a VM that is currently shutdown. I am current on VSphere. I am current taking a simple veeamzip for each of the VMs on different host.

Neil

[Delo123]

Looks like an over utilized primary storage. What kind of storage is it and how is it connected? Any latency stats on the san? I would suspect very high latency during backups with this load on the source...

[dellock6]

I would also check that there are no pending snapshots left in the virtual machines, and what other workloads are running when the backup happens (defrag, maintenance, antivirus updates) in other VMs using the same storage.

[JohnGG]

@John Ram could definitely be the issue here. You say file copies run at 70MB/s (Megabytes correct?) which is quite much for a single usb disk. Anyway, probably windows file cache is helping you here. I assume when you copy a really big file speed will go down...
Anyway, the same windows file caching feature is probably killing your backup performance also, since especially 2008 didn't have a good way of releasing memory, so probably you are working on your swap file. Updating to 2012R2 or at least 2008r2 would be good. Also there are some tools to tweak windows caching on 2008. But if possible try to increase ram on your backup machine / proxy...

Sorry been on holidays (Yayyyyy but now back at work Nooooo!)
Anyway i was thinking it was RAM as well because its nearly at %90 with out running any backups and yes its just 2008 and not R2.
This wont be an ongoing thing but was mainly only for off site disaster recovery over the holidays.

I will see if i have some spare RAM sticks around and maybe give it a bit of a boost.

Cheers

[Shestakov]

Hi Shestakov
thanks for your replay. on the slower host the bottleneck stats are source 99% proxy 53% network 0% target 0%, and the stats is for a VM that is currently shutdown. I am current on VSphere. I am current taking a simple veeamzip for each of the VMs on different host.
Neil

Thanks for the reply, Neil!
What transport mode is used for these VMs?

host01 has backup rate of 40MB/s
host02 has backup rate of only 5MB/s (backup analysis indicating a bottleneck of source)

Is that a stable speed over several backup jobs` runs?
Thanks!

[Neilxu]

Thank you Shestakov, its on network mode at the moment, did you means what transport mode ??
host02 has average speed of 5m/s pretty stable.

[Delo123]

If you cannot do direct SAN simply install a proxy VM (install normal windows vm with some cpu and memory) and then install the proxy role with veeam. Make sure the host with the veeam proxy and veeam server are linked together with the fastest link you have available.
Now you can choose hotadd as transport mode which should be way faster than network mode...

[Shestakov]

Neil,
you are correct, I was talking about transport mode you are using.
Main reason of having source as an explicit bottleneck is the transport mode. Network mode is the slowest one. As John mentioned above, you can leverage Virtual Appliance mode to speed up backup processing.
Thanks!

[Neilxu]

Hi Shestakov thanks very much for the explanation,

I thing I am going to tackle the "same spec/environment, different backup speed" issue first before I start working on moving the transport mode to Virtual appliance. I have dug bit deeper into the log files and did a comparison between ones that are faster and ones that are slower. The log file I used was "VM name".source."name"."name".vmdk.log as it was the bottle neck. (source 99% Proxy 41% network 2% target 1%)

Observation:
somewhere in the log file,

for the faster ones I have

[05.01.2016 11:08:01] < 10100> vim| [SOAP] Successfully logged in ( server: [VESPA], user: [Domain\neilxu], sessionKey: [52464e2c-dc7c-1cd8-04c3-d75e55fe4cc7] )
[05.01.2016 11:08:01] < 10100> vim| VI SOAP connection initialized.
[05.01.2016 11:08:01] < 10100> vim| CTK_change_extent. Start [0]. Size [64424509440]. Extents number: [38].

for the slower ones I have

06.01.2016 10:02:33] < 12120> vim| [SOAP] Successfully logged in ( server: [VESPA], user: [Domain\neilxu], sessionKey: [5263d0cd-16ab-d309-174f-3cc1e5a365f0] )
[06.01.2016 10:02:33] < 12120> vim| VI SOAP connection initialized.
[06.01.2016 10:02:34] < 12120> vim| Operation 'RequestDiskChanges' failed: 'Soap fault. Error caused by file /vmfs/volumes/52e0d703-fa713f72-5501-64700213048d/APPSERV/APPSERV_1-000002.vmdkDetail: '', endpoint: '''. Retrying...
[06.01.2016 10:02:37] < 12120> vim| [SOAP] Logged out from the server [VESPA].
[06.01.2016 10:02:37] < 12120> vim| [SOAP] Successfully logged in ( server: [VESPA], user: [Domain\neilxu], sessionKey: [52084d03-ef86-c7d2-9f88-754dca9aaee3] )
[06.01.2016 10:02:37] < 12120> vim| Operation 'RequestDiskChanges' failed: 'Soap fault. Error caused by file /vmfs/volumes/52e0d703-fa713f72-5501-64700213048d/APPSERV/APPSERV_1-000002.vmdkDetail: '', endpoint: '''. Retrying...
[06.01.2016 10:02:40] < 12120> vim| [SOAP] Logged out from the server [VESPA].

so far that is the difference I found by comparing the log files, seems the "requestDiskchange" operation is the dealbreaker ?!?!

Couple of Question :
1 what does SOAP do ?
2 and does the fact "Operation 'RequestDiskChanges' failed" have anything to do with difference backup performance ???
3 I have checke the datastore for the slower VM, the slower machine does even have a VMDK file named "APPSERV_1-000002.vmdk", is this some temp file created during backup process ??

Thanks very much for answering those questions!!