Application Aware Backup Takes 8 Hrs

[mlavie]

Hi,

[Newbie alert - new to VEEAM and new to VM's altogether...]

I have a Hyper-V running MS Exchange Server 2019 on WS2019. This is a small lab environment. About 3 really active mail boxes. The EDB file is about 6 GB.

I am backing it up Application Aware. It is taking 8 hrs.

Now, this is a lab environment. The Hyper-V physical sever isn't a "real" server - it's an i7 8700 with 32 GB RAM, and an SSD M2 drive. This is hardly a professional server, but considering all it's doing is running ES2019 on one VM, and on the other VM is running a PDC for a 5 computer and 3 user domain, that should be good enough.

The Backup Server running VEAM is a very old weak computer, but since I've offloaded processing onto the Hyper-V, I assumed it wouldn't matter. It's drive is a "regular" hard drive.

My question: how can I determine what is causing the slowness?

TIA,
mlavie

[nielsengelen]

If you look at the backup job status window it will tell a bottleneck which can be source, network, target,... this is a good start to figure out where the issue may reside. More information can be found in the userguide.

[mlavie]

Hi vmniels,

OK - it is showing bottleneck = Source, which is busy 73% of the time. I notice that the disk reading time is listed as "Hard disk 1 (128.0 GB) 18.5 GB read at 623 KB/s [CBT]" 623 KB/s seems pretty slow for an SSD M2 disk to be read.

Do you have any ideas based on this?

TIA,
mlavie

[mlavie]

Hi,

Any ideas? I should also mention that ever once in a while. the backup is not insanely slow. Occasionally it reads at 4 MB/s [CBT] instead of around 600 KB/s [CBT].

I'd really appreciate some help.

mlavie

[nielsengelen]

This would mean your source is too busy. Could you tell a bit more on the hyper-v setup? Type of disks, raid being used, how many VMs are running?

[mlavie]

Beste Niels,


This is a lab setup.

The physical Hyper-V "server" is basically a strong desktop: i7 8700, 32 GB RAM, Disk is a 1 TB SSD M2. There is no RAID or anything else "fancy" going on. It (and all its VMs) are WS2019 Datacenter Edition. The backup is Reverse Incremental, Application Aware, with off-loading onto the Hyper-V host proxy. All computers are on the same physical LAN, connected to the same "dumb" switch.
There are 3 VMs:

• VM1: PDC. 4 logical processors. 8 GB RAM.
  However, there are only 3 active users and 4 computers in the domain, so it doesn't actually do much (there is another physical DC in the domain, as a backup).

• VM2: Exchange Server 2019 and RRAS. 6 logical processors. 13 GB RAM.
  There are only really 3 active Exchange mailboxes. One contains a lot of old mails, so the EDB size is about 6 GB. RRAS is only very rarely accessed. Due to a bug in RRAS 2019, 2 NICs are required.

• VM3: Basically running nothing. 2 logical processors. 3 GB RAM.
  This is used for experimentation occasionally.
  
  As you can see, I have allocated the maximum amount of RAM to the VMs as Hyper-V will allow, so I suppose the Hyper-V itself might be low on memory when Veeam runs.
  
  I can tell you that the Veeam backup of the physical DC is much faster (also Application Aware) - it is about 100 times faster (about 410 MB/s [CBT]).

Unfortunately - "Murphy's Law" always holds true - I got up last night at 01:00 so I could monitor the Hyper-V and see if CPU/Memory/IO were overloaded - but the backup was not very slow this time. By this, I mean the processing read rate was 4 MB/s [CBT]. Actually, even 4 MB/s is pretty slow, isn't it? CPU/Memory/IO did not seem to be even close to being overloaded - so why only 4 MB/s?
I will keep on waking up at 01:00 until I catch it processing super-slow and can see if anything appears overloaded.

Any ideas?


Bij voorbaat dank,
mlavie

[mlavie]

Beste Niels,

The problem has returned.

I am noticing that there is a "stop and go" effect -it seems to be processing for a while, then pauses, and then works some more. The progress graph almost looks like many sharks' fins sticking up through the water...

Please see a screenshot:
https://xcserver102.xcircular.com/EraseMe/untitled.png

mvg,
mlavie

[Mike Resseler]

Hey mlavie,

There is probably something in your setup causing those "shark fins". In one of your previous jobs you talked about offhost proxy. I am assuming you are using on-host proxy? Not offhost?

Is there AV running? And if so, did you excluded the necessary Veeam items?

Last but not least, to really figure out what is causing this (which I am pretty much convinced it will be most likely something outside Veeam), logs would need to be investigated. So I would consider now to create a support call (and please post the case ID here, and afterwards the outcome from the engineers)

Thanks
Mike

[mlavie]

Newbie alert: What is "AV"? What are the necessary Veeam Items I need to exclude?

Regards,
mlavie

[Mike Resseler]

Sorry, AV means Anti-Virus.

The exclusions are described here: https://www.veeam.com/kb1999

Brgds,
Mike

[nielsengelen]

Have you also considered stepping away from reverse incremental? This is very intensive in terms of IO usage (4x required) where (forever) forward incremental is much more friendly for lower end NAS devices. Reverse incremental usually requires RAID10 to be of value with a low amount of disks.

[mlavie]

I tried going to regular forward backup. Didn't help. I'm about to remove Application Aware, to see if that makes any difference.

[mlavie]

Hi All,

I found the problem.

Since the Hyper-V was a brand new physical server, I did not suspect hardware failure. But a friend suggested I try copying a large file manually from the Hyper-V to the Veeam Server. It was bizarrely slow. After ruling out cables, switches, etc., I discovered that the NIC was at fault. I put a new NIC in the Hyper-V. The latest 8 hr. incremental backup turned into a 15 min full backup.

I should mention that the "Bottleneck" report was deceptive. "Source" was given as the bottleneck, not "Network". I guess by "Network", Veeam means *overload* of the network, and not that network *slowness* is causing delay. If I might respectfully make a suggestion to Veeam Product Management, I would suggest either that the name of the bottleneck issue be renamed to "network overload", or even better, that there be an additional bottleneck category called "network slowness".

In any case, thanks to all who responded.

[Mike Resseler]

Hey Mlavie,

First: It is great that you found it. A bit surprised that a faulty network card can cause this issue, especially since it still (more or less) worked, just slow.
Second: Good suggestion, but in this case I kind of fear that the problem is indeed at the source since it starts there and it might be very difficult for us to "see it" unless we would start monitoring specifically NICs. But I will ask and see if they would be able to see something at that specific point in the chain.