Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 07, 2019 3:55 pm

Hi,

[Newbie alert - new to VEEAM and new to VM's altogether...]

I have a Hyper-V running MS Exchange Server 2019 on WS2019. This is a small lab environment. About 3 really active mail boxes. The EDB file is about 6 GB.

I am backing it up Application Aware. It is taking 8 hrs.

Now, this is a lab environment. The Hyper-V physical sever isn't a "real" server - it's an i7 8700 with 32 GB RAM, and an SSD M2 drive. This is hardly a professional server, but considering all it's doing is running ES2019 on one VM, and on the other VM is running a PDC for a 5 computer and 3 user domain, that should be good enough.

The Backup Server running VEAM is a very old weak computer, but since I've offloaded processing onto the Hyper-V, I assumed it wouldn't matter. It's drive is a "regular" hard drive.

My question: how can I determine what is causing the slowness?

TIA,
mlavie

vmniels
Veeam Software
Posts: 2200
Liked: 482 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by vmniels » Feb 07, 2019 4:01 pm

If you look at the backup job status window it will tell a bottleneck which can be source, network, target,... this is a good start to figure out where the issue may reside. More information can be found in the userguide.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 07, 2019 4:24 pm

Hi vmniels,

OK - it is showing bottleneck = Source, which is busy 73% of the time. I notice that the disk reading time is listed as "Hard disk 1 (128.0 GB) 18.5 GB read at 623 KB/s [CBT]" 623 KB/s seems pretty slow for an SSD M2 disk to be read.

Do you have any ideas based on this?

TIA,
mlavie

mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 08, 2019 10:21 pm

Hi,

Any ideas? I should also mention that ever once in a while. the backup is not insanely slow. Occasionally it reads at 4 MB/s [CBT] instead of around 600 KB/s [CBT].

I'd really appreciate some help.

mlavie

vmniels
Veeam Software
Posts: 2200
Liked: 482 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by vmniels » Feb 09, 2019 9:49 am

This would mean your source is too busy. Could you tell a bit more on the hyper-v setup? Type of disks, raid being used, how many VMs are running?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 09, 2019 11:12 am

Beste Niels,


This is a lab setup.

The physical Hyper-V "server" is basically a strong desktop: i7 8700, 32 GB RAM, Disk is a 1 TB SSD M2. There is no RAID or anything else "fancy" going on. It (and all its VMs) are WS2019 Datacenter Edition. The backup is Reverse Incremental, Application Aware, with off-loading onto the Hyper-V host proxy. All computers are on the same physical LAN, connected to the same "dumb" switch.
There are 3 VMs:
  • VM1: PDC. 4 logical processors. 8 GB RAM.
    However, there are only 3 active users and 4 computers in the domain, so it doesn't actually do much (there is another physical DC in the domain, as a backup).
  • VM2: Exchange Server 2019 and RRAS. 6 logical processors. 13 GB RAM.
    There are only really 3 active Exchange mailboxes. One contains a lot of old mails, so the EDB size is about 6 GB. RRAS is only very rarely accessed. Due to a bug in RRAS 2019, 2 NICs are required.
  • VM3: Basically running nothing. 2 logical processors. 3 GB RAM.
    This is used for experimentation occasionally.

    As you can see, I have allocated the maximum amount of RAM to the VMs as Hyper-V will allow, so I suppose the Hyper-V itself might be low on memory when Veeam runs.

    I can tell you that the Veeam backup of the physical DC is much faster (also Application Aware) - it is about 100 times faster (about 410 MB/s [CBT]).
Unfortunately - "Murphy's Law" always holds true - I got up last night at 01:00 so I could monitor the Hyper-V and see if CPU/Memory/IO were overloaded - but the backup was not very slow this time. By this, I mean the processing read rate was 4 MB/s [CBT]. Actually, even 4 MB/s is pretty slow, isn't it? CPU/Memory/IO did not seem to be even close to being overloaded - so why only 4 MB/s?
I will keep on waking up at 01:00 until I catch it processing super-slow and can see if anything appears overloaded.

Any ideas?


Bij voorbaat dank,
mlavie

mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 11, 2019 9:31 am

Beste Niels,

The problem has returned.

I am noticing that there is a "stop and go" effect -it seems to be processing for a while, then pauses, and then works some more. The progress graph almost looks like many sharks' fins sticking up through the water...

Please see a screenshot:
https://xcserver102.xcircular.com/EraseMe/untitled.png

mvg,
mlavie

Mike Resseler
Product Manager
Posts: 5226
Liked: 549 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by Mike Resseler » Feb 12, 2019 7:08 am

Hey mlavie,

There is probably something in your setup causing those "shark fins". In one of your previous jobs you talked about offhost proxy. I am assuming you are using on-host proxy? Not offhost?

Is there AV running? And if so, did you excluded the necessary Veeam items?

Last but not least, to really figure out what is causing this (which I am pretty much convinced it will be most likely something outside Veeam), logs would need to be investigated. So I would consider now to create a support call (and please post the case ID here, and afterwards the outcome from the engineers)

Thanks
Mike

mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 12, 2019 8:10 am

Newbie alert: What is "AV"? What are the necessary Veeam Items I need to exclude?

Regards,
mlavie

Mike Resseler
Product Manager
Posts: 5226
Liked: 549 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by Mike Resseler » Feb 12, 2019 8:17 am

Sorry, AV means Anti-Virus.

The exclusions are described here: https://www.veeam.com/kb1999

Brgds,
Mike

vmniels
Veeam Software
Posts: 2200
Liked: 482 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by vmniels » Feb 12, 2019 9:00 am

Have you also considered stepping away from reverse incremental? This is very intensive in terms of IO usage (4x required) where (forever) forward incremental is much more friendly for lower end NAS devices. Reverse incremental usually requires RAID10 to be of value with a low amount of disks.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

mlavie
Influencer
Posts: 12
Liked: never
Joined: Jan 31, 2019 5:25 pm
Contact:

Re: Application Aware Backup Takes 8 Hrs

Post by mlavie » Feb 13, 2019 3:59 pm

I tried going to regular forward backup. Didn't help. I'm about to remove Application Aware, to see if that makes any difference.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests