BUC Odd issues?

[kspare]

I'm trying to figure out how to maximize BUC for a customer.

I have 2 server 2016 REPO's, connected to 40gb storage servers. each server has 40gb connectivity. I have compression turned off and inline dedup turned off.

These are virtual servers sitting on esxi, so they do only have 10gb adapters within vmware.

I'm only seeing 3gb/s of transfer speed, and the job shows the source as the issue. 3/10 of a gig link.

The storage server is very fast and will move data for storage vmotion at around 8-12gb/s basically 100% of a gigi link.

Running esxi 6.5, intel 40gb adapters, all esxi updates are done, all vmware tools updates are done.

Am I missing something?

[foggy]

Hi Kevin, since source is identified as the bottleneck, you should pay attention to data retrieval from the source storage - could you please tell what transport mode is being utilized (you can see it in the job session log right next to the proxy server name)?

[kspare]

This is all I see.

Code: Select all

Copying restore point 5/8/2020 8:10:07 AM from backup repository Server1
Queued for processing at 5/8/2020 9:17:29 AM 	
Required backup infrastructure resources have been assigned 	
VM processing started at 5/8/2020 9:17:29 AM 	
VM size: 5.0 TB 	
Saving FS01.vmx 	00:00
Saving FS01.nvram 	00:01
Saving FsAwareMeta:eab24e31-9c31-4c9b-89f3-bd961c3de30c:2034 	00:01
Saving FsAwareMeta:eab24e31-9c31-4c9b-89f3-bd961c3de30c:2033 	00:01
Saving FsAwareMeta:eab24e31-9c31-4c9b-89f3-bd961c3de30c:2000 	00:01
Saving FsAwareMeta:eab24e31-9c31-4c9b-89f3-bd961c3de30c:2001 	00:01
Saving GuestIndexData.zip 	00:01
Saving GuestMembers.xml 	00:01
Hard disk 1 (120.0 GB) 78.7 GB read at 114 MB/s	11:46
Hard disk 3 (1024.0 GB) 334.5 GB read at 110 MB/s	52:14
Error: Job has been disabled by user 
Busy: Source 99% > Proxy 24% > Network 6% > Target 4% 	
Primary bottleneck: Source 	
Network traffic verification detected no corrupted blocks 	
Processing finished with errors at 5/8/2020 10:22:17 AM 	

If it helps, when I created the backup job, they processed at about 4 times the speed....thats why i'm confused about how slow the BUC job is.

[foggy]

Ah, so you're talking about backup copy job... Then the source is the source repository for this job - check if it is busy with some other tasks as the job is randomly reading data from the backup files on the source.

[kspare]

Literally nothing else on the storage server. 12 raid 10 mirrors, ssd caching..etc etc....the storage server is not the bottle neck...see my notes above. I can saturate a 10gb link with storage v motions...the two windows machines acting as repo’s sit on nvme storage. Everything is very fast and state of the art.....the full backup went 3 times faster than the backup copy...makes no sense.

[foggy]

Full backup, as well as Storage vMotion, are sequential read workloads, while backup copy reads randomly - that's not a fair comparison.

[kspare]

Why would it be a random read. I did the backup with no compression and no inline dedup.

We are also comparing write speed vs read speed.

Read speed is accelerated by 2tb of ssd....

It should be a sequential read.

[veremin]

By default backup copy job uses periodic type of copy mode, which indeed leverages random reads. Thanks!

[kspare]

Irrelevant. The san the data is copying from is idle. Iops are low. It’s barely doing anything. That’s fine if it has random reads but it’s not copying from a single disk, it’s a 24 drive raid 10 array....disk iops are not the issue.

[Gostev]

If it's not the storage itself, then the only other reason for Source 99% bottleneck can be issues with the connectivity to the storage array. What's clear from the bottleneck statistics you shared, is that backup copy job spends most time just waiting for the issued read operations to return data, while the rest of components are practically idling due to lack of incoming data. And since you said SAN is also mostly idle, that leaves fabric as the only suspect to me. So, check to see if everything is in order there. And if you can't spot any issues, please open a support case - and our engineers should be able to demonstrate the issue without Veeam present in the picture, so you could take it further to your hardware vendors. Thanks!

[HannesK]

Hello,

These are virtual servers sitting on esxi, so they do only have 10gb adapters within vmware.

Have you ever tried iperf between those two Windows VMs? I heard from exactly that scenario that Windows is simply not really fast... not even if two windows machines run on the same ESXi host without any physical network involved.

Two Linux system: almost 10 GBit with iperf
Two Windows 2019: very much slower. With larges packets in iperf they could tune a little bit, but it never got really fast (I remember something between 3-4GBit/s)

And yes of course, please open a case and post the case number for reference.

The customer I'm talking about opened a case with Microsoft, but no solution so far.

Best regards,
Hannes

[kspare]

Well, an easy way to demonstrate this, would be to simply copy the vbk file from one server to the other via smb, I can also try it directly from one storage server to the other as well.

I’ll post back how that goes.

[kspare]

Has Veeam seen BUC jobs run faster than this, or is this just a limitation?

Our customer is looking to get rid of tapes, and we were trying to prove that you could have 2 8 bay qnaps that we would rotate in weekly, do a BUC job and take it back offsite.

[HannesK]

Has Veeam seen BUC jobs run faster than this, or is this just a limitation

2 GByte/s (about 20 GBit/s) over 150 KM with a 100 GBit/s connection is something I remember from one of my old customers. Physical servers of course.

I would expect the software to do at least 4 GByte/s (40 Gbit/s) with physical servers. That was the fastest backup speed I have seen so far for a single proxy & repo server (that required some tuning on the network card). Backup Copy Job and Backup Job use the same data mover.

8 bay qnaps

Well, I think that this can also explain a lot

[kspare]

Explains what? If you read my original post, I'm doing the testing with 2 24 bay servers using raid 10......

[YouGotServered]

It's a running joke in the Veeam community to use QNAPs, or any other prosumer grade storage like that. Sure, QNAP and Synology can saaay they're business class, but are they? I wouldn't run my business on one.

Gostev has sent out several digests outlining the various issues with these type of devices. The most serious issue regards SMB write cache, where the devices will return a "success" for writing the SMB block to storage, despite it still being in cache. This can cause corruption and data loss in the event of a power outage.

[Gostev]

OK, I guess I need to clarify this then: there are actually two distinctly separate issues here.

Unfortunately, the SMB issue I was talking about lies in the SMB stack itself and will affect any storage at all, be it customer-grade or enterprise-grade. The only way to work around this particular issue is to use "continuously available" (CA) file shares. For example, NetApp has this feature, and of course Windows Failover Clusters can do this to > SMB Transparent Failover. The special CA flag such shares carry makes SMB clients use write-through when writing data to a share. Besides this, there's currently no way to force SMB clients to use write-through, which is why we recommend using NFS for any backup storage that does not provide continuously available file shares as a feature.

Now, as far as low end NAS storage, the issue I was talking about there is different, and perpendicular. It comes down to the fact that such storage devices almost never carry battery-backed write cache (BBWC) in their RAID controllers, which are typically software-based to start with. And this presents is a huge problem for any RAID storage in various corner cases, regardless of the file protocol in use. In fact, you will have issues even if you consume such NAS as a block storage (via iSCSI). The infamous one lately is ReFS volumes turning raw after some power or networking glitch.

[kspare]

It's a running joke in the Veeam community to use QNAPs, or any other prosumer grade storage like that. Sure, QNAP and Synology can saaay they're business class, but are they? I wouldn't run my business on one.

Gostev has sent out several digests outlining the various issues with these type of devices. The most serious issue regards SMB write cache, where the devices will return a "success" for writing the SMB block to storage, despite it still being in cache. This can cause corruption and data loss in the event of a power outage.

You are assuming i'm using SMB shares....but i'm not.

[YouGotServered]

You are assuming i'm using SMB shares....but i'm not.

I'm not assuming anything, I'm just queuing you in on the joke. I also provided some supplemental information as to why they kind of suck, and then Gostev elaborated more. As he pointed out, SMB isn't even the whole issue I don't think I ever said you were using SMB.

[david.matthewson]

But surely you would have all your comms kit on a decent UPS/generator so power outages should not be a thing. Decent QNAPS have dual psus as well..And, let's face it, are relatively low cost so hit the SME space neatly. Horses for course perhaps?

[Gostev]

Unfortunately, power outage is not the only scenario that may lead to data loss when BBWC is missing, there are also scenarios around networking glitches.

While you're right in that it's often "risk vs. reward" situation in SME space, I guess the main point here is that with Veeam, you can completely eliminate this risk by going with a general purpose server for just a slightly higher price (the cost of enterprise-grade RAID controller). So, do those risks worth a few hundred bucks when we're talking about hardware that will be used for 10 years?

Besides, some customers are able to simply re-use a server chassis they already have - either a decommissioned one, or by stuffing more hard drives into an existing server, which makes it significantly cheaper than acquiring a new NAS.

Of course, each situation is different. I'm merely trying to break the "SME backup storage equals low end NAS" mentality, and make sure all Veeam partners consider general-purpose server first and foremost. I mean, you really want a standalone server regardless, in order to run Veeam itself. For example, running a backup server as a VM on the hypervisor host it is protecting is never a great idea.