Problems after Veeam 7.0 patch 4 update

[r.mckeon]

I have a Veeam B&R installed on a Windows Server 2008 VM, running on and backing up VMs on two VMware vSphere 5.5 hosts. After upgrading my Veeam 7.0 to patch 4, I've been having some problems (surprise, surprise...a patch has created more problems than it solved). As soon as I patched the Veeam, I've been having the following problems during the successive backups:

Firstly, I'm getting the following errors in my Windows event log (during the times that the backups are running). I NEVER got them before, so after months of using this particular Veeam system I'm assuming it's due to the patch update:

source: PlugPlayManager EventID: 12 Type: Error
The device 'VMware Virtual disk SCSI Disk Device' (SCSI\Disk&Ven_VMware&Prod_Virtual_disk&Rev_1.0\4&1588251b&0&000600) disappeared from the system without first being prepared for removal.

Secondly, (might be related to the error), the performance of the Replication jobs has decreased dramatically. The backup jobs have remained the same, but the replication jobs are now approximately 3x slower! Before they were around 21MB/s processing rate (still slow but Veeam support couldn't help me out), but since the patch update they are now approx. 7MB/s processing rate.

Is this a known issue?

[r.mckeon]

My current set up is this: Veeam 7.0 B&R on a Windows server 2008 VM, backing up 3 VMs spread over 2 ESXi 5.5 hosts. I take a backup job of each VM, and as soon as they're done a replication job of each to a different location.

Now usually, as you can imagine, if say the Exchange VM's backup job processed 10GB of data, then the Exchange VM's replication job will also process approx. 10GB of data as the two jobs only about 1 or 2 hours apart. This is even more true for the file server and SQL server as the backups/replications happen during non-office hours so nobody's adding or changing any data, it's only metadata which is changing.

For months this setup has produced the same results: if for example 7.5GB are processed by a VM's backup job, then 7.5GB are processed by the same VM's replication job.

But after updating to patch 4, the backup job of the Exchange Server VM processed 7.5GB of data, but the replication job processed 19.5GB. That almost 3 x more the amount of data after a span of around an hour and a half (backup job started at 21:00 and replication job started at 22:30). No changes were made on the Exchange server, especially not during that time spam. And I sincerely doubt that amount of data in emails could have come in, especially considering this set up has produced the same results for months.

Also, the same thing happened for the file server VM jobs and the SQL server VM jobs, the data processed by the replication jobs was triple that processed by the backup jobs.

Could this be a patch issue? Or is there something I can check on the Veeam B&R?

[Gostev]

The first message is not specific to any version, if you search these forums you will find existing topics mentioning this message, spanning different B&R versions. This can be ignored as virtual disks are mounted read-only, and don't need to be prepared for removal, like USB sticks do for instance.

Second issue is not known and is unlikely to have anything to deal with patch #4, as the patch is pretty old by now and have had tens of thousands of downloads already. While with our customer base size, any real issue always results in a few pages long topic within a few days of patch release.

Please include support case ID for this issue here if you'd like to keep this topic open, as requested when you click New Topic. Also, please don't create multiple topics about the same problem.

[r.mckeon]

It's not the same problem, in fact I opened different topics since in my view they might be separate issues. There are 3 problems in all (different problems, not the same).

1. The Virtual disk error

2. The slower speed

3. The difference in processed data between backup jobs and replications jobs of the same VM

[Gostev]

I was referring to the replication problem here. One post says replication became 3x slower than backup, and another one says replication started transferring 3x more data than backup. How is this not the same problem? It takes 3x more time to transfer 3x more data...

I personally would just keep monitoring both jobs to see if the issue persists the following days. In which case, I honestly will have no explanation to this behavior, because both jobs are using the same CBT data. So the only plausible explanation I currently have, is some Exchange database maintenance job that happened right in between your backup and replication jobs. The following job runs should basically either prove, or bust this idea. Thanks!

[r.mckeon]

Thanks for your reply Gostev. Maybe we misunderstood eachother but let me elaborate...

I'm talking about two different counters here: "Processing Rate" and "Transferred Data".

Now the difference in Transferred Data between a replication job and a backup job of the same VM only 1 hour 30 mins apart is 3 times higher for the replication job, and this happens for ALL VMs, ALL jobs. Very, very strange and I cannot understand why it's happening, so the reason I assumed it was due to patch 4 was that I've had this set up for a few months now and it never happened (I check Veeam history every day), but it started right after I upgraded to patch 4 and has remained the same since.

As for the Processing Rate being approx. 3x slower, even if the transferred data were to increase as it did, it shouldn't affect the processing rate itself...it should increase the Duration but not the Processing Rate, right? That's why I see it as being unrelated, but I may be mistaken.

[r.mckeon]

Actually I found one of the issues! Compression rate...

In the counter of Transferred Data it looks something like this: 17.5GB (1.1x)

I assume the value in the brackets is compression and deduplication rate? Anyway, that value is in fact 3 times less in the replication jobs than the backup jobs, whereas before it was always equal or very similar. Do you have any suggestions on where I can start checking to see why the compression has gone down so much?

[Gostev]

Check Advanced settings for the replication job and see what traffic compression is selected. Indeed, Transferred should normally be at least 2x less than Processed because of compression.

[r.mckeon]

Traffic compression is set to Optimal (recommended), but it's been like that since the very beginning. Only since a few days ago has the compression rate reduced, and for the first time it's very different from the backup jobs.

[foggy]

Have you also checked the backup job compression settings? How about the Read counter for both cases?

[r.mckeon]

Backup job compression settings are identical. I'm listing the statistics of one of the VMs as examples. I took samples from before the patch and from after the patch, to show what I'm talking about:


Before patch:

VM1 Backup job:
Processing rate 7MB/s
Load: Source 15% > Proxy 10% > Network 0% > Target 99%
Read: 17.7GB
Transferred: 5.2GB (3.4x)

VM1 Replication job (runs almost right after backup job):
Processing rate 22MB/s
Load: Source 95% > Proxy 30% > Network 9% > Target 21%
Read: 17.3GB
Transferred: 5.0GB (3.5x)


After patch:

VM1 Backup job:
Processing rate 7MB/s
Load: Source 15% > Proxy 10% > Network 49% > Target 72%
Read: 16.9GB
Transferred: 5.1GB (3.3x)

VM1 Replication job (runs almost right after backup job):
Processing rate 7MB/s
Load: Source 21% > Proxy 12% > Network 85% > Target 94%
Read: 16.7GB
Transferred: 15.8GB (1.1x)

[foggy]

Are you observing this behavior consistently after applying the patch (how many job runs have you performed since applying it)? Have you already contacted support with that?

[r.mckeon]

Every backup/replication run since the patch update (last Friday) has produced the same results. I have not contacted support yet...I will open a ticket with this information.

[r.mckeon]

Case # 00616816

[foggy]

Thanks for providing the case ID. Let's see what our engineers can say based on your logs.

Btw, didn't you notice some other changes in your setup? Is the transport mode used to populate the target datastore the same as before? Are the same proxy servers used for replication (do you have proxies on both ends?)?

[r.mckeon]

Yes the transport mode was one of the first things I checked...still the same as before. Nothing other than compression rate seems to have changed (and obviously the processing rate, load and duration as a by-product of that).

As for the proxy, nothing changed on that either. There's only one proxy as the whole setup, including destination locations, are on the same network in the same location, and there's only one Veeam server.