Slow performance with Agent backup

[mlott@gie.com]

Running CentOS 6.8 on a vSphere 5.5 VM with 8G RAM and 4 virtual cores. During the backup run, load is hovering around 1 with about 50% memory utilization. Network is E1000.

We are backing up via B&R, which is a on a beefy HP server.

The backup repo is a re-purposed dual quad-core Xeon server with 20G RAM running CentOS 6.3. Load is not much more than zero. Network is dual bonded Intel Corporation 82576.

The backup source is a set of snapshot LUNs mounted via iSCSI on a separate virtual NIC to our EMC VNX. These are a copy of our primary file server LUNs, which we are trying to get to tape.

Processing rate has been hovering around 4MB/s for 15 hours now. I am trying to figure out where the bottleneck is. Per veeam it is the Agent, that being the VM used to mount the snapshots and run the backup job.

My next step is probably to be done with the VM and use the backup target to mount the LUNs so that all of the backup activity is local. If I have enough free ports I might even be tempted to mount those snapshots over FC.

[PTide]

Hi,

During the backup run, load is hovering around 1 with about 50% memory utilization.

Have you tried to get some info on what process/es consumes CPU and RAM?

<...>use the backup target to mount the LUNs so that all of the backup activity is local.

Please be aware that in this case you'll have to unload veeamsnap module manually in order to unmount the snapshot when the backup session ends. Next time you mount the snapshot it will be treated as a new block device thus you'll get full backup instead of incremental.

As I know you have a physical CentOS server connected via FC to SAN, can you try installing VAL onto that fileserver?

Thanks

[mlott@gie.com]

The veaamagent process is using roughly as much as 45% CPU and 44% RAM consistently on the agent server/source.

I don't want to install or run the agent on the file server itself due to the potential load. Also, since I am working with a snapshot on a SAN it shouldn't matter where I mount that as long as the connectivity and available horsepower is sufficient. I suppose I will need to try connecting the backup repo server to the FC switch or at least allow it connection to iSCSI. We should have at least one port available for FC. That way I can remove the VM from the equation.

veeam tries to create a snapshot, and probably does so for the local machine which is using it but NOT for the volumes we are trying to protect which do not use LVM. We create the snapshot on the VNX using our own script. We plan to call this script before the job. That worked well even called remotely from B&R when we were trying to do file to tape. I can then call a script to unmount and remove it post-backup.

We want to do a full backup once per month and then copy that to tape. Our incrementals are handled separately at this point. However, once we get enough reliability out of using Veeam for this we should be able to that here instead. That would be a relief.

[PTide]

I don't want to install or run the agent on the file server itself due to the potential load.

You can lower agent's priority in /etc/veeam/veeam.ini config file, should you decide to give it a try.

veeam tries to create a snapshot, and probably does so for the local machine which is using it but NOT for the volumes we are trying to protect which do not use LVM.

Veeam does not use LVM snapshots, it uses its own snapshot mechanism instead.

Kindly collect and provide logs from Veeam Agent (use 'M' button on the main screen) so our dev-team can do some research on what's going on and why does source agent consume that much of resources.

Thank you

[mlott@gie.com]

At this point I have moved on and have the snapshot mounted on the same server that will be the repo. The job is configured and running at 37MB/s. Much better.

However, there is a new issue as always. There are currently 5 active snapshots per LUN, only one of which is to be concerned with this backup. Due to required config on the VNX, this server sees all of the snapshots. When I select the folder where the snapshot set I want is mounted, veeam decides that it needs to backup another LUN for a snap mounted elsewhere. It went through that small LUN and considered itself done. I suppose I will have to select the LUNs I really want instead.

[mlott@gie.com]

That worked. Moving along at around 40MB/s.

[mlott@gie.com]

Twice, the backup crashed the server running the backup after a few hours causing it to reboot. I was able to switch this to backup the folder where they are mounted using veeamconfig to create a file backup including only the folder where the LUNs are mounted. It is still early but seems to work now. The GUI doesn't show any progress but the backup file size is growing.

[mlott@gie.com]

New case 02081728...

We have gone through many different configurations. Again, I have an older 32TB CentOS 6 server with FC access to the SAN acting as a repo and NFS server. This is a Supermicro board with dual quad-core Xeon, 20G RAM, a 3ware card with a RAID6 array, and a single port 8GB FC card with 4 paths to the SAN. OS is installed on mirrored SSD. Network is 1G but this particular box has a bonded connection of 2 cards.

1. I mounted the snapshots from the SAN and installed the agent locally. I ran volume a level backup and speed was close to 40MB/s. The server crashes and reboots after a few hours.
2. Running a file level backup including the folder of mounted snapshots works but at best 4MB/s. This is true whether using the same server as local storage or through B&R as a repo.
3. We setup a CentOS 7 VM and mounted the snapshots there. Running a backup to the CentOS 6 box whether as a repo or via local NFS mount tops out around 4MB/s.

It's really odd that the backup will progress in bursts and then sit for hours with no apparent disk activity, i.e. the backup file size and time stamp remain static. Since the source files should be static I would expect it to only have to get a file list once. So, I don't know what is causing it to stall. CPU and memory utilization are never particularly high on the repo or the agent machine.

[mlott@gie.com]

FYI, I received a new version of the agent which seems to be stable after a couple of days performing a volume backup. Speed is around 26MB/s currently but it has been as high as 40.