Standalone backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
Frohn
Novice
Posts: 9
Liked: 3 times
Joined: Oct 17, 2018 6:13 pm
Full Name: Christian Petersen
Contact:

SQL failover cluster, backup issue

Post by Frohn » Sep 13, 2019 8:55 am

Hello

We have been using Veeam for about two years now and so fare we like the software very much.

We do however have enormous issues with Veeam when we are trying to do a backup of our physical SQL cluster (failover).
The job fails with the following error:

Code: Select all

Error: The system cannot find the file specified.
Failed to open I/O device
Failed to open emulated disk.
Failed to open disk for read.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
The error is triggered because of the Volsnap error 25:

Code: Select all

The shadow copies of volume L: were deleted because the shadow copy storage could not grow in time.  Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.
This error is only happing on server 1.

we created a support case (case ID: 03607055) and got a few points and possible work arounds.

The two servers are "twins" so to speak and the volumes are as follow:

Server 1 (The one that has the Vonsnap error)
C:\ - 156 GB (OS Win 2012 fully patched)
D:\ - 2,4 TB (High Prio DB's) (Free space 26%)
L:\ - 2,5 TB (High Prio DB's logs) (Free space 41%)
S:\ - 17,2 TB (Low Prio DB's) (Free space 27%)
X:\ - 5,4 TB (Low Prio DB's logs) (Free space 44%)

Server 2
C:\ - 156 GB (OS Win 2012 fully patched)
D:\ - 2,4 TB (High Prio DB's) (Free space 37%)
L:\ - 2,5 TB (High Prio DB's logs) (Free space 79%)
S:\ - 17,2 TB (Low Prio DB's) (Free space 16%)
X:\ - 5,4 TB (Low Prio DB's logs) (Free space 44%)

The Veeam backupjob has been configured as a SQL failover job - https://www.veeam.com/kb2463

Backup Mode: Entire computer
When we run the job (around 500-700 MB/s), the Volsnap error occurs on the L:\ after about 8-9 hours. The Veeam backup job is handling the volumes in this order: C:\ -> D:\ -> S:\ -> L:\ -> X:\.
As you can read the Veeam job is not processing the volumes in an alphabetic order. If the job could handle the volumes in alphabetic order, then our problem would be resolved. This how is not possible according to Veeam support.

Backup mode: Volume
We then tried to use volume backup instead, first try was just D, L, S, X - But the Volsnap happened again.
We then decided to split the job in two (as instructed by Veeam Support). This however resolves the issue with volsnap, but introduces another issue with the SQL log backup.
The first job is processing D:\ and L:\ the second job is processing S:\ and X:\. - It's the second job that fails with the warning:

Code: Select all

Failed to prepare guest for SQL Server transaction log backup Details: Another job is currently backing up SQL transaction logs
We have now opened a support ticket for the SQL log warning (case id: 03761452)

All we want is backup of the SQL DB and the logs (there is about 300 DB's in total)
Has anybody experienced the same challenge? If so, did you resolve it and how?

Best regards
Christian

HannesK
Veeam Software
Posts: 4299
Liked: 540 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: SQL failover cluster, backup issue

Post by HannesK » Sep 16, 2019 6:03 am

Hello,
in general it must work with "entire computer". So I recommend to focus on that setup with support (actually I don't see where support recommended multiple jobs).

The cluster is running on Server1? I guess that his is the reason why you see the error only on Server1.

Best regards,
Hannes

Frohn
Novice
Posts: 9
Liked: 3 times
Joined: Oct 17, 2018 6:13 pm
Full Name: Christian Petersen
Contact:

Re: SQL failover cluster, backup issue

Post by Frohn » Sep 17, 2019 9:42 am

Hello HannesK

Thank you - We do want it to work with "entire computer", but as mentioned the Veeam agent is processing the volumes in an order where it process the largest volume (S:\) is processed as the second one and therefor the volsnap is happing on the L: becasue of the backup time is taking to long.
If we where able to control the order the volumes are processed in, then we would have the backup process the volumes in aliphatic order and the problem would be resolved. Unfortunately the software dosen't give that option.

This was one of the work arounds support suggested:
First, one of the most optimal methods considering such VSS behavior is to split the job into to 2 or n-number of jobs.
It's normal flexible practice when specific large volumes are processed within separate tasks, it might be we can do the same for a volume S:\ by creating a separate volume-level job with only S:\ volume included - https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

It should help unload the main job and its VSS workflow.
Unfortunately, also due to VSS limitations, it's not possible to handle several parallel VSS activities at time (like 2 independent VSS jobs), so these might be scheduled one by one.
Then, anyway, you can easily restore any volumes you want using the volume-recovery or bare-metal recovery;

HannesK
Veeam Software
Posts: 4299
Liked: 540 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: SQL failover cluster, backup issue

Post by HannesK » Sep 17, 2019 11:25 am

Hi,
hmm, if I understood the volsnap error correctly, then it is because the server is running out of disk space on that volume. Correct?

Would it be possible that you move the VSS snapshot area to a different volume where the server has more space? http://itsimple.info/?p=1046

Best regards,
Hannes

Frohn
Novice
Posts: 9
Liked: 3 times
Joined: Oct 17, 2018 6:13 pm
Full Name: Christian Petersen
Contact:

Re: SQL failover cluster, backup issue

Post by Frohn » Sep 17, 2019 4:30 pm 1 person likes this post

Hello HannesK

Yes that is correct. We tried pointing the VVS snapshot to other drives, but the same thing happend. Even though there is 41% free disk space on the L volume.
The next question could be "Why not buy more storage then?". To be fair, buying disk to make room for a VSS snap on a volume with 41% free space is not a good case, and if you consider the only reason to buy more storage is for the backup software to do a backup.

Best regards
Christian

HannesK
Veeam Software
Posts: 4299
Liked: 540 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: SQL failover cluster, backup issue

Post by HannesK » Sep 18, 2019 6:15 am

Hello Christian,
no, it's not about buying more storage :-)

According to your values, you have a lot of free space. So I asked myself why VSS believes that the disk is getting full. Is the disk really getting full during backup?

I just saw that D:\ L:\ and S:\ have different free space. I believed that they are the same shared volumes?

Best regards,
Hannes

HAntonisen
Lurker
Posts: 2
Liked: never
Joined: Sep 19, 2019 5:18 am
Full Name: Henrik
Contact:

Re: SQL failover cluster, backup issue

Post by HAntonisen » Sep 19, 2019 5:25 am

Not that I have any suggestions, just want to say that I am having the same issue on a stand alone SQL server. Backups has been working ok for 1½ years but now we keep getting the same volsnap errors...
Have been working with support to solve it but have until now failed. Not having a backup of our sql server for 10 days means we are out of business if disaster strikes.

I agree that it should not be a solution to buy more diskspace but I have ordered some extra drives in desperation.........

wishr
Veeam Software
Posts: 1334
Liked: 133 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: SQL failover cluster, backup issue

Post by wishr » Sep 19, 2019 9:14 am

Hello Henrik,

Could you please share your support case ID with us?

Thanks in advance

HAntonisen
Lurker
Posts: 2
Liked: never
Joined: Sep 19, 2019 5:18 am
Full Name: Henrik
Contact:

Re: SQL failover cluster, backup issue

Post by HAntonisen » Sep 19, 2019 9:17 am

sure: 03765753
Right now trying only to backup the drive that fails, just to have some backup.

wishr
Veeam Software
Posts: 1334
Liked: 133 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: SQL failover cluster, backup issue

Post by wishr » Sep 19, 2019 9:33 am

Thank you!

I have found that as well as some other articles on the internet. Some of those recommend adjusting the page file to 1.5 times the amount of RAM. Could you please take a look. The issue is definitely infrastructure-related.

Frohn
Novice
Posts: 9
Liked: 3 times
Joined: Oct 17, 2018 6:13 pm
Full Name: Christian Petersen
Contact:

Re: SQL failover cluster, backup issue

Post by Frohn » Sep 19, 2019 8:14 pm

Hello all

Thank you for taking the time to post in this thread.

@HannesK
Thank you for agreeing on the storage :)
We have been monitoring the volume doing backup, and it's not getting full, and we do not any storage warnings either.

No, they do not share volume - They are two servers, that are identical i specs (CPU, RAM, Storage). The rolle is SQL cluster with failover (So if one dies, the other one can keep running from where the otherone left of)

@HAntonisen
Thank you for the insight of your own case and issue - I can say that we are the same situation with our SQL servers. If disaster strikes then.... well yeah, not good.
We are currently using the Ole Hallagrens SQL backup scripts undtil we get the issue resolve with Veeam support - Hope that this can help you sleep a little better.
Link: https://ola.hallengren.com/sql-server-backup.html

@Wishr
Thank you for the link.
We have tried moving the VSS to other drives, but without succes unfortunately.
Some of those recommend adjusting the page file to 1.5 times the amount of RAM <- Can you go into a little more details on this subject?, I could only find this article about, but can't seem to understand how this will help with the issue?
https://blogs.technet.microsoft.com/mot ... ive-guide/

Best regards
Christian

HannesK
Veeam Software
Posts: 4299
Liked: 540 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: SQL failover cluster, backup issue

Post by HannesK » Sep 20, 2019 6:13 am

Good Morning,
No, they do not share volume
just to clarify: we are talking about a SQL always On Availability group cluster?

Best regards,
Hannes

Frohn
Novice
Posts: 9
Liked: 3 times
Joined: Oct 17, 2018 6:13 pm
Full Name: Christian Petersen
Contact:

Re: SQL failover cluster, backup issue

Post by Frohn » Sep 23, 2019 5:28 pm

Hello Hannes

Yes, that is correct - It's an SQL always On Availability group cluster

Frohn
Novice
Posts: 9
Liked: 3 times
Joined: Oct 17, 2018 6:13 pm
Full Name: Christian Petersen
Contact:

Re: SQL failover cluster, backup issue

Post by Frohn » Oct 07, 2019 2:36 pm 1 person likes this post

Thank you all for helping out.

For whomever that has the same issue (i hope not) the solution for this issue was to add an iscsi drive (in my case a synology NAS) and then point the VSS snapshot from the D volume to the iscsi drive.

Post Reply

Who is online

Users browsing this forum: No registered users and 20 guests