What happens if copy job full backup is interrupted?

[TitaniumCoder477]

What happens if the copy job's full backup is interrupted due to a server restart or even a periodic copy (pruning) cycle that is insufficient to encompass the completion of the full backup in one session? Does the copy job just "pick up where it left off" or does it start from scratch? I have the same question for incremental backups, though the chances of them being interrupted is obviously less than a full backup due to less time needed.

Thanks!

[foggy]

Hi James, it will pick up from where it left off and resume copying data normally.

[vmtech123]

It depends of your definition of picking up where it left off.

If it fails mid job, it has to start the job over. "Picking up from where it left off" to me makes me think of you replicated for 11.9 hours and had 2 minutes left, the server rebooted and you assume there is 2 minutes left.

Veeam will retry the last job again, and not require previous jobs to be re run, but in the above case, you will be waiting the whole 12 hours for the job again as well.

[foggy]

I mean it will copy only the new data required to build the new restore point corresponding to the VM latest state and will not re-copy everything that is already there.

[TitaniumCoder477]

It depends of your definition of picking up where it left off.

If it fails mid job, it has to start the job over. "Picking up from where it left off" to me makes me think of you replicated for 11.9 hours and had 2 minutes left, the server rebooted and you assume there is 2 minutes left.

Veeam will retry the last job again, and not require previous jobs to be re run, but in the above case, you will be waiting the whole 12 hours for the job again as well.

This is indeed what I was talking about. Very disheartening. I thought the copy job was designed to be robust, and starting over from scratch does not qualify in my book. If it's already got the blocks over there, it should not send them again over the pipe. Ok, I guess that means I have to open the sync cycle to like 5 days to get the 5TB across the pipe in one session.

[vmtech123]

Foggy correct me if I am wrong, but if a copy job fails, it requires everything from the previous restore point right?

TitaniumCoder, think of it like this. If you copy a huge file in windows and it fails in the middle of the file transfer, you don't have half the file on the other side and need to only copy half.

Or same with downloads. There are programs that allow this, but the confirmations and error checking will take a ton of CPU and resources effecting performance as well.

The Veeam copyjob is very robust in my opinion and works amazing. Once you are seeded the incremental should be smaller to keep things in sync.

Veeam will max out my 10 GB connection if I let it, perhaps the issue is you need a bigger pipe. I replicate plenty of 30 TB VM's to a secondary site. After they are synced up my backup windows have been fine.

[Moebius]

I don't want to hijack this thread, but I also have a question that is loosely related to the OP's.

I had to disable a backup copy job because a user copied some 4TB of garbage to the server being backed up. The primary backup repo could bear the load, the backup copy repository couldn't. The backup copy failed when the repository went out of space in mid-run.
I am going to re-enable the backup copy job after several weeks, since the garbage data have been deleted.
What will happen to the backup copy? Will it resume normally as before the incident? What will happen to the garbage data that have already been partially included in the backup copy?

[mweissen13]

Especially when backing up huge VMs via mediocre internet connection this is a real problem. I am amazed that Veeam still has no real solution for that, besides using a portable disk for the first full backup (which can be hard to do if you want to attach the disk to "the cloud").

[vmtech123]

I don't want to hijack this thread, but I also have a question that is loosely related to the OP's.

I had to disable a backup copy job because a user copied some 4TB of garbage to the server being backed up. The primary backup repo could bear the load, the backup copy repository couldn't. The backup copy failed when the repository went out of space in mid-run.
I am going to re-enable the backup copy job after several weeks, since the garbage data have been deleted.
What will happen to the backup copy? Will it resume normally as before the incident? What will happen to the garbage data that have already been partially included in the backup copy?

If the data is on the primary SAN, it will be replicated to the secondary. That is how it works. You cant "skip" an incremental backup and expect things are going to work.
Veeam is acting as designed. If I have a user copy 10TB of data to a folder, It gets backed up. That isn't a Veeam issue. If they fill the drive and it goes offline, I wouldn't blame Microsoft for that either.

You could lower the retention way down to like a day, start fresh with an active full, but to me, if 4TB in a day is going to blow up your secondary, you need more storage in your secondary SAN. This is more of a design issue than a "Veeam" issue.

[vmtech123]

Especially when backing up huge VMs via mediocre internet connection this is a real problem. I am amazed that Veeam still has no real solution for that, besides using a portable disk for the first full backup (which can be hard to do if you want to attach the disk to "the cloud").

Your amazed Veeam doesn't have a solution for your mediocre internet connection?
If I need to copy 100TB, and only have a 100MB or 1000 connection, how is that a Veeam problem?

What do you propose the solution is? Veeam will use the hardware given to it and the underlying infrastructure. I can max out my 16GB fiber, and 10GB eth. Yes, replication of a 100TB VM is going to take a few days, but short of this, or a portable backup taken to a DR site I'd really like to know how to get the data there.

[Moebius]

If the data is on the primary SAN, it will be replicated to the secondary. That is how it works. You cant "skip" an incremental backup and expect things are going to work.
Veeam is acting as designed. If I have a user copy 10TB of data to a folder, It gets backed up. That isn't a Veeam issue. If they fill the drive and it goes offline, I wouldn't blame Microsoft for that either.

You could lower the retention way down to like a day, start fresh with an active full, but to me, if 4TB in a day is going to blow up your secondary, you need more storage in your secondary SAN. This is more of a design issue than a "Veeam" issue.

I must have expressed myself very badly. I am not blaming anyone, neither Veeam nor Microsoft nor anyone else. I was just asking for advice.
The 4TB were copied in the production data by mistake, were (obviously) backed up by the primary backup job, and were unsuccessfully attempted to be copied by the backup copy job.
I suspended the backup copy job, removed the 4TB from the production data (looks like you missed that part), let the primary backup job run some more times and then re-enabled the backup copy job.
All jobs are running happily now.

[mweissen13]

Your amazed Veeam doesn't have a solution for your mediocre internet connection?
If I need to copy 100TB, and only have a 100MB or 1000 connection, how is that a Veeam problem?

What do you propose the solution is? Veeam will use the hardware given to it and the underlying infrastructure. I can max out my 16GB fiber, and 10GB eth. Yes, replication of a 100TB VM is going to take a few days, but short of this, or a portable backup taken to a DR site I'd really like to know how to get the data there.

I am not talking about the physical limitations of the connections. Sure, Veeam can't solve that. But I am talking about the fact that it can happen anytime that a connection breaks intermittently. Especially when leveraging an internet connection for cloud backups.
What I am cirtisizing is that Veeam is not able to continue to copy where it "left off" but always starts from scratch. That's very frustrating when you have been copying for 10 days and at 99% something happens and you can wait 10 days again before the whole backups is moved across the line.

[vmtech123]

I understand.

This comes down to getting a solid internet connection, and fast. Similar to having a secondary site, it's not much good if you cant reach it, or if it's too slow to access.

It is common practice to use 2 ISP connections, from different providers, and also multiple cloud providers etc. Going fully redundant DR is not a cheap solution by any means.

As far as the file transfer, I suppose Veeam COULD implement this, but it would change the way the program needs to work, and cause a ton of I/O and checkpoints as it copies data. There are also repercussions as to what happens if your link goes down, Veeam will start backing up a job, but would be able to merge the file due to the fact it is in the middle of a copy job. Do you want your Primary REPO to fill up also? The only way this could work would be if it keeps doing incremental and never merging until the copy job completes. If you are using reverse incremental it wouldn't work at all I'd assume.

As someone who copies very large VM's (40+TB) I get what you are saying. That first sync takes a while, but after that it seems to go pretty good. If you are to copy a 40+TB file in Windows over the network and it fails after 2 days you are in the same boat.

[maanlicht]

In the organization I work for we has to deal with many sites in Africa. Getting the expensive bandwidth is one thing, latency and line stability is a whole other thing and often just impossible. We have been searching for ways to transport data from Africa to Europe and tested many different technologies. Veeam is BY FAR the best tool out there to deal with poor bandwidth and high latency. No matter the latency we can always max out my lines.

However stability is a real problem with Veeam. For that initial backup to get trough.. I sometimes have to wait for months for the planets to align to create a stable window large enough for it to get trough. Every time it restarts. Bandwidth-wise it would only take a couple of days.

BackupCopy jobs do tolerate micro outages to a degree. But if I remember correctly it only tolerates about 5 or 6 before it fails. Would be nice to have the ability to increase this value.
For now I have to resort to using BITS to transport those initial copys or even shipping USB disks. I am sure Veeam could do that much more efficiently if the DEVs focused on it.