Target refused global dedupe session?

[Dave-Departed]

Hi All,

Just checking through our job logs this morning, as usual, and I see something in one of our backup copy job logs which says:

08/10/2013 06:30:51 :: Target refused global dedupe session because of growth failing. Disable global dedupe.

Never seen this before. Any idea what it means, and why it's happening?

Thanks in advance.

Dave

[veremin]

Hi, Dave. Looks like you don’t have enough space at target location. This prevents target cache from growing, and, hence, there is a corresponding error. The issue should be resolved by allocating additional space. Thanks.

[Dave-Departed]

Hi Vladimir,

No more space to allocate, unfortunately! So this leads me to a question...

Our normal onsite backup jobs are set to keep 90 restore points, and this only uses up 5TB in total on our main onsite backup storage. Yet, our offsite storage has 3TB and we have only set our backup copy jobs to keep 7 restore points, and they have almost used up all of the 3TB available! I would've thought that using WAN accelerator and compression, the backup copy job files would be even smaller than the original backup job files which they are created from? Why are our backup copy job files taking up so much space?

Thanks

Dave

[veremin]

Hmm, then, I’m wondering what the size of target cache is. Thanks.

[Dave-Departed]

We have 2 remote WAN accelerators configured, both with 100GB limits set.

The first is currently at 203GB(?! not really sure why/how?) and the second is at 94GB.

[Dave-Departed]

And if I look at the same day for a backup job and a backup copy job, I see this:

Backup job file: 12GB
Backup copy job file: 21GB

Any idea why this would happen?

[foggy]

Our normal onsite backup jobs are set to keep 90 restore points, and this only uses up 5TB in total on our main onsite backup storage. Yet, our offsite storage has 3TB and we have only set our backup copy jobs to keep 7 restore points, and they have almost used up all of the 3TB available! I would've thought that using WAN accelerator and compression, the backup copy job files would be even smaller than the original backup job files which they are created from? Why are our backup copy job files taking up so much space?

Dave, I would say it is expected. Backups produced by backup copy jobs are just ordinary backup files, similar to those produced by ordinary backup jobs. WAN acceleration is used for data transfer over the slow link, not for storing data on the repository. Those blocks that are already in cache and are not required to be transferred are read from the cache and written to the corresponding restore point file, so the sizes of backups for both types of jobs should be comparable. Assuming you're using reversed incremental mode in your ordinary backup jobs, full backup is what takes most of the space on both backup repositories and the rest of the space is occupied by incrementals (90 and 7 respectively), which explains the difference in sizes backups take on your repositories (5TB vs 3TB).

[veremin]

Any idea why this would happen?

What schedule does the backup job have and what synchronization interval is set for backup cop job? Thanks.

[Dave-Departed]

Normal backup job runs at 21:00 and takes around 2.5hrs

Backup copy job runs continuously, and sync interval is set at every 1 day, starting at 12:00

[Vitaliy S.]

Might be a silly question - do you have the same set of VMs being backed up and transferred by your backup copy job?

[Dave-Departed]

Yes, all of our backup copy jobs are mirrors of our normal backup jobs

[foggy]

And if I look at the same day for a backup job and a backup copy job, I see this:

Backup job file: 12GB
Backup copy job file: 21GB

Any idea why this would happen?

Is it also typical for other days?

[Dave-Departed]

Is it also typical for other days?

Yes, give or take.

[foggy]

Should not be the case, I suggest asking support for assistance.

[Gostev]

We have 2 remote WAN accelerators configured, both with 100GB limits set.
The first is currently at 203GB(?! not really sure why/how?) and the second is at 94GB.

Does the first one have more than one WAN accelerator partner? Remember there is a separate cache for each WAN accelerator pair.

As far as the difference in backup sizes, sounds like there are different compression levels on source backup job and backup copy job. Or, target repository has an option to decompress before storing enabled. That is, assuming we are comparing apples to apples: does the source backup job use forward incremental backup mode, and are you comparing the size of incremental backup files (VIB)?

[Dave-Departed]

Does the first one have more than one WAN accelerator partner? Remember there is a separate cache for each WAN accelerator pair.

As far as the difference in backup sizes, sounds like there are different compression levels on source backup job and backup copy job. Or, target repository has an option to decompress before storing enabled. That is, assuming we are comparing apples to apples: does the source backup job use forward incremental backup mode, and are you comparing the size of incremental backup files (VIB)?

Ahh ok, now it begins to make sense! Yes, the first accelerator has more than one partner (although we eventually separated it out into the 2 accelerators at each end, as I mentioned above somewhere)... So that explains the large cache size!

As for the backup size difference, it looks like we are comparing apples to pears, then... Our regular backup jobs are reverse incrementals (we used to do D2D2T, and we just left them that way when we upgraded to v7) with compression set to high (again, default due to in-place upgrade, as I understand), and our backup copy jobs are forward incrementals, obviously using the 'new' compression and set to 'auto'. I may set the backup copy jobs to 'high', because space is more important to us than using additional CPU resources right now.

Cheers!

Dave

[foggy]

High provides only 10% of additional compression ratio over Optimal level, so do not expect the sizes to be equal after applying new level. 2x difference is mostly explained by different backup modes, I suppose.

[Dave-Departed]

I came in this morning to see this message, and wondered if it's related to the above?

Code: Select all

11/10/2013 07:28:52 :: Processing '<VMname>' Error: An existing connection was forcibly closed by the remote host
--tr:Cannot read data from the socket. Requested data size: [1048552].
--tr:Receive file failed
--tr:Wan connection failed to run command.
--tr:Wan connection failed to get target file '{d03e1a8e-cf4b-48a8-83d3-3e508f8dbf3a}'
--tr:Sync global cache failed.
--tr:Disk processor failed to init global dedup.
--tr:Disk processor failed to process disk.
--tr:Sync task failed to process disk.Arguments:

[Vitaliy S.]

Looks like it is just an intermittent connection drop, which is not related to the original issue.