Need help designing backup jobs (Tape and Data Domain)

[jeremyve]

I am looking for recommendations for my backup job settings as I am a little overwhelmed with the options I have. I am having issues where my Production_RAM job runs from Wednesday night until Friday morning when it's doing it's active full so I miss Thursday evening backups all together. I then also have the issue of my Friday evening backup (Production_TAPE) running 25 hours and then the (Production_RAM) doesn't run on Fridays so I don't have backups to the DD6200 for that day nor do I have it replicated to the other site. Plus I only have one week of retention due to it being on local disk.

FYI I had issues with slow copy jobs to tape when trying to copy from the Data Domain which is why I have a separate job.

Overview
TS3100 Tape Library with one LTO6 tape drive for VEEAM
EMC DD6200 Data Domain in Primary Site 15.7TB Usable Space (Replication is occurring between the data domains)
EMC DD6200 DR Site 15.7TB Usable Space
HP Proliant DL380 Gen8 with 29TB of local disk

What I have been doing:
Job Title: Production_RAM
Backup Repository: EMC DD6200 Data Domain in Primary Site
Restore Points to Keep: 60
Backup Mode: Incremental
Active Full Backup: Weekly on Wednesday
Schedule: Every Day at 9:00 PM
Runtime: Incremental-12 Hours, Full-36 Hours

Job Title: Production_TAPE
Backup Repository: HP Proliant DL380 Gen8 Local Disk
Restore Points to Keep: 1
Backup Mode: Incremental (Really never an incremental as I run it only on the same day I have the active full backup running)
Active Full Backup: Weekly on Friday
Schedule: Weekly on Friday at 6:00 PM
Secondary Target: Backup to Tape (TS3100 Tape Library)
Runtime: 25.5 Hours to the local disk, 15 Hours to Tape

So to sum it all up I would say I need my jobs designed so that I get a backup every evening no matter what day of the week and possibly decrease run times.

[jeremyve]

I should also mention I have 3 VEEAM Backup Proxies that do that Hot-Add for the backups.

[veremin]

So, the first issue to address would be not job redesign, but rather full backup taking a lot of time. Can you tell me how many VMs are present inside backup job? What component is identified as a major bottleneck during full backup cycle? Is hot add mode indeed used during backup jobs (look for special [hot-add] metric in the corresponding job session statistics). Thanks.

[jeremyve]

Hi v.Eremin, you might be right about just reducing the time it takes for the full backup to complete. I currently have 104 virtual machines backing up using hot-add through the veeam virtual proxies to the DD6200 using DDBOOST. I've removed the server names for security sake.

Code: Select all

Backup job: Production_RAM (Full)
at 5/1/2014 7:06:46 AM.Error
104 of 104 VMs processed 
Wednesday, September 9, 2015 9:00:05 PM
Success97Start time9:00:05 PMTotal size15.8 TBBackup size11.0 TB
Warning2End time5:10:44 AM (+2)Data read11.3 TBDedupe1.4x
Error5Duration32:10:38 Transferred10.4 TBCompression1.0x
Details 
NameStatusStart timeEnd timeSizeReadTransferredDurationDetails
Success9:06:31 PM9:29:38 PM60.0 GB59.1 GB59.1 GB0:23:07  
Success9:06:31 PM10:26:11 PM300.0 GB165.6 GB150.9 GB1:19:40  
Success9:12:07 PM3:14:46 AM (+1)1.2 TB1018.0 GB1014.0 GB6:02:38  
Success9:27:52 PM10:26:36 PM50.0 GB40.8 GB40.8 GB0:58:44  
Success9:28:02 PM10:36:43 PM60.0 GB44.0 GB41.3 GB1:08:41  
Success10:01:20 PM2:28:25 AM (+1)810.0 GB756.1 GB738.6 GB4:27:04  
Success10:24:01 PM2:17:58 AM (+1)60.0 GB25.6 GB24.0 GB3:53:56  
Error10:24:07 PM1:15:31 AM (+1)0.0 KB0.0 KB0.0 KB2:51:24 
Success12:05:34 AM2:47:19 AM75.0 GB25.4 GB22.8 GB2:41:45  
Success12:14:42 AM3:35:33 AM130.0 GB128.2 GB111.2 GB3:20:51  
Success2:08:20 AM8:14:20 AM860.0 GB488.1 GB463.9 GB6:05:59  
Success2:17:53 AM3:59:04 AM150.0 GB142.2 GB102.8 GB1:41:10  
Success2:28:16 AM4:04:43 AM67.5 GB39.7 GB36.0 GB1:36:26  
Success2:47:31 AM4:24:06 AM60.0 GB59.0 GB59.0 GB1:36:35  
Success3:24:22 AM4:38:30 AM60.0 GB45.8 GB45.2 GB1:14:08  
Success3:26:58 AM5:43:24 AM120.0 GB51.0 GB43.8 GB2:16:25  
Success3:38:11 AM5:32:02 AM50.0 GB48.0 GB27.3 GB1:53:51  
Success3:59:07 AM3:54:40 PM1.5 TB1.2 TB1.1 TB11:55:32  
Success4:37:22 AM9:03:12 AM60.0 GB20.2 GB18.7 GB4:25:50  
Success4:37:22 AM9:02:25 AM60.0 GB33.2 GB31.0 GB4:25:02  
Success5:18:39 AM9:02:53 AM250.0 GB38.8 GB38.7 GB3:44:14  
Success7:38:20 AM9:35:29 AM74.5 GB70.6 GB66.8 GB1:57:09  
Success8:02:27 AM9:52:05 AM100.0 GB96.0 GB67.1 GB1:49:37  
Error8:13:57 AM9:36:58 AM67.4 GB67.4 GB64.1 GB1:23:01 
Success8:36:59 AM10:01:41 AM50.0 GB37.5 GB37.5 GB1:24:42  
Success9:03:02 AM10:02:10 AM60.0 GB26.4 GB25.0 GB0:59:08  
Success9:10:54 AM10:20:04 AM120.0 GB51.8 GB44.3 GB1:09:09  
Success9:35:18 AM10:41:39 AM90.0 GB71.3 GB69.7 GB1:06:21  
Success9:51:52 AM11:04:09 AM120.0 GB88.2 GB60.3 GB1:12:16  
Success9:53:08 AM10:59:07 AM120.0 GB83.5 GB58.0 GB1:05:58  
Success9:53:29 AM11:04:28 AM100.0 GB92.0 GB92.0 GB1:10:59  
Success10:01:16 AM11:18:06 AM50.0 GB35.6 GB33.6 GB1:16:50  
Success10:13:04 AM12:36:12 PM136.7 GB134.7 GB134.3 GB2:23:07  
Success10:19:47 AM11:32:15 AM100.0 GB37.0 GB33.3 GB1:12:27  
Success10:42:58 AM11:35:11 AM120.0 GB28.1 GB25.8 GB0:52:12  
Success11:04:19 AM1:22:13 AM (+1)2.1 TB1.8 TB1.7 TB14:17:53  
Success11:17:02 AM11:36:35 AM18.0 GB8.1 GB8.1 GB0:19:32  
Error11:17:17 AM12:34:25 PM224.6 GB221.4 GB196.7 GB1:17:08 
Success11:32:01 AM12:33:26 PM9.6 GB9.5 GB1.3 GB1:01:24  
Success11:35:03 AM1:40:35 PM70.0 GB51.7 GB46.2 GB2:05:32  
Success11:45:31 AM1:40:39 PM120.0 GB55.8 GB55.7 GB1:55:07  
Success12:19:31 PM1:40:36 PM60.0 GB24.3 GB21.4 GB1:21:05  
Success12:33:15 PM2:01:32 PM60.0 GB33.2 GB21.8 GB1:28:17  
Success12:33:15 PM1:40:08 PM60.0 GB30.2 GB28.7 GB1:06:53  
Success1:04:36 PM2:29:20 PM100.0 GB96.0 GB68.1 GB1:24:43  
Success1:04:36 PM2:01:57 PM50.0 GB31.6 GB29.8 GB0:57:20  
Success1:27:20 PM2:34:36 PM30.0 GB24.4 GB24.4 GB1:07:15  
Success1:27:31 PM2:49:39 PM64.0 GB61.3 GB58.0 GB1:22:08  
Success1:40:00 PM2:34:15 PM160.0 GB22.0 GB20.4 GB0:54:14  
Success2:21:54 PM3:03:44 PM20.0 GB16.0 GB2.4 GB0:41:50  
Success2:28:56 PM3:23:02 PM80.0 GB42.5 GB40.8 GB0:54:05  
Success2:34:08 PM4:56:52 PM380.0 GB282.4 GB271.6 GB2:22:43  
Error2:49:23 PM3:44:04 PM160.0 GB61.8 GB59.4 GB0:54:40 
Success3:23:33 PM4:57:22 PM60.0 GB21.7 GB19.1 GB1:33:49  
Success3:42:22 PM5:03:46 PM100.0 GB96.0 GB62.4 GB1:21:23  
Success3:42:48 PM5:47:46 PM110.0 GB75.0 GB73.7 GB2:04:58  
Success3:42:48 PM5:02:13 PM100.0 GB20.9 GB19.7 GB1:19:25  
Success4:18:07 PM5:35:47 PM50.0 GB19.0 GB19.0 GB1:17:40  
Success4:56:37 PM5:36:08 PM110.0 GB42.7 GB39.9 GB0:39:30  
Success4:56:47 PM5:43:50 PM30.0 GB22.7 GB22.7 GB0:47:03  
Success5:31:56 PM7:14:04 PM260.0 GB96.1 GB88.1 GB1:42:08  
Success5:32:16 PM7:04:06 PM200.0 GB86.5 GB86.5 GB1:31:50  
Success5:33:17 PM7:14:47 PM60.0 GB31.8 GB29.1 GB1:41:30  
Success5:33:42 PM7:14:27 PM30.0 GB23.7 GB23.1 GB1:40:44  
Success5:44:06 PM8:53:56 PM30.0 GB21.4 GB21.4 GB3:09:50  
Success6:51:37 PM8:58:35 PM30.0 GB28.8 GB28.0 GB2:06:57  
Success6:51:38 PM8:57:37 PM70.0 GB62.4 GB62.4 GB2:05:59  
Success6:51:58 PM8:58:02 PM60.0 GB28.8 GB26.9 GB2:06:03  
Success7:02:10 PM9:14:43 PM150.0 GB134.6 GB123.5 GB2:12:32  
Warning7:14:19 PM9:56:20 PM60.0 GB20.6 GB20.6 GB2:42:00 
Success7:41:47 PM10:38:25 PM50.0 GB46.0 GB46.0 GB2:56:38  
Success8:53:22 PM10:38:45 PM120.0 GB117.0 GB71.0 GB1:45:23  
Success8:53:42 PM10:48:59 PM120.0 GB58.1 GB55.8 GB1:55:16  
Success10:14:19 PM11:43:55 PM30.0 GB24.4 GB24.4 GB1:29:35  
Success10:35:36 PM11:59:22 PM100.0 GB66.8 GB60.4 GB1:23:46  
Error10:35:51 PM11:59:41 PM96.0 GB89.4 GB86.3 GB1:23:49 
Success10:36:07 PM12:19:01 AM (+1)100.0 GB49.4 GB49.4 GB1:42:54  
Success10:49:01 PM1:10:35 AM (+1)300.0 GB68.9 GB68.9 GB2:21:34  
Success11:43:49 PM1:12:07 AM (+1)50.0 GB26.9 GB25.3 GB1:28:18  
Success11:49:41 PM1:12:07 AM (+1)60.0 GB21.1 GB19.7 GB1:22:25  
Success11:50:12 PM1:11:44 AM (+1)60.0 GB25.4 GB25.4 GB1:21:32  
Success11:58:55 PM1:46:14 AM (+1)120.0 GB105.1 GB95.1 GB1:47:19  
Success12:25:07 AM1:46:38 AM80.0 GB30.1 GB30.0 GB1:21:31  
Success12:41:08 AM1:57:31 AM80.0 GB52.0 GB51.8 GB1:16:23  
Success12:41:08 AM1:57:09 AM60.0 GB30.5 GB27.8 GB1:16:01  
JBMANW7Success12:41:23 AM1:57:32 AM60.0 GB58.0 GB54.8 GB1:16:08  
Warning1:10:20 AM2:01:05 AM60.0 GB20.9 GB19.5 GB0:50:44 
Success1:22:05 AM2:00:49 AM30.0 GB24.4 GB24.4 GB0:38:44  
Success1:22:10 AM2:01:05 AM60.0 GB34.3 GB23.0 GB0:38:54  
Success1:42:15 AM3:05:12 AM70.0 GB69.5 GB68.4 GB1:22:57  
Success1:42:35 AM3:47:59 AM280.0 GB252.7 GB239.1 GB2:05:24  
Success1:57:06 AM3:05:14 AM80.0 GB78.2 GB39.8 GB1:08:07  
Success1:57:12 AM3:20:42 AM100.0 GB96.0 GB59.2 GB1:23:30  
Success2:33:36 AM3:47:33 AM75.0 GB35.6 GB35.6 GB1:13:56  
Success2:33:41 AM3:48:31 AM60.0 GB37.7 GB34.0 GB1:14:49  
Success2:33:57 AM4:04:31 AM160.0 GB110.3 GB109.1 GB1:30:34  
Success3:03:26 AM4:35:16 AM80.0 GB75.2 GB63.7 GB1:31:50  
Success3:09:33 AM5:10:08 AM545.0 GB472.9 GB400.2 GB2:00:35  
Success3:47:54 AM5:05:31 AM70.0 GB48.1 GB46.0 GB1:17:37  
Success3:48:04 AM4:59:00 AM100.0 GB84.9 GB84.8 GB1:10:56  
Success4:01:33 AM5:07:21 AM60.0 GB21.7 GB20.3 GB1:05:48  
Success4:01:43 AM5:09:52 AM120.0 GB98.2 GB84.2 GB1:08:08  
Success2:01:38 PM3:03:17 PM30.0 GB25.0 GB25.0 GB1:01:38  
Success9:29:32 PM10:26:09 PM0.0 KB0.0 KB0.0 KB0:56:36  

Veeam Backup & Replication 8.0.0.2084

[jeremyve]

Also, the target is identified as the bottleneck. Here's a copy of what I normally see for full backup job (not the one that goes to local disk and off to tape but the one that goes to the Data Domain.)

7/31/2015 4:45:10 AM :: Load: Source 24% > Proxy 22% > Network 66% > Target 73%

[rccl_ecain]

Going to DD, target will always be the bottleneck unless it's on a 10/100 network.

It looks like the overall average on the Active full is ~100MB/s not including the time for each VM to do pre and post processing, etc.

You have a DD620? Correct? Was there ever a 6200?

Personally, I would have that 104 VMs broken into 4 jobs to spread it out a bit. DataDomain and Single Stream writing sucks, and as far as I have seen, on a normal Active Full / Inc DDBoost isn't mind blowing in performance in the current state.

I have jobs with ~100 VMs going to a very large DD, that don't exceed 16-30MB/s but I can run 20 of those jobs at around those speeds.

[foggy]

Ethan is spot on. Besides the fact that we do not recommend using any deduplicating storage as a primary backup repository (it is a perfect secondary backup target for long-term backups archival), with a single job you're not fully utilizing the available I/O it provides. Multiple jobs will create multiple I/O streams resulting in better ingestion rate.