Comprehensive data protection for all workloads
Post Reply
scottstabb
Novice
Posts: 4
Liked: never
Joined: Jul 17, 2012 10:46 pm
Full Name: Scott Stabb
Contact:

1 year retention in the cloud

Post by scottstabb »

case ID 00162944
I dont know were to start. I am new to VMWare, Veeam and TwinStrata I want to use get my backups offsite probably using TwinStrata and S3.
I have a little less than 1TB that compresses down to less than 400GB using Veeam that consists of 12 VMs and my daily change files are about 3 to 7 GB compressed. I have a dedicated Cable connection with about 6m upload speed.

I have my local backups under control, but I need a way to get some retention off site and be able to recover entire VMs if necessary. If I send a full backup each month it will take a couple of weeks to get there and several TB by the end of the 13 Months. (this is how it was done on tape)
Our policy is to be keep Monthly backups for 13 months and the monthly backups need to be kept off site.
Our local policy is backing up on a NAS and I can go back for several weeks on a daily basis.

I really don't know where to start I would like to think I could have a full backup in the cloud and send the changes monthly. Or even weekly Maybe.
Was I supposed to open a case for this since it was non technical?

Many thanks!
pendragoncrw
Enthusiast
Posts: 38
Liked: 3 times
Joined: Jun 14, 2010 3:06 am
Full Name: C White
Contact:

Re: 1 year retention in the cloud

Post by pendragoncrw »

I am also very curious in what people are using to obtain cloud backups using Veeam and TwinStrata. If I understand the TwinStrata architecture correctly, you map what would be your NAS (or local storage, etc.) as a VMWare datastore, add a volume on that datastore to the TwinStrata box, and use the TwinStrata box as the local storage for Veeam.

Once we get to the cloud storage part, it seems very difficult to estimate what normal usage would be. I like to use reverse incrementals for my NAS storage but it seems like forward incrementals would make the most sense for cloud storage.

Inquiring minds are curious.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 1 year retention in the cloud

Post by Gostev »

Hi Scott, support case is only required when posting about an error or technical issue. You don't need to open a ticket in all other cases, just ask your question. Thanks!
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: 1 year retention in the cloud

Post by deduplicat3d »

I would break the VM's into a couple of different jobs, that way each full it smaller... then you can rotate when each job is running it's fully. So if you do 4 jobs (3 VM's in each assuming they are equal size), full of job1 on week one, full of job2 on week two, etc... Also you may want to think about doing a fully less frequently than once a month. Remember Veeam is image level so increments can do full restores.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: 1 year retention in the cloud

Post by deduplicat3d »

Also, I don't know about TwinStrata but S3 you could easily script it to send the increments every day (since they are so small), just make sure to send the vib's and the vbm's... look up the scripts used for Backup Exec on this forums, you can use something similar with S3 integration.
murdocmk
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 14, 2009 5:10 pm
Contact:

Re: 1 year retention in the cloud

Post by murdocmk »

Breaking up the jobs is a good idea, allowing you to run the backups in smaller batches more often. I actually think that reversed incrementals would be the best job type to use, to avoid ever needing to do an active or synthetic full (since either of those would result in a lot of data to send to the cloud provider).

You could set up a single job, or multiple jobs, reversed incremental job type, running each weekend starting on Friday night. Over a weekend with a 6 Mbps you should be able to get 100 - 120 GBs of changes uploaded. Based on the information you've provided your weekly changes should be much less than that. At the end of the year you'd have one large VBK file (your most recently backup) and 51 or 52 reversed incremental VRB files.

Now, how much space would that take up? I'm not sure because I'm not sure what your weekly changes would be. If you can get the weekly changes uploaded with no problem then you might try running the backup job every 2 weeks instead (to reduce the total amount of space needed). I'm not sure if you'd be able to get to the point of running monthly backups only.
scottstabb
Novice
Posts: 4
Liked: never
Joined: Jul 17, 2012 10:46 pm
Full Name: Scott Stabb
Contact:

Re: 1 year retention in the cloud

Post by scottstabb »

I like the breaking up the jobs idea, that can give me a lot more control as some of the VMs are a lot less important. For the Local Jobs I have 14 restore points, Reversed Incremental, and its doing a Active full backup Every Sunday. And then every morning I connect an External USB driver to the NAS and use Robocopy to copy all the data so I can take it home. The Reversed incremental injects the previous day into the full backup, so that means the large full backup changes daily and has to be copied to the external drive. On Monday there is a the full backup with changes from the prior week and the new full backup from Sunday that has changed. With a reverse incremental changing the full backup every time, wont the full backup need to be copied to the cloud each time? I'm sure i am missing something. Many thanks for your help.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: 1 year retention in the cloud

Post by deduplicat3d »

I prefer the reverse incremental, but for this, I don't believe it would be a good fit especially since you are doing active fulls every week (thus not really gaining the benefits of reverse incremental). You are correct that your full changes every day and furthermore, Veeam takes an exclusive lock on the full, so robocopying it may become problematic.

For your setup, I would just recommend Forward Incremental, active fulls once a week. Then any backups that are complete, you can copy off immediately as they will not change in the future. This means that you can do the copy to USB once a week or as often as everyday.
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: 1 year retention in the cloud

Post by cparker4486 »

murdocmk wrote:If you can get the weekly changes uploaded with no problem then you might try running the backup job every 2 weeks instead (to reduce the total amount of space needed). I'm not sure if you'd be able to get to the point of running monthly backups only.
Am I misunderstanding you or are you really suggesting that it's acceptable to take backups 4 or fewer times per month?
-- Chris
scottstabb
Novice
Posts: 4
Liked: never
Joined: Jul 17, 2012 10:46 pm
Full Name: Scott Stabb
Contact:

Re: 1 year retention in the cloud

Post by scottstabb »

deduplicat3d wrote:I prefer the reverse incremental, but for this, I don't believe it would be a good fit especially since you are doing active fulls every week (thus not really gaining the benefits of reverse incremental). You are correct that your full changes every day and furthermore, Veeam takes an exclusive lock on the full, so robocopying it may become problematic.
The above is my current local backup. I am really trying to grasp the concepts of being able to backup to to the Cloud and not having to do anything manual. I will keep my local backup plan in place because it gives me very fast recovery and I would be able to copy it to a USB drive once in while if a hurricane was approaching...

Back to the cloud. Monthly Full's of all VMs don't seem to make much sense because some of the VMs just wont change that much, it would take nearly a month to get all that data to the cloud and I would be using some serious space. Would it be a good idea to do a Full backup and then forward incremental weekly, every other week or Monthly to capture the changes? Can forward incremental backups be done for an entire year like that? That just seems risky to me asking a software to keep all that together. Especially after the response from support (below) This is frustrating, We are very small and I don't think it should be that difficult. I tend to over think things sometimes, I'm sure i am missing the keep it simple answer. I'm thinking about giving TwinStrata a call. Their support was great getting the software implemented and then I had to have support again because we replaced the NAS and had to do the setup all over again. When I submitted this as a case (i know now i did not have to) , Veeam support called me back and I missed the call. But as I was writing this they sent me an email and we have an appointment at 2PM to talk with them. I don't really think its their responsibility to support users trying to understand the "concepts" so this is a very big thumbs up for Veeam support. I will let you know what comes of the conversation if anyone cares to know.

Response from Support yesterday...
I tried calling out but got a message that it's after-hours at your office.

If your fulls are 400gb, and you want to do monthly offsite backups with 13 months of retention, then I'm afraid there's really no wait to avoid needing 6+ TB of space for your offsite backups. You could switch to reverse incremental instead of always-full, but the reduction in size would be minimal, performance over the WAN would be poor, and I don't think the loss of reliability from having files dependent on each other is worthwhile. You may be able to further reduce the size of your backup files (and reduce the amount of time required to transfer the files over the WAN) by changing your deduplication settings to Extreme compression (8 processor cores on the proxy recommended) and choose WAN optimization.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: 1 year retention in the cloud

Post by deduplicat3d »

It sounds to me that you are only interesting in the cloud for DR and not archival purposes in which case I would only keep one or two full backups there. That means that you will have little difference between forward and reverse incremental because you just want a single point in time. If I'm wrong and you want multiple points in time I would then recommend forward incremental. I think taking a look at page 27 of the userguide will answer your question:

http://www.veeam.com/veeam_backup_6_use ... are_pg.pdf
andersonts
Veteran
Posts: 307
Liked: 31 times
Joined: Mar 21, 2012 9:56 pm
Full Name: Tim Anderson
Contact:

Re: 1 year retention in the cloud

Post by andersonts »

Hi have you reviewed this Best Practices guide provided by TwinStrata? http://www.twinstrata.com/files/Veeam-B ... ctices.pdf
gkettmann
Technology Partner
Posts: 4
Liked: 1 time
Joined: Oct 19, 2011 5:42 pm
Full Name: Gregory Kettmann
Contact:

Re: 1 year retention in the cloud

Post by gkettmann »

I'm with TwinStrata. In the previously mentioned best practices guide Veeam recommends "Incremental mode with synthetic full". Using synthetic full backups it's possible to limit the data transfer requirements to just the change rate.

CloudArray (our gateway to cloud storage) basically uses SAN to SAN asynchronous replication with the offsite copy being in the cloud. With that in mind we allow volume level snapshots, exactly like with a SAN device, except they take place in the Cloud. There is no additional data transfer required, we simply mark the current view of the cloud volume read-only and make changes off to the side.

In addition we add byte level deduplication to the outbound data stream, further reducing the cloud storage (and bandwidth) requirements. Using these mechanisms I'd be surprised if your 400GB backups, and 12 monthly copies, exceeded 800GB.

NOTE - you can validate the integrity of your synthetic full volume backups by using mechanisms within Veeam or by starting a test instance of Veeam and mounting one of the CloudArray snapshots on the test instance. Verify that you can recover one or more machines and then, when done, delete all the test instances.

Greg Kettmann
murdocmk
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 14, 2009 5:10 pm
Contact:

Re: 1 year retention in the cloud

Post by murdocmk »

cparker4486 wrote: Am I misunderstanding you or are you really suggesting that it's acceptable to take backups 4 or fewer times per month?
He is doing local backups every day. His tape procedure was to send monthly tapes off-site, so this discussion was taking place around replacing that off-site tape procedure with Veeam and a cloud-based backup solution. He was trying to conserve space in his off-site storage of monthly backups, while balancing that with the amount of time required to send those changes to the cloud. So my recommendations were really just around the off-site "monthly tape replacement" part of his backup plan. I definitely think he should be backing up daily, and I would also suggest keeping a few recent backups (not counting the monthly backups) off-site for DR purposes.
murdocmk
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 14, 2009 5:10 pm
Contact:

Re: 1 year retention in the cloud

Post by murdocmk »

gkettmann wrote:I'm with TwinStrata. In the previously mentioned best practices guide Veeam recommends "Incremental mode with synthetic full". Using synthetic full backups it's possible to limit the data transfer requirements to just the change rate.

CloudArray (our gateway to cloud storage) basically uses SAN to SAN asynchronous replication with the offsite copy being in the cloud. With that in mind we allow volume level snapshots, exactly like with a SAN device, except they take place in the Cloud. There is no additional data transfer required, we simply mark the current view of the cloud volume read-only and make changes off to the side.

In addition we add byte level deduplication to the outbound data stream, further reducing the cloud storage (and bandwidth) requirements. Using these mechanisms I'd be surprised if your 400GB backups, and 12 monthly copies, exceeded 800GB.

NOTE - you can validate the integrity of your synthetic full volume backups by using mechanisms within Veeam or by starting a test instance of Veeam and mounting one of the CloudArray snapshots on the test instance. Verify that you can recover one or more machines and then, when done, delete all the test instances.

Greg Kettmann
Oops, I forgot the way that TwinStrata worked and the fact that synthetic fulls result in only the changes sent to the cloud storage. I need to remember to consider each cloud vendor's approach -- for some a reversed incremental is the ideal way to go, but not in this case.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 282 guests