Testing Veeam B&R v8 and I want to archive data to Amazon s3

[veremin]

Actually, you can ship compressed and deduplicated VB&R backups to amazon S3 storage. However, in order to achieve that Amazon Storage Gateway will be required, as well as, tape jobs pointed to it. Thanks.

[AtechAddam]

Hi All,

Sorry if this is a bit of a n00b question.

I have a customer that wishes to have their daily backup copied off to Amazon S3.

We have the backup set up going to network storage and then I have created a backup copy job to copy this over to S3.

This appears to be working, however it is a continuous job. I have set the schedule so that it doesn't interfere with any of their other backups, however once the job pauses, it does not restart.

I have had to cancel the job and start it again so that it runs at the next interval.

Other than this, it runs continuously and even though I've set a schedule, it doesn't appear to be running to this schedule.

I'd prefer it to copy this backup over to S3, finish the job and then the next morning at 1am for it to start again.

I'm wondering if this is by design, or if I am doing something wrong?

Is a backup copy job the best way to do this, or would a normal backup be better?

Thanks,

Addam

[veremin]

Hi, Addam,

Your post has been merged into existing discussion, so, see the answers provided above for more information about backup to Amazon S3.

Thanks.

[tym]

Hi

I just wanted to start moving from tape to cloud backup and what did I find? VEEAM cancelled support for Amazon Glacier in that they discontinued the cloud edition. Very sad and a huge drawback. Do I have to use the amazon gateway now? I dont want to use another backup software like CloudBerry Backup for this.

[veremin]

Yep, you need to leverage Amazon Storage Gateway in order to get tape-like experience with Amazon Glacier or Amazon S3 storage.

Alternatively, you can use Service Provider lookup, find an SP meeting your requirements and just point your backup and/or backup copy jobs to the cloud repositories provided by them. This will provide much better performance for backups and more importantly for restores (enabling restores directly from the cloud), plus greater manageability due to being fully integrated.

Thanks.

[Gostev]

I just wanted to start moving from tape to cloud backup and what did I find? VEEAM cancelled support for Amazon Glacier in that they discontinued the cloud edition. Very sad and a huge drawback.

So, your first sentence is the most important. Thing is, most users are unwilling to move from tape to cloud until they can backup and restore their data to/from the cloud faster than using tape. As you correctly noted, we did try Amazon-based solution, but most of our users found backup and restore performance to be unacceptable (at least for image-based backups), which is exactly why this functionality was discontinued and replaced with Cloud Connect (which provides at least by an order of magnitude faster backup and restore performance, making it a real alternative to tape).

[Meyercord]

Is anyone writing backups to cloud storage without the Cloud Connect product? I need a reliable way to get my data offsite into inexpensive long-term cloud storage like S3 or Glacier, but I want to do it directly without relying on a third-party partner.

I've tested backing up to virtual tape using Amazon's VTL solution, but because Veeam requires writing the entire backup set to disk before performing a restore, recovering even a single file or VM would require downloading 4-6TB of data, making it prohibitively expensive and creating a significant delay.

One potential solution I can think of is to host a repository on Amazon Storage Gateway Cached Volume (http://docs.aws.amazon.com/storagegatew ... cepts.html), but I have a number of concerns and I was hoping someone here had either looked into this as well or even implemented it in some form and could comment on their experience:

1) How would you deal with maintaining restore points over the long-term (5-7 years)? Would you continually write them to a single massive volume, or create smaller volumes and swap them out as they fill up? Does it make sense to take Amazon EBS snapshots at intervals and then delete the older Veeam recovery points after they had been snapshot?

2) Are snapshots even practical? The cost to store snapshot data is >3 times the S3 storage cost ($0.095/GB vs $0.030/GB)

3) Is there any way to move the data from S3 to Glacier to reduce cost ($0.010/GB vs $0.030/GB)? I don't see anything in the documentation indicating that it's possible to move Gateway data to Glacier.

4) In the situations above, if I roll off older restore points by swapping out volumes, or snapshotting, or moving them to Glacier, how does Veeam deal with the disappearance of these recovery points? How can I find them in my inventory years from now when I need to restore a file or VM from a specific date or based on other criteria? Do I have to keep everything online all the time just so Veeam doesn't lose track of it? Is there any way to tell Veeam that the data has been archived, and then locate that archived data years later?

5) For performing restores would it be beneficial to build a Veeam proxy server in Amazon EC2 that would have faster access to the data in S3? If I did this, could would it be possible to restore a VMWare guest directly to EC2? Is this a practical way to restore files or VMs, either to an EC2 instance or back to my local LAN?

Am I overthinking this? Is is better to just use Cloud Connect or the Amazon Virtual Tape Library? If you've used either of these, Id like to hear about your experience

[mongie]

I believe in V9 there is a mount server which would allow your point 5 to be valid.

To speed restores in V8, I think you could do this with a full veeam server, and manual imports to the database.

[Tailwind]

Hello everyone,

I'm looking for some advice:

I am using B&R to backup 20 HyperV VMs to a QNAP backup device. Currently have no offsite solution though, so I am trying to get something in place ASAP!
Now, QNAP has the ability to upload to Amazon S3 cloud storage, I am trying to look for the best Veeam backup method to use with the intention of uploading the backups to Amazon S3 cloud storage, whilst also keeping the cost as low as possible. Suggestions anyone?

[foggy]

Please review this thread for suggestions and feel free to ask for any further clarification, if required.

[SisTecno]

Hi,

We are configuring the Veeam Backup & Replication v8 and we want to use the AWS as a Service Provider to make the Cloud Backups, is it posible.
Can you explain me how to configure.

Thanks

[veremin]

What kind of storage are you planning to use in combination with that VM? EBS, S3, Glacier, etc.?

[SisTecno]

I want to use S3 for the newers backup and want to move to Glacier the olders. I want to do so making a Budget and in the options make a lifetime to move to glacier older than 3 months

[veremin]

In this case, you would need Amazon Storage Gateway. More information regarding the applicable scenario, as well as, drawbacks it contains can be found on the first page of this discussion; kindly, take a look. Thanks.

[113]

Hello,

Has anyone tried restoring Data from AWS using veeam vtl?

Thanks

[readie]

Starting a new topic, for reasons explained below.
We have been looking at Glacier for 'archiving', as the costs of storage are very low, and we don't expect to have to restore.
Came across 'Veeam Backup and Replication Cloud Edition' and it looked just the job. Simple GUI which allows connection to many different cloud storage providers, including Amazon S3 and Glacier. Started to investigate. Is it a separate product from B&R V8 which we are running? Additional purchase?
Then realised that the videos and descriptions I had been seeing were all dated around 2013/2014, and it seems that Veeam have now 'bundled' it into V8 which we have.
However, the bundling seems to have complicated the situation, and now we either need a 'Cloud Service Provider' (additional costs) and/or Amazon's AWS Gateway (additional monthly cost $125+).
Some have suggested using Synology (which we have) which has the Glacier software built in to the latest version, but it will be a pity to use Synology rather than Veeam to do this bit of our backup strategy.
I should value any comments from Veeam themselves, or from anyone else who has managed to use Glacier as part of their backup strategy.

[veremin]

We don't allow duplicate discussions, so your post has been merged into the existing topic. You can find plenty of real world feedback and considerations here, kindly please review and feel free to ask additional questions if required. Thanks.

[veremin]

Has anyone tried restoring Data from AWS using veeam vtl?

Can you elaborate on your question, as Veeam doesn't provide any virtual tape library? Or you're talking about Amazon AWS Gateway?

[dellock6]

However, the bundling seems to have complicated the situation, and now we either need a 'Cloud Service Provider' (additional costs) and/or Amazon's AWS Gateway (additional monthly cost $125+).
Some have suggested using Synology (which we have) which has the Glacier software built in to the latest version, but it will be a pity to use Synology rather than Veeam to do this bit of our backup strategy.
I should value any comments from Veeam themselves, or from anyone else who has managed to use Glacier as part of their backup strategy.

Cloud Edition is not available anymore as the technology was not able to scale efficiently above few VMs. The "bundle" in v8 is not the same technology, as Cloud Connect is a completely new and more efficient solution.
If you do not want to go for Cloud Connect, there are around many solutions that can work as a cloud gateway towards Glacier, Synology is not at all the only option. Whatever can be presented to Veeam as a local disk or a SMB share can be the target for a backup or better a backup copy, then it's up to the appliance (hardware or software) to cache locally and send blocks towards Glacier. Just be aware that the limited cache of these solutions makes them slow to upload Veeam files, which are usually way bigger than their cache. The new Per-VM chains coming in v9 will definitely help.

[readie]

Thanks for merging v.Eremin, and for clarification of the situation Luca. After much research we are now thinking seriously of Crashplan, as all our repositories at the moment are on a single Windows 2012 R2 Server as shared iSCSI drive mounts (to various NAS stores) so we think the Windows server with Crashplan will be able to copy the ones we want up to the cloud.
Only thing I can't get my head round is how the permanent archiving system of Crashplan will manage with our incremental or reverse incremental backups . . . should we only backup the vbk and vbm files, and not the vbi? Would Backup Copies be less of a problem?
Anyone tried this, and/or any suggestions would be welcome.
(As an aside, I'm thinking of handling my home 3 PCs using VEB to backup two to a spare HD on the third, and then a single home Crashplan subscription to get it all up to the cloud.)
Sorry - realised that this is becoming a Crashplan thread rather than a Veeam thread, but leveraging what Veeam gives us to start with.

[veremin]

Based on quick google research, I don't think that the said solution will manage efficiently backup modes that imply transform operations (forward forever incremental, reversed incremental). In other words, it seems that each time .vbk file will be copied anew.

Though, this statement needs to be proven.

Thanks.

[readie]

Yes I suspect you're right, and we might start just for archiving our infrequent backup copy archives (annual, quarterly, or fortnightly). Of course Crashplan is not going to 'understand' Veeam's backup chain etc . . . . It's cheap, but more thought required!
Thanks for your input.
Bob

[veremin]

You're welcome. I'd be interested to hear the results of your findings, so, keep us updated on how it goes. Thanks.

[dellock6]

Many of this solutions are file-based replica, and not block-based. I did some researches some years ago, so it may have been changed, but those software are designed for the use case of a computer file system, where files are small, and thus they can be copied again if modified.
With Veeam, if we just inject few blocks into the vbk, the file is completely new from the point of view of the replication software, thus it has to be copied entirely again. So, Forever forward and reversed modes are to be excluded. You may try active full, but then each cycle you have to copy the whole full backup file.

I've seen some replication software able to work at the block-level, so they might be able to just replicate the changed block after a transform, but they are pricey, so every penny saved by going to these cheap storage solutions are lost.

Sorry for sounding like a marketing boy, but this is again the advantage you get by using Veeam Cloud Connect over plain cloud storage...

Luca

[readie]

Yea I know you're right, but we might start just with some occasional archives - e.g. at the moment we keep an annual archive on external USB HDs, and dumping them up to Crashplan (will take ages) might be worth a try. We have got a fairly good Internet (500Mbs - though it's not that upload speed) compared with some.
Bob

[SuperStrike]

Many of this solutions are file-based replica, and not block-based. I did some researches some years ago, so it may have been changed, but those software are designed for the use case of a computer file system, where files are small, and thus they can be copied again if modified.
With Veeam, if we just inject few blocks into the vbk, the file is completely new from the point of view of the replication software, thus it has to be copied entirely again. So, Forever forward and reversed modes are to be excluded. You may try active full, but then each cycle you have to copy the whole full backup file.

I've seen some replication software able to work at the block-level, so they might be able to just replicate the changed block after a transform, but they are pricey, so every penny saved by going to these cheap storage solutions are lost.

Sorry for sounding like a marketing boy, but this is again the advantage you get by using Veeam Cloud Connect over plain cloud storage...
Luca

CrashPlan uses block-level de-duplication backup.

CrashPlan detects the duplicate blocks of data and does not send them again. If the file changes, only the changed blocks are transferred.

I think CrashPlan is great for vbk backup.

[veremin]

So, you have tested CrasPlan with the .vbks produced by forever incremental modes - forward forever incremental and reversed incremental ones? When the retention comes, and full backup gets updated, CrashPlan will send just the new blocks, updating the .vbk file already copied to it? Thanks.

[readie]

We looked at Crashplan and thought I had checked that it does NOT do block level backups, so rather gave it up. (We also found that it was DESPERATELY slow as it only managed 2 -3 Mbps, even though we have 100+Mbps available upload . . . they let everyone share their available bandwidth!) However, I have just found this on their site . . .
"After initial backup of the file is complete, only new or changed information is sent when the file is backed up.
When CrashPlan scans a file, it knows that the file changed and the progress bar runs through the file as if the information is new. But as it goes, it discovers the information hasn't actually changed and only transmits the new information to the backup destination.
For the technically savvy: CrashPlan does incremental deltas by block within the file."
However, having ditched Crashplan I have been testing Google Cloud Nearline (using Cloudberry) which is definitely block level. I have had to do a cunning tweak, to convince Cloudberry that it is backing up the same file, even though Veeam has changed the name/date of the VBK file. I have a little batch file which creates a generic named (Windows) 'hard pointer' to the VBK file, and ask Cloudberry to backup this hard link. It does take quite some time to analyse the file before sending, as it has to compare every block with what it had before (I presume), but does then just backup the incremental blocks. . . . I haven't yet looked at restoring, as my initial tests are still progressing. A 2TB VBK file took 5 days to backup (first full backup) and its first incremental is tomorrow. Looking forward to 'per VM backup chains' as a means of spreading these initial full backups.
Happy to keep you all posted as we progress.

[marchawk]

Hi

I am looking into what options I have to start to replace my tape backups with a sync to AWS S3 and subsequent offiste Synthetic Full processing, my CIO wa s not impressed with the last tape invoice I just dropped on his desk.

I an ideal scenario I'd love to have :

- Veeam server onsite running a two week synthetic full schedule - Synthetic fulls on first and 3rd Friday of each month. Incremental each day there after.
- Incremental backups both stored on local oniste disk and transferred to AWS S3. Transfer of incremental veeam jobs to S3 would be daily as Veeam jobs complete.
- THIS IS THE PART I NEED GUIDANCE WITH: Veeam Synthetic full job then processed in AWS S3 (would need some form of EC2 instance with Veeam product I assume?). Want to only transfer incremental jobs to S3 then process synthetic fulls with some typeof compute form AWS.
- Synthetic Full also processed on local disk which would prominently be used for restores.

Can anyone please advise if this is possible or is their is a better way to approach this.

Thanks in advance

[barcode2328]

Hello,

We are looking for a new backup solution and one of them on the table is Veeam. There is only one question I have though, and I can't seem to find a direct answer for it. I am hoping that someone can help me out.

Will Veeam v9 backup directly to Amazon (AWS)?

That is our biggest thing we are looking for since we utilize AWS quite a bit right now. We do have storage gateways setup currently with Amazon which we use to send backups up there right now with our existing software. I am hoping to utilize those same gateways if we end up going with Veeam to backup up to AWS as well.

Researching this out on the internet I found that v8 can probably do this, and that on Amazon's side is capable; but then I hear stuff about using CloudBerry and that it might not be supported anymore? I don't want some roundabout way to get our backups up to AWS, I just want it pretty seamless getting them up there. And of course if we were to become a new user with Veeam we would be jumping on v9 I would imagine

Thanks for any answers or pointers.