Still in an implementation phase with Veeam and wondering what your biggest backup job is (how many VM's). Things I should look out for with having large backup jobs besides time taken to complete. Currently my largest backup job has 56 VM's and have had no issue with the size so far. I have a fairly large amount of VM's to backup, around 1200.
I am seeing slight issues with my backup copy job sizes (GFS retention 13 monthly's, 2 yearly's) and the time it takes to complete.
Thank you in advance
-
bpayne
- Enthusiast
- Posts: 55
- Liked: 12 times
- Joined: Jan 20, 2015 2:07 pm
- Full Name: Brandon Payne
- Contact:
-
ericcsinger
- Influencer
- Posts: 24
- Liked: 7 times
- Joined: May 11, 2014 8:52 pm
- Full Name: Eric Singer
- Contact:
Re: How big are your backup jobs?
we've just started and we size the vm's in our jobs based on a few factors:
1. Size of the VM's. Becasue we need to copy the files off to tape and potentially restore from tape, we want to make sure we're not in a scenario where we're restoring 4TB of data to recover a 50GB VM. So for smaller VM's we'll group them in a single job till they hit about the 500GB mark and cut it off from there. Large VM's get their own job...
2. Scheduling flexibility, if we need to make sure certain VM's don't start before "x"
4. Destinations: some vm's don't go to tape, some do... We don't use Veeam for tape (too many reason to list) so we're at the mercy of picking up vbk files.
5. VSS vs. no vss.
Honestly, we've been looking at migrating back to CommVault for VM backup's now that they support socket based licensing. Veeam, is nice, but I don't find that it scales very well, and for a product that's been out as long as they have, its disappointing. It's like "yay you added SQL explorer, but your tape solution still sucks".
1. Size of the VM's. Becasue we need to copy the files off to tape and potentially restore from tape, we want to make sure we're not in a scenario where we're restoring 4TB of data to recover a 50GB VM. So for smaller VM's we'll group them in a single job till they hit about the 500GB mark and cut it off from there. Large VM's get their own job...
2. Scheduling flexibility, if we need to make sure certain VM's don't start before "x"
4. Destinations: some vm's don't go to tape, some do... We don't use Veeam for tape (too many reason to list) so we're at the mercy of picking up vbk files.
5. VSS vs. no vss.
Honestly, we've been looking at migrating back to CommVault for VM backup's now that they support socket based licensing. Veeam, is nice, but I don't find that it scales very well, and for a product that's been out as long as they have, its disappointing. It's like "yay you added SQL explorer, but your tape solution still sucks".
-
foggy
- Veeam Software
- Posts: 21070
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: How big are your backup jobs?
Eric, could you please elaborate a bit on what do you mean by scaling here? Thanks.ericcsinger wrote:Veeam, is nice, but I don't find that it scales very well, and for a product that's been out as long as they have, its disappointing.
-
veremin
- Product Manager
- Posts: 20284
- Liked: 2258 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: How big are your backup jobs?
It will be also appreciated if you find time to identify the major show stoppers in current tape functionality; should help us make to our product better. Thanks.
-
ericcsinger
- Influencer
- Posts: 24
- Liked: 7 times
- Joined: May 11, 2014 8:52 pm
- Full Name: Eric Singer
- Contact:
Re: How big are your backup jobs?
I'll reply to both of you in the same post.
Lets start with what some scaling issues IMO, not all of them technical or product specific.
Technical / Design opinions:
1. I'd really like to see some options for placing jobs in to a group of some sort, that centrally controls settings for a given job. Think of it as almost like group policy, except I actually think it could be more flexible than that. Let me give you an example. We're in the process of ramping up our Veeam jobs (currently migrating from CV). We were finding that there were too many jobs scheduled to do a full backup on a given day. So in order to adjust these jobs, we had two options, draft up a powershell script to change the jobs, or go through the gui one by one and change the job. Same thing goes for something like the repository, or any other series of settings that exist in a given job. I'd much rather see a design where you create a parent level grouping of jobs, that centrally controls a jobs setting. Having the option to manually over ride an individual setting at a job level would still be nice (hence why i said a group policy type solution).
2. We've had at least two occasions where a proxy server was down, either by choice or by accident, and Veeam didn't auto fail over to a running Proxy. We ended up having to disable the proxy in the GUI. What's worse, is there was no immediate indication that the problem was a down proxy server. It required digging through logs to figure it out.
3. Your error messages are pretty cyrptic at times, and I'm not sure why? Something like the above issues, would have been easier to troubleshoot had your message said something like "Unable to contact proxy: %proxy server name%".
4. Tech support with you guys is hit or miss. I realize that's true for a lot of places, but CommVault, i'd say 99% of the time, we feel like we're talking to someone that knows the product well. CommVault is probably 100 times more complex than Veeam (which isn't a knock at you per se). So in turn, I'd like to assume it would be easier to have better trained staff on level 1. Once we get to level 2, things seem to go a lot better, but level 2 I would like to think should be for really complex problems. For example, I had an issue where replication was only going at 1Gbps on a 10Gbps network, and the storage nor the source were the bottleneck. Once i got to level 2, the guys was like, just use the reg hack to increase the number of network streams (or something to that effect). It was like a 5 minute call once i got to the right person.
5. I'll go into more specifics with the tape below, but your lacking in tape has forced us to stay with CV, So now we have to deal with gluing two systems together (made a lot easier with MVP systems JAMS software in case any one else has to deal with it).
6. Lack of the ability to install the Veeam console on a remote machine, or in other words having to remote into the Veeam server, this is just a poor design. I know you guys have the "enterprise console" but honestly, that feels a lot more like a hack to me. Give me one console to use and lets call it a day. I shouldn't need to setup tons of Veeam servers and then use an enterprise console to manage them. I also feel this is a problem with not only administration, but things like restores in remote sites. For example, if I have a server in Prod and copy my backups to DR (using a copy job), and want to recover windows guest files in DR, your console trys to mount the data in prod. I know there are work a rounds (other files, or setup another veeam server and scan the repository), but my point is, we shouldn't need to use a work around. I did this all the time in CV. Backup a file, send it to DR, restore it in DR, all in a GUI, all in a single console, and none of the data coming across the WAN link.
7. Your insistence on not offering a physical backup solution, as if physical servers are just going to disappear. It hasn't happened yet, and its probably not going to happen. There will always be a need for physical servers, I agree the virtual servers have a majority in a big way, but your lack of physical support forces us to run dual products, and that's not very enterprise nor does that scale well. Its another reason i'm considering going back to CV. They do it all, and they generally do it all very well. Plus there's just certain types of backups where application level is still better than VM level. SQL and Exchange being two of them.
8. On a non-technical note, I have no clue who my sales rep is. Which is a problem, i'd like to think a company that I buy a product from would at least try to reach out to me once in a while. VMware does this, CV does this, Dell does this, CDW does this, lot and lots of vendors have a sales rep that attempts to maintain a business relationship. I don't know, maybe with Veeam we need to spend "x" amount to get that level of service, but that's kind of disappointing.
9. Lack of job priority.
I'm sure there are more scaling issues I could think of. In the end, your product works pretty well, I don't want to make it sound like its all bad, but you're on v8 and I just honestly would have expected more by now. I brought your product with me so to speak when i changed jobs (back during 6.5) and while there certainly have been some nice additions, I would have expected more focus on enterprise level adjustments in your architecture.
As for tape, I've already made my suggestions known, as have others, and I'm really just going to be repeating what you guys have already said "yeah yeah we here you" (which is the kind of attitude i see on these forums a lot.
1. If I have a library with multiple tape drives, stop treating each tape drive and each tape as a singular entity. You should "pool" tape drives and pool "media" as a shared resource. Let me point my jobs at a tape pool and its inherit media pool and auto balance all my jobs across the tape drives.
2. As for how to handle different retention and different destinations, that's simple, have repository with these settings and have that repository also point at the same media pool / tape pool. Once a tape is written for a given repository, that tape now can only be used for jobs that specific to that repository (until jobs have expired, then its free for any use case).
Other than that, I can't honestly make any other tape recommendations, because I can't use the feature until you at least have these basic features.
All this said, if you wanted to chat in person about any of this, I'm totally open to it and also, really sorry for hijacking this thread.
Lets start with what some scaling issues IMO, not all of them technical or product specific.
Technical / Design opinions:
1. I'd really like to see some options for placing jobs in to a group of some sort, that centrally controls settings for a given job. Think of it as almost like group policy, except I actually think it could be more flexible than that. Let me give you an example. We're in the process of ramping up our Veeam jobs (currently migrating from CV). We were finding that there were too many jobs scheduled to do a full backup on a given day. So in order to adjust these jobs, we had two options, draft up a powershell script to change the jobs, or go through the gui one by one and change the job. Same thing goes for something like the repository, or any other series of settings that exist in a given job. I'd much rather see a design where you create a parent level grouping of jobs, that centrally controls a jobs setting. Having the option to manually over ride an individual setting at a job level would still be nice (hence why i said a group policy type solution).
2. We've had at least two occasions where a proxy server was down, either by choice or by accident, and Veeam didn't auto fail over to a running Proxy. We ended up having to disable the proxy in the GUI. What's worse, is there was no immediate indication that the problem was a down proxy server. It required digging through logs to figure it out.
3. Your error messages are pretty cyrptic at times, and I'm not sure why? Something like the above issues, would have been easier to troubleshoot had your message said something like "Unable to contact proxy: %proxy server name%".
4. Tech support with you guys is hit or miss. I realize that's true for a lot of places, but CommVault, i'd say 99% of the time, we feel like we're talking to someone that knows the product well. CommVault is probably 100 times more complex than Veeam (which isn't a knock at you per se). So in turn, I'd like to assume it would be easier to have better trained staff on level 1. Once we get to level 2, things seem to go a lot better, but level 2 I would like to think should be for really complex problems. For example, I had an issue where replication was only going at 1Gbps on a 10Gbps network, and the storage nor the source were the bottleneck. Once i got to level 2, the guys was like, just use the reg hack to increase the number of network streams (or something to that effect). It was like a 5 minute call once i got to the right person.
5. I'll go into more specifics with the tape below, but your lacking in tape has forced us to stay with CV, So now we have to deal with gluing two systems together (made a lot easier with MVP systems JAMS software in case any one else has to deal with it).
6. Lack of the ability to install the Veeam console on a remote machine, or in other words having to remote into the Veeam server, this is just a poor design. I know you guys have the "enterprise console" but honestly, that feels a lot more like a hack to me. Give me one console to use and lets call it a day. I shouldn't need to setup tons of Veeam servers and then use an enterprise console to manage them. I also feel this is a problem with not only administration, but things like restores in remote sites. For example, if I have a server in Prod and copy my backups to DR (using a copy job), and want to recover windows guest files in DR, your console trys to mount the data in prod. I know there are work a rounds (other files, or setup another veeam server and scan the repository), but my point is, we shouldn't need to use a work around. I did this all the time in CV. Backup a file, send it to DR, restore it in DR, all in a GUI, all in a single console, and none of the data coming across the WAN link.
7. Your insistence on not offering a physical backup solution, as if physical servers are just going to disappear. It hasn't happened yet, and its probably not going to happen. There will always be a need for physical servers, I agree the virtual servers have a majority in a big way, but your lack of physical support forces us to run dual products, and that's not very enterprise nor does that scale well. Its another reason i'm considering going back to CV. They do it all, and they generally do it all very well. Plus there's just certain types of backups where application level is still better than VM level. SQL and Exchange being two of them.
8. On a non-technical note, I have no clue who my sales rep is. Which is a problem, i'd like to think a company that I buy a product from would at least try to reach out to me once in a while. VMware does this, CV does this, Dell does this, CDW does this, lot and lots of vendors have a sales rep that attempts to maintain a business relationship. I don't know, maybe with Veeam we need to spend "x" amount to get that level of service, but that's kind of disappointing.
9. Lack of job priority.
I'm sure there are more scaling issues I could think of. In the end, your product works pretty well, I don't want to make it sound like its all bad, but you're on v8 and I just honestly would have expected more by now. I brought your product with me so to speak when i changed jobs (back during 6.5) and while there certainly have been some nice additions, I would have expected more focus on enterprise level adjustments in your architecture.
As for tape, I've already made my suggestions known, as have others, and I'm really just going to be repeating what you guys have already said "yeah yeah we here you" (which is the kind of attitude i see on these forums a lot.
1. If I have a library with multiple tape drives, stop treating each tape drive and each tape as a singular entity. You should "pool" tape drives and pool "media" as a shared resource. Let me point my jobs at a tape pool and its inherit media pool and auto balance all my jobs across the tape drives.
2. As for how to handle different retention and different destinations, that's simple, have repository with these settings and have that repository also point at the same media pool / tape pool. Once a tape is written for a given repository, that tape now can only be used for jobs that specific to that repository (until jobs have expired, then its free for any use case).
Other than that, I can't honestly make any other tape recommendations, because I can't use the feature until you at least have these basic features.
All this said, if you wanted to chat in person about any of this, I'm totally open to it and also, really sorry for hijacking this thread.
-
foggy
- Veeam Software
- Posts: 21070
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: How big are your backup jobs?
Eric, thanks for the effort in posting such a detailed feedback, much appreciated. I can assure you that we already work on addressing at least some of your points.
-
emachabert
- Veeam Vanguard
- Posts: 388
- Liked: 168 times
- Joined: Nov 17, 2010 11:42 am
- Full Name: Eric Machabert
- Location: France
- Contact:
Re: How big are your backup jobs?
This is a useful feedback and it contains a lot of things I often hear when speaking with my customers.
The good point is that we are on the Veeam forum and it will be taken into account by Anton and his team
The good point is that we are on the Veeam forum and it will be taken into account by Anton and his team
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
-
bpayne
- Enthusiast
- Posts: 55
- Liked: 12 times
- Joined: Jan 20, 2015 2:07 pm
- Full Name: Brandon Payne
- Contact:
Re: How big are your backup jobs?
I have some jobs with 50 or more VM's, currently one being about 20TB's, others smaller and others bigger. And this will grow substantially in the future as I have a ton more VM's to add. So far, this has not been a problem, but what am I going to run into when suddenly a particular backup job has 400 VM's and is 100TB's? How will that scale? Should I break up into smaller backup jobs or does that matter?
One problem that I've seen so far is that I have a few Backup Copy Jobs that are pretty big. One being 16TB's. This is to meet our GFS retention 13 monthly's and 2 yearly's. I see a major flaw in how Backup Copy Jobs are processed. It appears to only process one VM at a time. Why not use parallel processing like Backup Jobs???? This does not seem to scale well.
I am on Veeam v8 Patch 1 (the latest at this time). Backup Copy Jobs seemed somewhat OK at first when I was on Veeam v7, now they are dog slow. Almost every night I get "Copy interval has expired" and Failed events. I'm also seeing "Full backup file merge (GFS)" taking days to complete. The longest I've seen it take is about 4 days.
Are my Backup Copy Jobs too big? Do I need to break these up into smaller Backup Copy Jobs? Lots of questions in there I know. Thanks
One problem that I've seen so far is that I have a few Backup Copy Jobs that are pretty big. One being 16TB's. This is to meet our GFS retention 13 monthly's and 2 yearly's. I see a major flaw in how Backup Copy Jobs are processed. It appears to only process one VM at a time. Why not use parallel processing like Backup Jobs???? This does not seem to scale well.
I am on Veeam v8 Patch 1 (the latest at this time). Backup Copy Jobs seemed somewhat OK at first when I was on Veeam v7, now they are dog slow. Almost every night I get "Copy interval has expired" and Failed events. I'm also seeing "Full backup file merge (GFS)" taking days to complete. The longest I've seen it take is about 4 days.
Are my Backup Copy Jobs too big? Do I need to break these up into smaller Backup Copy Jobs? Lots of questions in there I know. Thanks
-
foggy
- Veeam Software
- Posts: 21070
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: How big are your backup jobs?
It does matter from the backup performance/manageability perspective. Btw, there're a lot of environments of this size backed up by Veeam B&R, so this should not be a problem, if planned properly.bpayne wrote:I have some jobs with 50 or more VM's, currently one being about 20TB's, others smaller and others bigger. And this will grow substantially in the future as I have a ton more VM's to add. So far, this has not been a problem, but what am I going to run into when suddenly a particular backup job has 400 VM's and is 100TB's? How will that scale? Should I break up into smaller backup jobs or does that matter?
Here's an existing thread discussing this, FYI.bpayne wrote:I see a major flaw in how Backup Copy Jobs are processed. It appears to only process one VM at a time. Why not use parallel processing like Backup Jobs???? This does not seem to scale well.
Who is online
Users browsing this forum: bmcguire, FrancWest, Mildur, Semrush [Bot] and 134 guests