Post 9.5 issues

Availability for the Always-On Enterprise

Re: Post 9.5 issues

Veeam Logoby Gostev » Fri Dec 02, 2016 9:18 pm

Thanks, hopefully the required logs are all there. We will investigate.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Post 9.5 issues

Veeam Logoby Gostev » Sat Dec 03, 2016 11:48 am

According to the logs, your 9.5 jobs are using hot add transport. So your "mega proxy" is not used at all - while hot add proxy probably lacks CPU, which perhaps explains this dramatic difference in performance between 9.0 and 9.5. So the troubleshooting vector will be to investigate why mega proxy is not being used... may its components were not update to 9.5 or something like this.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Sat Dec 03, 2016 2:31 pm

I have 2 vProxies I use for 4 dedicated servers which are the only 4 servers to use vProxies. I left those out of the original description for that reason. I can assure you my 'mega proxy' was in control of the jobs and processing using SAN snaphots. I studied the v9.5 environment for a week before deciding to roll back. I'm quite certain I would have noticed. In addition, while jobs were running I was logged onto the the 'mega proxy' watching the server stats: CPU/NIC/Disk and SAN exports. Further, when I decided to roll back I had to manually remove all the v9.5 components from all my proxies and redeploy the v9u2 agents because an error was rightfully displayed indicating the v9.5 version was incompatible with v9U2 console. So I respectfully disagree with your assertion my physical proxy was not in use. I will agree the log set that was requested/uploaded may not reflect this because they may not have included actual job logs.
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Post 9.5 issues

Veeam Logoby Gostev » Sat Dec 03, 2016 6:50 pm

OK. Well, if those are the only logs left from your 9.5 install, then there's not much we can do to troubleshoot the issue further... it must be something unique to your environment, because with 3PAR specifically, we see 2x speed improvement backing up from SAN snapshots with 9.5 in our own lab - and other customers with 3PAR are reporting the same increase. So, almost 3x decrease instead is definitely unusual.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Post 9.5 issues

Veeam Logoby guitarfish » Sat Dec 03, 2016 8:53 pm

In v9 & previous, you could go to "Files" and access files in the cloud repository and delete a file, rescan the repository, and then go to Backups > Cloud, and the restore point for the deleted file would have an indictor that it was missing. You could Remove from inventory or Delete from disk IIRC to clean it up. Now in 9.5, if you delete a file, the restore point still shows in Backups > Cloud, with not indication that it was been removed. There's also no option to delete/remove. I found the cloud part of the interface a bit quirky to use sometimes, but I could do what I needed. It's been changed now, and I don't know why. IMO it's worse, and there's no way to do what I need to do. I am really, really disappointed with 9.5 so far, which is surprising, because the upgrades since v7 have always been good. I don't understand why these changes were made.
guitarfish
Enthusiast
 
Posts: 67
Liked: 2 times
Joined: Wed Mar 06, 2013 4:12 pm

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Sun Dec 04, 2016 2:40 am

Gostev wrote:OK. Well, if those are the only logs left from your 9.5 install, then there's not much we can do to troubleshoot the issue further... it must be something unique to your environment, because with 3PAR specifically, we see 2x speed improvement backing up from SAN snapshots with 9.5 in our own lab - and other customers with 3PAR are reporting the same increase. So, almost 3x decrease instead is definitely unusual.

Would there be any useful logs we could gather from the physical proxy? Nothing was changed on it other than the software agents, upgraded then reinstall of v9u2.

Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Post 9.5 issues

Veeam Logoby Gostev » Sun Dec 04, 2016 5:51 pm

guitarfish wrote:I am really, really disappointed with 9.5 so far

Sorry to hear that. Purely by downloads to support cases ratio after almost 40000 downloads, 9.5 has been the highest quality release we ever had - and by far. But of course, you can't see that from the forums, as people rarely come here to say how happy they are with the new version ;)

guitarfish wrote:I don't understand why these changes were made.

Cloud Connect was rearchitected pretty heavily in 9.5 to support a number of new features on the service provider side (such as scale-out backup repository, per-VM backup file chains, advanced ReFS integration and so on). These features will improve performance and reduce your Cloud Connect bill, so there's a good reason why these changes are being made. Of course, as with any significant development, there is a chance of bugs introduced - be sure to report them through support, and we will address promptly in the upcoming updates!

SyNtAxx wrote:Would there be any useful logs we could gather from the physical proxy? Nothing was changed on it other than the software agents, upgraded then reinstall of v9u2.

Yes, but only if the issue is obvious (such as if NBD instead of direct SAN transport was used). If you are completely sure the proxy was using the correct transport mode (backup from storage snapshots), then the next troubleshooting step would be to collect performance debug logs - and this requires live deployment.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Post 9.5 issues

Veeam Logoby savmil » Sun Dec 04, 2016 11:19 pm

Hi all

In upgrading to 9.5 we've experienced a few issues (so far):

* Bug discovered with our Secondary Tape jobs (Case # 01990047) --- it is now writing a Monthly tape every day with zero data written (Currently under investigation by Veeam) on top of my daily incremental.
* Tape job properties have been reset to default, especially with the eject media checked. We run multiple tape jobs in sequence and this had caused tape lock errors.
* Backup Job --> Active Full backup Custom schedules in the Advanced Storage tab have been reset back to default (first Monday every month) for every single job. I had discovered End-Of-Year backups running this morning instead of next month wasting space (both disk and tape) and time to reset the jobs.

Anyone else a victim of this ??

Cheers
Milan
savmil
Novice
 
Posts: 3
Liked: never
Joined: Thu Dec 01, 2016 10:39 pm
Full Name: Mil Sav

Re: Post 9.5 issues

Veeam Logoby Butha » Mon Dec 05, 2016 4:31 am

Only had 2 issues so far, and a 3rd one that I was hoping might be solved by 9.5, but seems not.

One issue is more on the VeeamOne side - I'll mention it anyways for anybody reading this although it should go in that thread.

We run VeeamOne, Enterprise Manage and B&R on same VM host (only management, no proxy or transport role)

1. VeeamOne Report server service didn't "close down" on upgrade, you could continue but afterwards it fails to start - the solution was a registry entry to increase timeout values to start services - very strange indeed that this is needed, but it solved it.

2. The Remote Consoles for both products now regularly "disconnects" and you have to reconnect/close down and re-open - quite annoying more than anything else - doesn't effect functionality.

3. We have a case open for a while now being unable to replicate a VM with a virtual disk >2TB (in fact it's 3TB) using Hotadd mode (and Storage Snapshots). The writer process at target san "stalls" although it doesn't log this - only workaround is using network mode on the target proxy (We use Netapp + storage integration both source/target) - which runs slower than hotadd, and affects all other replication jobs using that same proxy. Many hours spend by support running "benchmarks" on storage - but we have no issues with any other jobs all running at the same time - infrastructure hasn't changed (in fact target SAN has doubled in spindle count - other jobs increased in speed). Initial responses centered around Vmware using "SESPARSE" mode for disks >2TB - and than Veeam or the vddk library used handles this differently, and was hoping in 9.5 it would work fine, but not the case. Engineer still busy though so cannot comment on possible fix.

Other than that no issues.
Butha
Enthusiast
 
Posts: 28
Liked: 11 times
Joined: Wed Oct 03, 2012 10:59 am
Full Name: Butha van der Merwe

Re: Post 9.5 issues

Veeam Logoby McClane » Mon Dec 05, 2016 11:49 am 1 person likes this post

savmil wrote:
* Bug discovered with our Secondary Tape jobs (Case # 01990047) --- it is now writing a Monthly tape every day with zero data written (Currently under investigation by Veeam) on top of my daily incremental.

Hi,

Link them to ticket #01981228. I had the same problem and got a hotfix.
McClane
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Sat Jun 20, 2009 12:47 pm

Re: Post 9.5 issues

Veeam Logoby wsuarez » Mon Dec 05, 2016 1:01 pm

We have seen the performance drops (significant) along with multiple job corruption and the inability to restore from existing backups. Also seeing issues where jobs with stop writing output data to Veeam Cloud and eventually fail with "closed connections".

Cases are opened and have been escalated. Support has been very good working with us but as of yet we have no root cause or resolution.
wsuarez
Lurker
 
Posts: 2
Liked: never
Joined: Thu Feb 23, 2012 1:41 pm
Full Name: Bill Suarez

Re: Post 9.5 issues

Veeam Logoby wsuarez » Mon Dec 05, 2016 1:09 pm

When a job times out, after a delay with no data being written, we see: "Error: Transmission pipeline hanged, aborting process"
wsuarez
Lurker
 
Posts: 2
Liked: never
Joined: Thu Feb 23, 2012 1:41 pm
Full Name: Bill Suarez

Re: Post 9.5 issues

Veeam Logoby plandata_at » Mon Dec 05, 2016 2:08 pm

Backup 2 Tape Job Speed before upgrade to 9.5 was about 520MB/sec, after upgrade it's less then 300Mb/sec, we doubled the amount of time tape Jobs Need. Open ticket for days now, but still no solution. (backup from direct attached sas reporitory to direct attached sas tape drives with parallel processing enabled) case nr. #01985204

I don't know if it has todo anything with veeam 9.5, but before update all backups from secondary snapvault Destination (NetApp) worked fine, after upgrade we started to get NFS Status Code 2 error on several Jobs on several vmdks (always the same vmdks) case #01991750. Veeam Support says it's a NetApp Problem, but i can boot the vmdk without any Problem from snapvault detsination when i set it writable. .....

From my Point of view. if you don't need feautires in 9.5, i couldn't experience any little improvement in performance. So bether wait.
plandata_at
Enthusiast
 
Posts: 66
Liked: 10 times
Joined: Tue Jan 26, 2016 2:48 pm
Full Name: Plandata Datenverarbeitungs GmbH

Re: Post 9.5 issues

Veeam Logoby skrause » Mon Dec 05, 2016 2:24 pm

So I have a thread about this already, but since 9.5, most of my GFS Backup Copy jobs (copy entire from source) are now running a full on both Saturday and Sunday when they are set to select a restore point closest to 0200-0400 on Sunday. Previously they had been running on Saturday which was a different issue as I need these to copy the data backed up during the Saturday night backup window. Now I have lost 2 weeks of my 5 weeks of archive backup points because they were removed per the retention policy automatically.

Case# 01994952

Otherwise, everything seems to be working just fine.
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 296
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: Post 9.5 issues

Veeam Logoby sullivas » Mon Dec 05, 2016 5:24 pm 1 person likes this post

savmil wrote:
* Backup Job --> Active Full backup Custom schedules in the Advanced Storage tab have been reset back to default (first Monday every month) for every single job. I had discovered End-Of-Year backups running this morning instead of next month wasting space (both disk and tape) and time to reset the jobs.

Milan


I just discovered this bug with my backup jobs as well. Today being the first Monday of December EVERYTHING is trying to do an active full this morning. No bueno....waiting for the jobs to terminate fully now before trying to clean things up so I don't run out of disk space on my repository from the unexpected influx of data.
sullivas
Novice
 
Posts: 9
Liked: 1 time
Joined: Thu Nov 13, 2014 7:14 pm

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Asahi, chrisit, Dschano, Egor Yakovlev and 43 guests