Post 9.5 issues

Availability for the Always-On Enterprise

Post 9.5 issues

Veeam Logoby Peter Just » Tue Nov 29, 2016 12:21 am

Has anyone experienced any issues after upgrading to 9.5? We upgraded from 9.0 last week. 9.0 worked very well, no complaints. With 9.5 we are observing slow response/sluggishness with the Backup Server and console. Throughput from the proxies is diminished. Jobs that usually took 40-50 minutes on average are at times taking longer than that or timing out. As a result, I am seeing more jobs running concurrently which is impacting things as well. I have a ticket ID# 01983297. However, we are not making much progress.

Any help would be appreciated.
Peter Just
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri Apr 04, 2014 2:29 am
Full Name: Peter Just

Re: Post 9.5 issues

Veeam Logoby mkretzer » Tue Nov 29, 2016 10:03 am

The only slowdown we experienced is in the restore context (Backups/Disk). Everything else is very fast..
mkretzer
Expert
 
Posts: 306
Liked: 67 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Post 9.5 issues

Veeam Logoby foggy » Tue Nov 29, 2016 10:32 am

Hi Peter, the behavior you're describing is not expected, so please continue looking into this with the help of our engineers. Meanwhile, you can check whether the same transport modes are used by proxies, probably temporarily decrease the max number of assigned tasks to involved servers. Is your backup server used as proxy as well? What about the amount of RAM/CPU utilized on it during backups?
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Post 9.5 issues

Veeam Logoby Gostev » Tue Nov 29, 2016 1:08 pm

Peter Just wrote:With 9.5 we are observing slow response/sluggishness with the Backup Server and console.

Did you wait enough time after finishing setup for the configuration database update to complete, as per the upgrade procedure?
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Post 9.5 issues

Veeam Logoby Peter Just » Tue Nov 29, 2016 1:48 pm 1 person likes this post

Gostev- I believe so. Is there a way to verify whether it ran properly?
Peter Just
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri Apr 04, 2014 2:29 am
Full Name: Peter Just

Re: Post 9.5 issues

Veeam Logoby foggy » Tue Nov 29, 2016 5:57 pm

I believe support will be able to verify that.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Tue Nov 29, 2016 7:20 pm

Mine seems to be sluggish/slow. I rebooted again and we'll see how it plays out. I already know my upgrade apparently didn't complete properly as I had about 8 jobs that failed to update the DB entries and needed New jobs created and Active fulls run. I was unable to use the existing chains.

-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Wed Nov 30, 2016 12:57 am

Ok, the rest of my jobs disappeared from the DB and now run as fulls as if the jobs never existed. Not happy!
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Post 9.5 issues

Veeam Logoby v.Eremin » Wed Nov 30, 2016 11:35 am

Then, it's high time you contact our support team for further investigation. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Wed Nov 30, 2016 2:26 pm

Already have a case open and a dedicated thread under vmware.

-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Post 9.5 issues

Veeam Logoby Peter Just » Thu Dec 01, 2016 9:57 pm

Just giving this a bump. I worked with support and we've implemented a few fixes. However, I am still seeing issues with throughput and overall backup performance. Has anyone experienced issues with degraded performance while using DirectSan and 9.5?
Peter Just
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri Apr 04, 2014 2:29 am
Full Name: Peter Just

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Fri Dec 02, 2016 4:30 pm 1 person likes this post

I've abandoned Veeam v9.5 and rolled back to v9u2 which in itself was a process of recreating all my jobs, somewhere around 125ish.

In addition, prior to rolling back I was able to compare job logs and saw a 2/3 decrease in performance. Prior to upgrade one job ran between 1GB/sec and 650MB/sec, post upgrade 100-250MB/sec. After the rollback my performance was again at the 1GB/sec and 650MB/sec levels. Nothing in the environment changed except the software.


-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Post 9.5 issues

Veeam Logoby guitarfish » Fri Dec 02, 2016 6:18 pm

I upgraded to v9.5 a few days ago. I run Backup Copy jobs to a Veeam cloud provider for DR replication purposes, and I had to wait for them to upgrade to 9.5 first. My issues:

#1 (MAJOR) - The Backup Copy process uploaded new Full backups for 2 VMs instead of incremental, and it did this 2x for one VM, 3x for the other. Now finally these are running incremental. It was a painful process, as these uploads spilled over into the workday and severely slowed internet access for the users. I have a case open, but don’t know if I’ll pursue it, as long as things run incremental as required.

#2 (MINOR) - Not sure how to describe this one. I have 4 sites. HQ has a number of hosts & VM backup jobs, the other 3 sites have 1 VM/backup job. As I upgraded each host to v9.5, a Components Update screen appeared which listed all Backup Copy jobs, and indicated “Legacy backup”, they needed to be upgraded.
Image

I upgraded the jobs that pertained to the site being upgraded as I did each site, and that seemed to work.
Now all upgrades are done now, but each site thinks the Backup Copy jobs for the other sites still need to be upgraded. It still shows them as Legacy backup even though they are upgraded. Rescanning the repository doesn’t help. And if I try to run the upgrade again, it fails with an error that “Unable to map service provider side to tenant cache because it is already mapped...”

#3 (MINOR) – Probably a consequence of the previous issue. Backup Copy jobs show in duplicate, with one having a yellow mark, for any jobs that Veeam thinks are still in Legacy mode.
ImagePrevious versions of Veeam would had a “Remove from configuration” option but that’s not available anymore. I’m not clicking Delete from Disk until I talk to Support.

In summary – no issues so far with local backup, but cloud BC jobs have not gone well.
guitarfish
Enthusiast
 
Posts: 67
Liked: 2 times
Joined: Wed Mar 06, 2013 4:12 pm

Re: Post 9.5 issues

Veeam Logoby Gostev » Fri Dec 02, 2016 7:19 pm

SyNtAxx wrote:I've abandoned Veeam v9.5 and rolled back to v9u2 which in itself was a process of recreating all my jobs, somewhere around 125ish.

In addition, prior to rolling back I was able to compare job logs and saw a 2/3 decrease in performance. Prior to upgrade one job ran between 1GB/sec and 650MB/sec, post upgrade 100-250MB/sec. After the rollback my performance was again at the 1GB/sec and 650MB/sec levels. Nothing in the environment changed except the software.

Nick, this is strange as vast majority of users who have upgraded to 9.5 are reporting the exact opposite, 50-100% performance improvement. Can we see the logs to understand the difference between how the two version behave in your environment? If you opened a support case for this, what was the case ID?
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Post 9.5 issues

Veeam Logoby SyNtAxx » Fri Dec 02, 2016 8:33 pm

Gostev,

Case # 01984571

Unfortunately I removed that snapshot after I rolled back to the v9u2 snap I made before upgrading. I did upload some logs for that case, but I don't think they where jobs logs from what I can recall. the best I might be able to do describe my environment.

1) Virtual Veeam Manager, proxy function disabled. Purely logistical operation.
2) 1 Mega physical proxy, 384GB RAM, 28 Physical cores + HT, 20Gbps LACP ethernet, 2 @ 8Gbps FC to the SAN Fabric
3) 3PAR V400 AND 7400
4) HPE StoreOnce 6500 @ 1PB disk: Proxy is set to use SMB/CIFS shares instead of catalyst because of lack of catalyst copy.
5) I use Direct SAN backups.

I wish I held on to the snap but I didnt :-/

NIck
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google Feedfetcher and 30 guests