Comprehensive data protection for all workloads
Peter Just
Influencer
Posts: 11
Liked: 2 times
Joined: Apr 04, 2014 2:29 am
Full Name: Peter Just
Contact:

Post 9.5 issues

Post by Peter Just » Nov 29, 2016 12:21 am

Has anyone experienced any issues after upgrading to 9.5? We upgraded from 9.0 last week. 9.0 worked very well, no complaints. With 9.5 we are observing slow response/sluggishness with the Backup Server and console. Throughput from the proxies is diminished. Jobs that usually took 40-50 minutes on average are at times taking longer than that or timing out. As a result, I am seeing more jobs running concurrently which is impacting things as well. I have a ticket ID# 01983297. However, we are not making much progress.

Any help would be appreciated.

mkretzer
Expert
Posts: 459
Liked: 97 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Post 9.5 issues

Post by mkretzer » Nov 29, 2016 10:03 am

The only slowdown we experienced is in the restore context (Backups/Disk). Everything else is very fast..

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Post 9.5 issues

Post by foggy » Nov 29, 2016 10:32 am

Hi Peter, the behavior you're describing is not expected, so please continue looking into this with the help of our engineers. Meanwhile, you can check whether the same transport modes are used by proxies, probably temporarily decrease the max number of assigned tasks to involved servers. Is your backup server used as proxy as well? What about the amount of RAM/CPU utilized on it during backups?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post 9.5 issues

Post by Gostev » Nov 29, 2016 1:08 pm

Peter Just wrote:With 9.5 we are observing slow response/sluggishness with the Backup Server and console.
Did you wait enough time after finishing setup for the configuration database update to complete, as per the upgrade procedure?

Peter Just
Influencer
Posts: 11
Liked: 2 times
Joined: Apr 04, 2014 2:29 am
Full Name: Peter Just
Contact:

Re: Post 9.5 issues

Post by Peter Just » Nov 29, 2016 1:48 pm 1 person likes this post

Gostev- I believe so. Is there a way to verify whether it ran properly?

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Post 9.5 issues

Post by foggy » Nov 29, 2016 5:57 pm

I believe support will be able to verify that.

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Post 9.5 issues

Post by SyNtAxx » Nov 29, 2016 7:20 pm

Mine seems to be sluggish/slow. I rebooted again and we'll see how it plays out. I already know my upgrade apparently didn't complete properly as I had about 8 jobs that failed to update the DB entries and needed New jobs created and Active fulls run. I was unable to use the existing chains.

-Nick

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Post 9.5 issues

Post by SyNtAxx » Nov 30, 2016 12:57 am

Ok, the rest of my jobs disappeared from the DB and now run as fulls as if the jobs never existed. Not happy!

v.eremin
Product Manager
Posts: 16134
Liked: 1315 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Post 9.5 issues

Post by v.eremin » Nov 30, 2016 11:35 am

Then, it's high time you contact our support team for further investigation. Thanks.

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Post 9.5 issues

Post by SyNtAxx » Nov 30, 2016 2:26 pm

Already have a case open and a dedicated thread under vmware.

-Nick

Peter Just
Influencer
Posts: 11
Liked: 2 times
Joined: Apr 04, 2014 2:29 am
Full Name: Peter Just
Contact:

Re: Post 9.5 issues

Post by Peter Just » Dec 01, 2016 9:57 pm

Just giving this a bump. I worked with support and we've implemented a few fixes. However, I am still seeing issues with throughput and overall backup performance. Has anyone experienced issues with degraded performance while using DirectSan and 9.5?

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Post 9.5 issues

Post by SyNtAxx » Dec 02, 2016 4:30 pm 1 person likes this post

I've abandoned Veeam v9.5 and rolled back to v9u2 which in itself was a process of recreating all my jobs, somewhere around 125ish.

In addition, prior to rolling back I was able to compare job logs and saw a 2/3 decrease in performance. Prior to upgrade one job ran between 1GB/sec and 650MB/sec, post upgrade 100-250MB/sec. After the rollback my performance was again at the 1GB/sec and 650MB/sec levels. Nothing in the environment changed except the software.


-Nick

guitarfish
Enthusiast
Posts: 77
Liked: 2 times
Joined: Mar 06, 2013 4:12 pm
Contact:

Re: Post 9.5 issues

Post by guitarfish » Dec 02, 2016 6:18 pm

I upgraded to v9.5 a few days ago. I run Backup Copy jobs to a Veeam cloud provider for DR replication purposes, and I had to wait for them to upgrade to 9.5 first. My issues:

#1 (MAJOR) - The Backup Copy process uploaded new Full backups for 2 VMs instead of incremental, and it did this 2x for one VM, 3x for the other. Now finally these are running incremental. It was a painful process, as these uploads spilled over into the workday and severely slowed internet access for the users. I have a case open, but don’t know if I’ll pursue it, as long as things run incremental as required.

#2 (MINOR) - Not sure how to describe this one. I have 4 sites. HQ has a number of hosts & VM backup jobs, the other 3 sites have 1 VM/backup job. As I upgraded each host to v9.5, a Components Update screen appeared which listed all Backup Copy jobs, and indicated “Legacy backup”, they needed to be upgraded.
Image

I upgraded the jobs that pertained to the site being upgraded as I did each site, and that seemed to work.
Now all upgrades are done now, but each site thinks the Backup Copy jobs for the other sites still need to be upgraded. It still shows them as Legacy backup even though they are upgraded. Rescanning the repository doesn’t help. And if I try to run the upgrade again, it fails with an error that “Unable to map service provider side to tenant cache because it is already mapped...”

#3 (MINOR) – Probably a consequence of the previous issue. Backup Copy jobs show in duplicate, with one having a yellow mark, for any jobs that Veeam thinks are still in Legacy mode.
ImagePrevious versions of Veeam would had a “Remove from configuration” option but that’s not available anymore. I’m not clicking Delete from Disk until I talk to Support.

In summary – no issues so far with local backup, but cloud BC jobs have not gone well.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post 9.5 issues

Post by Gostev » Dec 02, 2016 7:19 pm

SyNtAxx wrote:I've abandoned Veeam v9.5 and rolled back to v9u2 which in itself was a process of recreating all my jobs, somewhere around 125ish.

In addition, prior to rolling back I was able to compare job logs and saw a 2/3 decrease in performance. Prior to upgrade one job ran between 1GB/sec and 650MB/sec, post upgrade 100-250MB/sec. After the rollback my performance was again at the 1GB/sec and 650MB/sec levels. Nothing in the environment changed except the software.
Nick, this is strange as vast majority of users who have upgraded to 9.5 are reporting the exact opposite, 50-100% performance improvement. Can we see the logs to understand the difference between how the two version behave in your environment? If you opened a support case for this, what was the case ID?

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Post 9.5 issues

Post by SyNtAxx » Dec 02, 2016 8:33 pm

Gostev,

Case # 01984571

Unfortunately I removed that snapshot after I rolled back to the v9u2 snap I made before upgrading. I did upload some logs for that case, but I don't think they where jobs logs from what I can recall. the best I might be able to do describe my environment.

1) Virtual Veeam Manager, proxy function disabled. Purely logistical operation.
2) 1 Mega physical proxy, 384GB RAM, 28 Physical cores + HT, 20Gbps LACP ethernet, 2 @ 8Gbps FC to the SAN Fabric
3) 3PAR V400 AND 7400
4) HPE StoreOnce 6500 @ 1PB disk: Proxy is set to use SMB/CIFS shares instead of catalyst because of lack of catalyst copy.
5) I use Direct SAN backups.

I wish I held on to the snap but I didnt :-/

NIck

Locked

Who is online

Users browsing this forum: Karinne, Shinji and 15 guests