Discussions specific to the VMware vSphere hypervisor
lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 32 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: Post v8 Upgrade Observations

Post by lp@albersdruck.de » Nov 29, 2014 7:32 am 1 person likes this post

Just to chime in: the hotfix did resolve my problem. And so far has had no side effects (well after 12 hours :P )

Gostev
SVP, Product Management
Posts: 24169
Liked: 3299 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post v8 Upgrade Observations

Post by Gostev » Nov 29, 2014 7:50 pm

Thanks for the update, Lars. This sounds promising!

DiabInc
Novice
Posts: 4
Liked: never
Joined: Mar 16, 2014 4:19 am
Full Name: Mats Bengtsson
Contact:

Re: Post v8 Upgrade Observations

Post by DiabInc » Nov 29, 2014 11:09 pm

Minor bug report Veeam 8.

I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.

Vitaliy S.
Product Manager
Posts: 22521
Liked: 1471 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Post v8 Upgrade Observations

Post by Vitaliy S. » Nov 30, 2014 1:30 pm

Thanks for the heads up, Mats.

hyvokar
Expert
Posts: 346
Liked: 21 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Post v8 Upgrade Observations

Post by hyvokar » Dec 01, 2014 9:27 am

Gostev wrote: Please note that sequential write workload cannot represent storage performance, which is measured in IOPS. However, backup storage is not necessarily causing the issue anyway, see what the backup job reports as a bottleneck.
Hi!

This is absolutely true, but since nothing else has changed I just made sure that storage was ok.
It seems that the speed is permanently dropped, so backing up my file server takes now 70-80minutes instead of 25-35 minutes. It doesnt seem to have much impact if vbr has to transfer 7GB or 70GB. Anyway, I can wait for the patch #1, since my backup window is quite big.

Other thing I noticed, all my backup copy jobs to tape device fails now. Opened a Support case on this one.

Tape library: IBM 3573-TL A.40
Used tapes: FOE329L4 [FOE329L4]
GetTapeHeader failed ChannelError: ConnectionReset
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

v.eremin
Product Manager
Posts: 16325
Liked: 1343 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Post v8 Upgrade Observations

Post by v.eremin » Dec 01, 2014 9:33 am

Your issue seems to be similar to the one discussed here; might be worth posting your ticket number there in order to get a private fix as soon as possible. Thanks.

hyvokar
Expert
Posts: 346
Liked: 21 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Post v8 Upgrade Observations

Post by hyvokar » Dec 01, 2014 1:41 pm 1 person likes this post

Got the fix from Support and tape jobs are working now.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

jja
Enthusiast
Posts: 45
Liked: 8 times
Joined: Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen
Contact:

Re: Post v8 Upgrade Observations (Case # 00691208)

Post by jja » Dec 02, 2014 10:34 am

I registered a case, and uploaded log zip to your ftp.
Hopefully it's the same issue as Lars Pisanec had.

-j

patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 » Dec 02, 2014 2:37 pm

I am seeing the same backup time issues. They are taking way longer in the two environments I've upgraded to Veeam 8 on. A backup that used to take an hour on Veeam 7 now takes 3.5 hours. I am actually getting better throughput on veeam 8 at 102 MB/s vs Veeam 7 where the processing rate was 45-55 MB/s. Should I open a support case on this?

Vitaliy S.
Product Manager
Posts: 22521
Liked: 1471 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Post v8 Upgrade Observations

Post by Vitaliy S. » Dec 02, 2014 3:01 pm

Yes, you can and send all job logs to review bottleneck stats, amount of changed data and other statistics that would allow us to understand where this difference comes from.

patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 » Dec 02, 2014 3:36 pm

I am using Forward Incremental keeping 60 days of retention if that helps. It is not a SAN environment (Dell PowerEdge server approx 6 months old with 16 drives and plenty of iOPS). We are using vCenter 5.5 Update 2. Nothing changed infrastructure wise prior to upgrading to Veeam 8.

Thanks,
Patrick

JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO » Dec 03, 2014 11:29 am

@patrickl78 - my issues seems more or less identical to yours, forward incremental, local disk and 5.5u2. No infrastructure changes prior to upgrade.

I´m currently working with Veeam support on this. I actually installed the fix for bug 38623, wich improved performance a bit (from 4-5 times slower to appr. 2-3 times slower).

Still - I installed v8 expecting better performance and not much worse. Currently waiting for support log analysis.

patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 » Dec 03, 2014 2:47 pm

Jimmy,

I have a case open as well and have sent in the logs for the two servers I upgraded. On both servers I upgraded I also run replication jobs and those jobs get the weird Exchange truncating error even though the servers have never had Exchange on them.

Thanks,
Patrick

jja
Enthusiast
Posts: 45
Liked: 8 times
Joined: Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen
Contact:

Re: Post v8 Upgrade Observations

Post by jja » Dec 04, 2014 6:27 am 2 people like this post

Implemented the hotfix yesterday.
The backups used about 3 hours less time now, so this seems to have done the trick.

-j

patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 » Dec 05, 2014 3:16 am

Applied the bug fix today with Veeam support. Backups ran successfully but still 3.5 hours. It cut the backup window down by an hour but no where near the usual 1 hour and 15 min on Veeam 7. Support did confirm I was impacted by the issue the bug fix was targeted at but it has not alleviated all of my issues so far. There is a second bug that there is currently no fix for as i understand it from support. I'll upload my logs again to them once this job finishes and hopefully will help them figure out what is going on. There have been no changes to my network infrastructure or backup job prior to the upgrade.

JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO » Dec 05, 2014 7:34 am

Patrick - again it seems that we have the exact same issue. Working with the support yesterday (2:nd line), it seems that for many vm:s, a lot (if not all) of the added time is spent waiting after the snapshot is taken and before the vmx-copy begins. The problem is that the log is empty during this time, sometimes up to 5 minutes where nothing happens, and no resources on the server is in use. This issue was forwared to the developers yesterday.

patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 » Dec 06, 2014 6:01 am

Jimmy,

Thanks for the info. I plan on uploading more logs to them this weekend as well for analysis.

Thanks,
Patrick

jplavertu
Novice
Posts: 4
Liked: 1 time
Joined: Dec 19, 2014 3:17 pm
Contact:

Re: Post v8 Upgrade Observations

Post by jplavertu » Dec 19, 2014 3:23 pm

I seem to have similar issues. Mind you, I didn't upgrade from v7, I installed a new server and proxy, use the same configuration settings. Backing up the same VM, my Veeam v7 does it in 4 mins at 372 MB/s. My Veeam v8 setup does it in 18 mins, capped at 52 MB/s...

Both backups were started from scratch with no previous restore points.
Bottleneck is showing : source at 99%.

I double checked everything, iSCSI initiators, transport mode, firewall, nothing seems different except from the fact that the new server is physical and has such more resources to itself .

Source is on a HP P2000 G3 10GbE 24 drives SAN , target is a QNAP 1279URP with 12 drives , all connected on 10GbE fabric
Both backup jobs ( v7 and v8 ) have the same source and same target.

Gostev
SVP, Product Management
Posts: 24169
Liked: 3299 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post v8 Upgrade Observations

Post by Gostev » Dec 20, 2014 6:08 pm

Source bottleneck at 99% is never a Veeam issue. Direct SAN access mode capped at 52MB/s on 10Gb Ethernet would make me look at the entire networking stack between a backup proxy and a SAN. The simplest test would be to create an NTFS LUN on the SAN, and copying a large file from that LUN down to the backup proxy's local disk.

jplavertu
Novice
Posts: 4
Liked: 1 time
Joined: Dec 19, 2014 3:17 pm
Contact:

Re: Post v8 Upgrade Observations

Post by jplavertu » Dec 22, 2014 4:31 pm

Thank you for replying.

I did a few tests.

Backing up same VM
On my v7 setup : network utilization is on the iSCSI interface ( 10Gb/s ) for the Send ( Repository which is a iSCSI LUN and a NAS ) and Receive ( from the ESXI datastore )
On my v8 setup : Send and Receive are split on both the LAN interface ( 1Gbps ) and the SAN interface ( iSCSI 10GbE ), that is where the cap is.

Unfortunately, I cannot create an NFS LUN on the production SAN but I did transfer a large file from an NFS LUN on a NAS to the proxy ( proxy is hosted on the SAN, it's a VM ) and it was 300 MB/s ...

jplavertu
Novice
Posts: 4
Liked: 1 time
Joined: Dec 19, 2014 3:17 pm
Contact:

Re: Post v8 Upgrade Observations

Post by jplavertu » Dec 22, 2014 10:20 pm 1 person likes this post

Problem fixed.

Veeam Proxy wasn't configured properly on the SAN side.

I see 250+MB/s now :)

aidechoute
Influencer
Posts: 13
Liked: 1 time
Joined: Sep 12, 2011 8:11 am
Contact:

Re: Post v8 Upgrade Observations

Post by aidechoute » Jan 07, 2015 1:14 pm

Hi,
I upgrade veeam from V7 to v8.
Nothing was changed in job settings....but all my job do a full backup in a new folder at the first running in Veeam V8

Do you have this issue? Is it normal (i don't think so)

Thanks

foggy
Veeam Software
Posts: 17908
Liked: 1506 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Post v8 Upgrade Observations

Post by foggy » Jan 07, 2015 2:07 pm

This is definitely not expected. Looks like previous backups were not recognized (like if you use Remove from backups command to remove them from Veeam B&R console). Have you contacted technical support already?

aidechoute
Influencer
Posts: 13
Liked: 1 time
Joined: Sep 12, 2011 8:11 am
Contact:

Re: Post v8 Upgrade Observations

Post by aidechoute » Jan 07, 2015 3:24 pm

I will ask the end user to contact the support for this issue. (I'm currently not on site)
Thanks.
I will update the topic in order to help other user :p

JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO » Jan 13, 2015 1:25 pm 1 person likes this post

@patrikl78
Finally my performance problem with v8 has been solved (new version of VeeamAgent.exe).
Performance is now back to v7 level, or possibly al litte faster.

Gostev
SVP, Product Management
Posts: 24169
Liked: 3299 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post v8 Upgrade Observations

Post by Gostev » Jan 13, 2015 10:10 pm

@JimmyO what was the support case ID for this last case?

JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO » Jan 14, 2015 7:13 am

@Gostev - case 686930

However - I have not installed update1 and have not confirmed if the new veeamagent works with this.
Also - I have no confirmation if this new veeamagent will be included in update 2.

It would be much appreciated if I could get clarification of these questions.

JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO » Mar 02, 2015 8:21 am

I have now got confirmation that the working veeamagent (with performance fix) will be included in patch2 wich (probably) will be released in March.

DiabInc
Novice
Posts: 4
Liked: never
Joined: Mar 16, 2014 4:19 am
Full Name: Mats Bengtsson
Contact:

Re: Post v8 Upgrade Observations

Post by DiabInc » Oct 09, 2015 12:18 am

DiabInc wrote:Minor bug report Veeam 8.

I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.
This issue still exists in last update 3.

foggy
Veeam Software
Posts: 17908
Liked: 1506 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Post v8 Upgrade Observations

Post by foggy » Oct 09, 2015 11:41 am

Right, was not fixed in this update, however, still on our radar.

Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests