Host-based backup of VMware vSphere VMs.
lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 33 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: Post v8 Upgrade Observations

Post by lp@albersdruck.de » 1 person likes this post

Just to chime in: the hotfix did resolve my problem. And so far has had no side effects (well after 12 hours :P )
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post v8 Upgrade Observations

Post by Gostev »

Thanks for the update, Lars. This sounds promising!
DiabInc
Novice
Posts: 4
Liked: never
Joined: Mar 16, 2014 4:19 am
Full Name: Mats Bengtsson
Contact:

Re: Post v8 Upgrade Observations

Post by DiabInc »

Minor bug report Veeam 8.

I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Post v8 Upgrade Observations

Post by Vitaliy S. »

Thanks for the heads up, Mats.
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Post v8 Upgrade Observations

Post by hyvokar »

Gostev wrote: Please note that sequential write workload cannot represent storage performance, which is measured in IOPS. However, backup storage is not necessarily causing the issue anyway, see what the backup job reports as a bottleneck.
Hi!

This is absolutely true, but since nothing else has changed I just made sure that storage was ok.
It seems that the speed is permanently dropped, so backing up my file server takes now 70-80minutes instead of 25-35 minutes. It doesnt seem to have much impact if vbr has to transfer 7GB or 70GB. Anyway, I can wait for the patch #1, since my backup window is quite big.

Other thing I noticed, all my backup copy jobs to tape device fails now. Opened a Support case on this one.

Tape library: IBM 3573-TL A.40
Used tapes: FOE329L4 [FOE329L4]
GetTapeHeader failed ChannelError: ConnectionReset
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Post v8 Upgrade Observations

Post by veremin »

Your issue seems to be similar to the one discussed here; might be worth posting your ticket number there in order to get a private fix as soon as possible. Thanks.
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Post v8 Upgrade Observations

Post by hyvokar » 1 person likes this post

Got the fix from Support and tape jobs are working now.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
jja
Enthusiast
Posts: 46
Liked: 8 times
Joined: Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen
Contact:

Re: Post v8 Upgrade Observations (Case # 00691208)

Post by jja »

I registered a case, and uploaded log zip to your ftp.
Hopefully it's the same issue as Lars Pisanec had.

-j
patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 »

I am seeing the same backup time issues. They are taking way longer in the two environments I've upgraded to Veeam 8 on. A backup that used to take an hour on Veeam 7 now takes 3.5 hours. I am actually getting better throughput on veeam 8 at 102 MB/s vs Veeam 7 where the processing rate was 45-55 MB/s. Should I open a support case on this?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Post v8 Upgrade Observations

Post by Vitaliy S. »

Yes, you can and send all job logs to review bottleneck stats, amount of changed data and other statistics that would allow us to understand where this difference comes from.
patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 »

I am using Forward Incremental keeping 60 days of retention if that helps. It is not a SAN environment (Dell PowerEdge server approx 6 months old with 16 drives and plenty of iOPS). We are using vCenter 5.5 Update 2. Nothing changed infrastructure wise prior to upgrading to Veeam 8.

Thanks,
Patrick
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO »

@patrickl78 - my issues seems more or less identical to yours, forward incremental, local disk and 5.5u2. No infrastructure changes prior to upgrade.

I´m currently working with Veeam support on this. I actually installed the fix for bug 38623, wich improved performance a bit (from 4-5 times slower to appr. 2-3 times slower).

Still - I installed v8 expecting better performance and not much worse. Currently waiting for support log analysis.
patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 »

Jimmy,

I have a case open as well and have sent in the logs for the two servers I upgraded. On both servers I upgraded I also run replication jobs and those jobs get the weird Exchange truncating error even though the servers have never had Exchange on them.

Thanks,
Patrick
jja
Enthusiast
Posts: 46
Liked: 8 times
Joined: Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen
Contact:

Re: Post v8 Upgrade Observations

Post by jja » 2 people like this post

Implemented the hotfix yesterday.
The backups used about 3 hours less time now, so this seems to have done the trick.

-j
patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 »

Applied the bug fix today with Veeam support. Backups ran successfully but still 3.5 hours. It cut the backup window down by an hour but no where near the usual 1 hour and 15 min on Veeam 7. Support did confirm I was impacted by the issue the bug fix was targeted at but it has not alleviated all of my issues so far. There is a second bug that there is currently no fix for as i understand it from support. I'll upload my logs again to them once this job finishes and hopefully will help them figure out what is going on. There have been no changes to my network infrastructure or backup job prior to the upgrade.
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO »

Patrick - again it seems that we have the exact same issue. Working with the support yesterday (2:nd line), it seems that for many vm:s, a lot (if not all) of the added time is spent waiting after the snapshot is taken and before the vmx-copy begins. The problem is that the log is empty during this time, sometimes up to 5 minutes where nothing happens, and no resources on the server is in use. This issue was forwared to the developers yesterday.
patrickl78
Service Provider
Posts: 32
Liked: 7 times
Joined: Jan 09, 2014 11:16 pm
Full Name: Patrick Leonard
Contact:

Re: Post v8 Upgrade Observations

Post by patrickl78 »

Jimmy,

Thanks for the info. I plan on uploading more logs to them this weekend as well for analysis.

Thanks,
Patrick
jplavertu
Novice
Posts: 4
Liked: 1 time
Joined: Dec 19, 2014 3:17 pm
Contact:

Re: Post v8 Upgrade Observations

Post by jplavertu »

I seem to have similar issues. Mind you, I didn't upgrade from v7, I installed a new server and proxy, use the same configuration settings. Backing up the same VM, my Veeam v7 does it in 4 mins at 372 MB/s. My Veeam v8 setup does it in 18 mins, capped at 52 MB/s...

Both backups were started from scratch with no previous restore points.
Bottleneck is showing : source at 99%.

I double checked everything, iSCSI initiators, transport mode, firewall, nothing seems different except from the fact that the new server is physical and has such more resources to itself .

Source is on a HP P2000 G3 10GbE 24 drives SAN , target is a QNAP 1279URP with 12 drives , all connected on 10GbE fabric
Both backup jobs ( v7 and v8 ) have the same source and same target.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post v8 Upgrade Observations

Post by Gostev »

Source bottleneck at 99% is never a Veeam issue. Direct SAN access mode capped at 52MB/s on 10Gb Ethernet would make me look at the entire networking stack between a backup proxy and a SAN. The simplest test would be to create an NTFS LUN on the SAN, and copying a large file from that LUN down to the backup proxy's local disk.
jplavertu
Novice
Posts: 4
Liked: 1 time
Joined: Dec 19, 2014 3:17 pm
Contact:

Re: Post v8 Upgrade Observations

Post by jplavertu »

Thank you for replying.

I did a few tests.

Backing up same VM
On my v7 setup : network utilization is on the iSCSI interface ( 10Gb/s ) for the Send ( Repository which is a iSCSI LUN and a NAS ) and Receive ( from the ESXI datastore )
On my v8 setup : Send and Receive are split on both the LAN interface ( 1Gbps ) and the SAN interface ( iSCSI 10GbE ), that is where the cap is.

Unfortunately, I cannot create an NFS LUN on the production SAN but I did transfer a large file from an NFS LUN on a NAS to the proxy ( proxy is hosted on the SAN, it's a VM ) and it was 300 MB/s ...
jplavertu
Novice
Posts: 4
Liked: 1 time
Joined: Dec 19, 2014 3:17 pm
Contact:

Re: Post v8 Upgrade Observations

Post by jplavertu » 1 person likes this post

Problem fixed.

Veeam Proxy wasn't configured properly on the SAN side.

I see 250+MB/s now :)
aidechoute
Influencer
Posts: 18
Liked: 1 time
Joined: Sep 12, 2011 8:11 am
Contact:

Re: Post v8 Upgrade Observations

Post by aidechoute »

Hi,
I upgrade veeam from V7 to v8.
Nothing was changed in job settings....but all my job do a full backup in a new folder at the first running in Veeam V8

Do you have this issue? Is it normal (i don't think so)

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Post v8 Upgrade Observations

Post by foggy »

This is definitely not expected. Looks like previous backups were not recognized (like if you use Remove from backups command to remove them from Veeam B&R console). Have you contacted technical support already?
aidechoute
Influencer
Posts: 18
Liked: 1 time
Joined: Sep 12, 2011 8:11 am
Contact:

Re: Post v8 Upgrade Observations

Post by aidechoute »

I will ask the end user to contact the support for this issue. (I'm currently not on site)
Thanks.
I will update the topic in order to help other user :p
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO » 1 person likes this post

@patrikl78
Finally my performance problem with v8 has been solved (new version of VeeamAgent.exe).
Performance is now back to v7 level, or possibly al litte faster.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Post v8 Upgrade Observations

Post by Gostev »

@JimmyO what was the support case ID for this last case?
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO »

@Gostev - case 686930

However - I have not installed update1 and have not confirmed if the new veeamagent works with this.
Also - I have no confirmation if this new veeamagent will be included in update 2.

It would be much appreciated if I could get clarification of these questions.
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Post v8 Upgrade Observations

Post by JimmyO »

I have now got confirmation that the working veeamagent (with performance fix) will be included in patch2 wich (probably) will be released in March.
DiabInc
Novice
Posts: 4
Liked: never
Joined: Mar 16, 2014 4:19 am
Full Name: Mats Bengtsson
Contact:

Re: Post v8 Upgrade Observations

Post by DiabInc »

DiabInc wrote:Minor bug report Veeam 8.

I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.
This issue still exists in last update 3.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Post v8 Upgrade Observations

Post by foggy »

Right, was not fixed in this update, however, still on our radar.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot] and 80 guests