-
- Enthusiast
- Posts: 82
- Liked: 33 times
- Joined: Mar 25, 2013 7:37 pm
- Full Name: Lars Pisanec
- Contact:
Re: Post v8 Upgrade Observations
Just to chime in: the hotfix did resolve my problem. And so far has had no side effects (well after 12 hours )
-
- Chief Product Officer
- Posts: 31623
- Liked: 6783 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Post v8 Upgrade Observations
Thanks for the update, Lars. This sounds promising!
-
- Novice
- Posts: 4
- Liked: never
- Joined: Mar 16, 2014 4:19 am
- Full Name: Mats Bengtsson
- Contact:
Re: Post v8 Upgrade Observations
Minor bug report Veeam 8.
I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.
I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.
-
- VP, Product Management
- Posts: 27156
- Liked: 2729 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Post v8 Upgrade Observations
Thanks for the heads up, Mats.
-
- Veteran
- Posts: 409
- Liked: 30 times
- Joined: Nov 21, 2014 10:05 pm
- Contact:
Re: Post v8 Upgrade Observations
Hi!Gostev wrote: Please note that sequential write workload cannot represent storage performance, which is measured in IOPS. However, backup storage is not necessarily causing the issue anyway, see what the backup job reports as a bottleneck.
This is absolutely true, but since nothing else has changed I just made sure that storage was ok.
It seems that the speed is permanently dropped, so backing up my file server takes now 70-80minutes instead of 25-35 minutes. It doesnt seem to have much impact if vbr has to transfer 7GB or 70GB. Anyway, I can wait for the patch #1, since my backup window is quite big.
Other thing I noticed, all my backup copy jobs to tape device fails now. Opened a Support case on this one.
Tape library: IBM 3573-TL A.40
Used tapes: FOE329L4 [FOE329L4]
GetTapeHeader failed ChannelError: ConnectionReset
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
-
- Product Manager
- Posts: 20286
- Liked: 2258 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Post v8 Upgrade Observations
Your issue seems to be similar to the one discussed here; might be worth posting your ticket number there in order to get a private fix as soon as possible. Thanks.
-
- Veteran
- Posts: 409
- Liked: 30 times
- Joined: Nov 21, 2014 10:05 pm
- Contact:
Re: Post v8 Upgrade Observations
Got the fix from Support and tape jobs are working now.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
-
- Enthusiast
- Posts: 46
- Liked: 8 times
- Joined: Nov 13, 2013 6:40 am
- Full Name: Jannis Jacobsen
- Contact:
Re: Post v8 Upgrade Observations (Case # 00691208)
I registered a case, and uploaded log zip to your ftp.
Hopefully it's the same issue as Lars Pisanec had.
-j
Hopefully it's the same issue as Lars Pisanec had.
-j
-
- Service Provider
- Posts: 32
- Liked: 7 times
- Joined: Jan 09, 2014 11:16 pm
- Full Name: Patrick Leonard
- Contact:
Re: Post v8 Upgrade Observations
I am seeing the same backup time issues. They are taking way longer in the two environments I've upgraded to Veeam 8 on. A backup that used to take an hour on Veeam 7 now takes 3.5 hours. I am actually getting better throughput on veeam 8 at 102 MB/s vs Veeam 7 where the processing rate was 45-55 MB/s. Should I open a support case on this?
-
- VP, Product Management
- Posts: 27156
- Liked: 2729 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Post v8 Upgrade Observations
Yes, you can and send all job logs to review bottleneck stats, amount of changed data and other statistics that would allow us to understand where this difference comes from.
-
- Service Provider
- Posts: 32
- Liked: 7 times
- Joined: Jan 09, 2014 11:16 pm
- Full Name: Patrick Leonard
- Contact:
Re: Post v8 Upgrade Observations
I am using Forward Incremental keeping 60 days of retention if that helps. It is not a SAN environment (Dell PowerEdge server approx 6 months old with 16 drives and plenty of iOPS). We are using vCenter 5.5 Update 2. Nothing changed infrastructure wise prior to upgrading to Veeam 8.
Thanks,
Patrick
Thanks,
Patrick
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: Post v8 Upgrade Observations
@patrickl78 - my issues seems more or less identical to yours, forward incremental, local disk and 5.5u2. No infrastructure changes prior to upgrade.
I´m currently working with Veeam support on this. I actually installed the fix for bug 38623, wich improved performance a bit (from 4-5 times slower to appr. 2-3 times slower).
Still - I installed v8 expecting better performance and not much worse. Currently waiting for support log analysis.
I´m currently working with Veeam support on this. I actually installed the fix for bug 38623, wich improved performance a bit (from 4-5 times slower to appr. 2-3 times slower).
Still - I installed v8 expecting better performance and not much worse. Currently waiting for support log analysis.
-
- Service Provider
- Posts: 32
- Liked: 7 times
- Joined: Jan 09, 2014 11:16 pm
- Full Name: Patrick Leonard
- Contact:
Re: Post v8 Upgrade Observations
Jimmy,
I have a case open as well and have sent in the logs for the two servers I upgraded. On both servers I upgraded I also run replication jobs and those jobs get the weird Exchange truncating error even though the servers have never had Exchange on them.
Thanks,
Patrick
I have a case open as well and have sent in the logs for the two servers I upgraded. On both servers I upgraded I also run replication jobs and those jobs get the weird Exchange truncating error even though the servers have never had Exchange on them.
Thanks,
Patrick
-
- Enthusiast
- Posts: 46
- Liked: 8 times
- Joined: Nov 13, 2013 6:40 am
- Full Name: Jannis Jacobsen
- Contact:
Re: Post v8 Upgrade Observations
Implemented the hotfix yesterday.
The backups used about 3 hours less time now, so this seems to have done the trick.
-j
The backups used about 3 hours less time now, so this seems to have done the trick.
-j
-
- Service Provider
- Posts: 32
- Liked: 7 times
- Joined: Jan 09, 2014 11:16 pm
- Full Name: Patrick Leonard
- Contact:
Re: Post v8 Upgrade Observations
Applied the bug fix today with Veeam support. Backups ran successfully but still 3.5 hours. It cut the backup window down by an hour but no where near the usual 1 hour and 15 min on Veeam 7. Support did confirm I was impacted by the issue the bug fix was targeted at but it has not alleviated all of my issues so far. There is a second bug that there is currently no fix for as i understand it from support. I'll upload my logs again to them once this job finishes and hopefully will help them figure out what is going on. There have been no changes to my network infrastructure or backup job prior to the upgrade.
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: Post v8 Upgrade Observations
Patrick - again it seems that we have the exact same issue. Working with the support yesterday (2:nd line), it seems that for many vm:s, a lot (if not all) of the added time is spent waiting after the snapshot is taken and before the vmx-copy begins. The problem is that the log is empty during this time, sometimes up to 5 minutes where nothing happens, and no resources on the server is in use. This issue was forwared to the developers yesterday.
-
- Service Provider
- Posts: 32
- Liked: 7 times
- Joined: Jan 09, 2014 11:16 pm
- Full Name: Patrick Leonard
- Contact:
Re: Post v8 Upgrade Observations
Jimmy,
Thanks for the info. I plan on uploading more logs to them this weekend as well for analysis.
Thanks,
Patrick
Thanks for the info. I plan on uploading more logs to them this weekend as well for analysis.
Thanks,
Patrick
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Dec 19, 2014 3:17 pm
- Contact:
Re: Post v8 Upgrade Observations
I seem to have similar issues. Mind you, I didn't upgrade from v7, I installed a new server and proxy, use the same configuration settings. Backing up the same VM, my Veeam v7 does it in 4 mins at 372 MB/s. My Veeam v8 setup does it in 18 mins, capped at 52 MB/s...
Both backups were started from scratch with no previous restore points.
Bottleneck is showing : source at 99%.
I double checked everything, iSCSI initiators, transport mode, firewall, nothing seems different except from the fact that the new server is physical and has such more resources to itself .
Source is on a HP P2000 G3 10GbE 24 drives SAN , target is a QNAP 1279URP with 12 drives , all connected on 10GbE fabric
Both backup jobs ( v7 and v8 ) have the same source and same target.
Both backups were started from scratch with no previous restore points.
Bottleneck is showing : source at 99%.
I double checked everything, iSCSI initiators, transport mode, firewall, nothing seems different except from the fact that the new server is physical and has such more resources to itself .
Source is on a HP P2000 G3 10GbE 24 drives SAN , target is a QNAP 1279URP with 12 drives , all connected on 10GbE fabric
Both backup jobs ( v7 and v8 ) have the same source and same target.
-
- Chief Product Officer
- Posts: 31623
- Liked: 6783 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Post v8 Upgrade Observations
Source bottleneck at 99% is never a Veeam issue. Direct SAN access mode capped at 52MB/s on 10Gb Ethernet would make me look at the entire networking stack between a backup proxy and a SAN. The simplest test would be to create an NTFS LUN on the SAN, and copying a large file from that LUN down to the backup proxy's local disk.
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Dec 19, 2014 3:17 pm
- Contact:
Re: Post v8 Upgrade Observations
Thank you for replying.
I did a few tests.
Backing up same VM
On my v7 setup : network utilization is on the iSCSI interface ( 10Gb/s ) for the Send ( Repository which is a iSCSI LUN and a NAS ) and Receive ( from the ESXI datastore )
On my v8 setup : Send and Receive are split on both the LAN interface ( 1Gbps ) and the SAN interface ( iSCSI 10GbE ), that is where the cap is.
Unfortunately, I cannot create an NFS LUN on the production SAN but I did transfer a large file from an NFS LUN on a NAS to the proxy ( proxy is hosted on the SAN, it's a VM ) and it was 300 MB/s ...
I did a few tests.
Backing up same VM
On my v7 setup : network utilization is on the iSCSI interface ( 10Gb/s ) for the Send ( Repository which is a iSCSI LUN and a NAS ) and Receive ( from the ESXI datastore )
On my v8 setup : Send and Receive are split on both the LAN interface ( 1Gbps ) and the SAN interface ( iSCSI 10GbE ), that is where the cap is.
Unfortunately, I cannot create an NFS LUN on the production SAN but I did transfer a large file from an NFS LUN on a NAS to the proxy ( proxy is hosted on the SAN, it's a VM ) and it was 300 MB/s ...
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Dec 19, 2014 3:17 pm
- Contact:
Re: Post v8 Upgrade Observations
Problem fixed.
Veeam Proxy wasn't configured properly on the SAN side.
I see 250+MB/s now
Veeam Proxy wasn't configured properly on the SAN side.
I see 250+MB/s now
-
- Influencer
- Posts: 18
- Liked: 1 time
- Joined: Sep 12, 2011 8:11 am
- Contact:
Re: Post v8 Upgrade Observations
Hi,
I upgrade veeam from V7 to v8.
Nothing was changed in job settings....but all my job do a full backup in a new folder at the first running in Veeam V8
Do you have this issue? Is it normal (i don't think so)
Thanks
I upgrade veeam from V7 to v8.
Nothing was changed in job settings....but all my job do a full backup in a new folder at the first running in Veeam V8
Do you have this issue? Is it normal (i don't think so)
Thanks
-
- Veeam Software
- Posts: 21078
- Liked: 2116 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Post v8 Upgrade Observations
This is definitely not expected. Looks like previous backups were not recognized (like if you use Remove from backups command to remove them from Veeam B&R console). Have you contacted technical support already?
-
- Influencer
- Posts: 18
- Liked: 1 time
- Joined: Sep 12, 2011 8:11 am
- Contact:
Re: Post v8 Upgrade Observations
I will ask the end user to contact the support for this issue. (I'm currently not on site)
Thanks.
I will update the topic in order to help other user :p
Thanks.
I will update the topic in order to help other user :p
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: Post v8 Upgrade Observations
@patrikl78
Finally my performance problem with v8 has been solved (new version of VeeamAgent.exe).
Performance is now back to v7 level, or possibly al litte faster.
Finally my performance problem with v8 has been solved (new version of VeeamAgent.exe).
Performance is now back to v7 level, or possibly al litte faster.
-
- Chief Product Officer
- Posts: 31623
- Liked: 6783 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Post v8 Upgrade Observations
@JimmyO what was the support case ID for this last case?
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: Post v8 Upgrade Observations
@Gostev - case 686930
However - I have not installed update1 and have not confirmed if the new veeamagent works with this.
Also - I have no confirmation if this new veeamagent will be included in update 2.
It would be much appreciated if I could get clarification of these questions.
However - I have not installed update1 and have not confirmed if the new veeamagent works with this.
Also - I have no confirmation if this new veeamagent will be included in update 2.
It would be much appreciated if I could get clarification of these questions.
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: Post v8 Upgrade Observations
I have now got confirmation that the working veeamagent (with performance fix) will be included in patch2 wich (probably) will be released in March.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Mar 16, 2014 4:19 am
- Full Name: Mats Bengtsson
- Contact:
Re: Post v8 Upgrade Observations
This issue still exists in last update 3.DiabInc wrote:Minor bug report Veeam 8.
I was running a SureBackup job when i noticed that progress indicator stayed at 0% in the SureBackup job tab, while the progress indicator (% completed) worked normal in running jobs tab.
-
- Veeam Software
- Posts: 21078
- Liked: 2116 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Post v8 Upgrade Observations
Right, was not fixed in this update, however, still on our radar.
Who is online
Users browsing this forum: InFrance, jevans99vm, Pat490, Paul.Loewenkamp and 86 guests