[ALL UPDATES] Known Issues and Fixes

[Gostev]

Topic for tracking common issues, hot fixes and patches for Veeam Backup & Replication 8.0

B&R 8.0 update history at a glance:

• Update 3 (build 8.0.0.2084) [October 8, 2015] (KB2068)
• Update 2b (build 8.0.0.2030) [June 16, 2015] (KB2024)
• Update 2a (build 8.0.0.2029) [June 4, 2015] (superseded by Update 2b)
• Update 2 (build 8.0.0.2021) [April 28, 2015] (superseded by Update 2a)
• Patch #1 (build 8.0.0.917) [December 19, 2014] (KB1982)

7.0 Tracker
6.5 Tracker

[Gostev]

New Features and Enhancements

Microsoft
• Support for Windows 10 as guest virtual machines (VMs), including application-aware processing.
• Support for installation of Veeam Backup & Replication and all of its components on Windows 10.

VMware
• Support for vSphere 6.0 Update 1 (SSL issue).
• Added ability to convert "VMware Tools not found" job warning to informational event. To enable this behavior, set DisableVMwareToolsNotFoundWarning (REG_DWORD) registry value to 1.

Veeam Explorer for Microsoft SQL Server
• Log truncation is automatically skipped for stopped and unsupported SQL Server instances to prevent errors from being logged.
• Preferred network settings will now also apply to transaction log backup jobs.
• Support for SQL Servers using non-standard port number.

Veeam Explorer for Microsoft SharePoint
• Support for ADFS authentication.
• Support for UPN credentials.

Linux
• Support for the following additional SSH ciphers: aes128-ctr, aes192-ctr, aes256-ctr, aes192-cbc and aes256-cbc.

SureBackup
• Proxy appliance will now respond to ping from within Virtual Lab to prevent firewalls automatically blocking its network as public.

Other
• Removed weakly encrypted username and password logging from guest processing components using networkless (VIX) guest interaction mode. Veeam thanks Pasquale Fiorillo and Francesco Ongaro of ISGroup for vulnerability discovery.
• Improved performance of certain user interface operations, such as opening Backups node in large environments.
• Reduced load on SQL Server hosting product configuration database, while improving SQL interaction reliability in large environments.
• Improved performance of configuration backup in large environments.
• Minor enhancements and bug fix in various product areas.

Update 3 is generally available for download starting October 8, 2015 (KB2068)
This update is cumulative and includes all fixes and enhancements from previous updates below

[Gostev]

New Features and Enhancements

Engine
• Job performance improvements. A number of optimizations focused on reducing the time jobs spend performing auxiliary tasks should significantly reduce the overall job execution time for incremental runs of backup and Backup Copy jobs.
• Slow backup storage optimization. Target data mover now caches metadata from backup files, instead of requesting it from backup files residing multiple times over the course of the job run. This significantly improves performance of jobs targeting storage with poor random I/O performance (such as EMC Data Domain), while reducing the overall storage load. Important notes:
- The cache is only enabled when using a 64-bit OS on backup repository (or gateway server), and increases RAM consumption of each job by 2GB on average, depending on backup chain size. Using non-default backup block size changes cache RAM requirements proportionally.
- Using the default "Automatic selection" gateway server setting for Shared folder, EMC DataDomain or HP StoreOnce based backup repositories disables the cache.
- Disabling built-in deduplication disables the cache, because it is not required when deduplication is not in use.
• 64-bit Linux data mover. For increased scalability, a 64-bit data mover is now used for 64-bit backup repositories with an OS kernel version 2.6.18 or later.
• Linux data mover update. To enable new functionality, jobs will leverage the new version of the data mover on Linux servers with kernel version 2.6 or later. For compatibility with existing servers, jobs will use a legacy data mover when OS kernel version is earlier than 2.6 or cannot be detected.
• Direct data mover communication. When both data movers are running on the same server (e.g., when backing up to a local storage attached to a backup proxy server), they will now exchange data through shared memory. This should improve data transfer performance of local backup jobs currently reporting Network as the bottleneck, or when you see high load on some of the backup proxy server NICs when the data was supposed to stay local to the server.
• vPower NFS performance. Increased vPower NFS server scalability by significantly reducing CPU usage associated with guest I/O operations.
• VeeamZIP performance. Improved VeeamZIP performance when using a Veeam backup repository as the target.
• Replication from backup file enhancements. Replication from backup files no longer requires matching block size between the backup file and replica job settings

VMware
• vSphere 6 support. Added support for ESXi 6.0 and vCenter Server 6.0 (including vCenter Server Appliance).
• vCloud Director 5.6 support. Added support for backup and restore of vCloud Director 5.6 virtual machines (VMs) and vApps.
• VMware Virtual Volumes (VVols) support. VMs residing on virtual volumes can be processed in Virtual Appliance (Hot Add) and Network (NBD) processing modes. Hot add processing mode requires that all proxy VM disks are located on the same virtual volume with the processed VM.
• VMware Virtual SAN (VSAN) 2.0 support. VMs residing on VSAN can be backed up in Virtual Appliance (Hot Add) and Network (NBD) processing modes. For VMs with existing snapshots, hot add processing requires that all proxy VM disks are located on the same VSAN datastore.
• Storage Policy-Based Management (SPBM) policy backup and restore. Storage policy associations are backed up and restored for each virtual disk upon full VM restore. This eliminates the manual process, which directly impacts recovery time. SPBM policies are important to restore because "out of policy" VMs can impact availability of either the restored VM itself or other VMs sharing the same storage.
• Support for Fault Tolerant VMs (FT VMs) for backups and replicas. VMs with Fault Tolerance enabled can now be backed up and replicated. Legacy Fault Tolerant VMs must have Fault Tolerance re-enabled after vSphere 6 upgrade using vSphere Web Client to enable this capability.
• vSphere 6 tags integration. vSphere 6 introduces new APIs for programmatic access and management of vSphere tags. With vSphere 6 tag support, you can continue building new advanced backup policies based on tags, even after you upgrade to vSphere 6. Due to new unique tag IDs in vSphere 6, be sure to review your job setup after upgrading to vCenter Server 6.0 and running your jobs at least once to let them match tags automatically based on the tag name.
• Cross-vCenter vMotion awareness. Added support for cross-vCenter vMotion to the Quick Migration functionality. Now, when migrating a VM to another vCenter Server, the associated entries on backup or replication jobs will be updated automatically to keep the VM protected.
• Quick Migration to VVols. Enables moving VMs to VVols when using vMotion is not an option due to unreliable or slow network links, in scenarios when vMotion is not supported, or due to lack of VMware licensing. This functionality can help perform full migrations to new vSphere 6 clusters built from the ground up without inheriting design problems from previous clusters.
• Hot Add transport mode of SATA virtual disks. On ESXi 6.0, SATA virtual disks can now be processed in Virtual Appliance (Hot Add) processing mode, which was previously limited to SCSI virtual disks only.
• vSphere tags priority. vSphere tags priority has been raised one level above containers priority to ensure that jobs behave in line with customers' expectations when job's inclusion and/or exclusion lists use both containers and tags.
• Disable CBT reset. The workaround for VMware Changed Block Tracking (CBT) corruption issue at 128GB boundaries (VMware KB2090639) can now be disabled by customers with ESXi hosts patched against this issue. To disable automatic CBT reset upon virtual disk size change, create the ResetCBTOnDiskResize (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to 0.

Storage integration
• Granular rescan settings. Added ability to control which storage LUNs and volumes should be rescanned periodically for new VMs. These settings can be accessed directly from the Add Storage wizard while registering the storage, or by right-clicking the already added storage in the Storage Infrastructure tab.
• NetApp integration enhancements. The following NetApp-specific enhancements have been added:
- Added support for triggering updates of non-qtree and vFiler-based SnapMirror and SnapVault
- Added support for backing up VMs from NetApp MetroCluster in the failed over state
- Preferred Network settings of Global Network Traffic Rules dialog now also used to pick preferred storage adapters for data transfer. And in cases when multiple adapters use the same subnet, NetAppOrderedIPList (STRING) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication can now be created to specify preferred adapters' IP addresses separated by a semicolon.

Hyper-V
• SMB3 restore performance. Improved performance of full VM restore to SMB3 shares with caching disabled (such as in Nutanix).
• Hyper-V Integration Services status check. To prevent situations with some VMs unexpectedly going into the Saved State during a volume snapshot creation, jobs will now check that Hyper-V Integration Services (HIS) are running and reachable, waiting for HIS to respond for up to 10 minutes before failing the VM processing.
• Replica VM adjustment. Maximum RAM value of the replica VM is now automatically reduced according to the target host capabilities.

File-Level Recovery
• Native 4K disk support. Windows File-Level Recovery now supports mounting native 4K virtual disks from backups out of the box, without requiring a user to switch to the mount engine via the corresponding registry value.
• Linux ACL preservation. Multi-OS File-Level Recovery now preserves Linux ACL when restoring a file via Restore and Copy To operations. This requires that both backed up and destination VM runs Linux with kernel version 2.6 or later.
• Linux host selection. Multi-OS File-Level Recovery now allows choice of a target Linux host to restore guest files to without having to add it to the managed servers first.
• Restored files logging. Multi-OS File-Level Recovery now logs restored files in its session log, similar to the way Windows File-Level Recovery performs its restore activity logging.
• SNMP traps. Both Windows and Multi-OS File-Level Recovery can now trigger SNMP notifications. To enable such notifications, create the EnableRestoreSNMPTraps (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to 1.

Veeam Explorer for Microsoft SQL Server
• Improved restore performance. Database restore performance has been improved by up to 3x.
• Increased scalability. Reduced load on SQL Server hosting Veeam® Backup & Replication™ database in scenarios when transaction logs are being backed up from multiple SQL servers on a frequent schedule.
• Log truncation retry. In cases when log truncation fails under the specified guest processing account, it will be retried from the LOCAL SYSTEM account, which often carries the necessary privileges to perform this operation.
• SQL database restore. Added a warning when restoring a SQL database to a different location where the database with the same name already exists when using the Enterprise Manager web UI
• Improved issues reporting. Backup job now detects and reports more issues around transaction log processing, such as failure to truncate logs. This may result in the appearance of a new warning after the update, simply because existing issues were not previously reported.

Veeam Cloud Connect
• Multiple subscriptions support. Added ability to add the same service provider multiple times using different credentials.
• Reconnect performance improvements. Reconnecting to a service provider should now occur much faster when a cloud provider has multiple Cloud Gateways deployed.
• Enhancements for service providers. The following are enhancements relevant to Veeam Cloud Connect service providers only:
- Added ability to granularly purge global cache from WAN accelerator for the specific tenant.
- When deleting a tenant, global cache belonging to the tenant is now deleted automatically.
- Added ability to query the amount of protected VMs for the given tenant through RESTful API using the vmCount property.
- Service Providers node of the management tree is always shown when Cloud Connect Service Provider license is installed, without requiring the service provider to register at least one hypervisor host first. This enables service providers to easily setup Cloud Connect client consoles for testing or manipulating client backups.
- Enhanced debug logging to include initiator information for when the connection is terminated.

Tape
• Job performance. File to Tape and Backup to Tape job performance was further improved.
• Synthesized full backup enhancements. Synthesized full backup functionality has been reworked for better reliability, performance and support for very large backup files.
• Third-party tape handling. Added support for detecting and reusing tapes written by third-party tape backup vendors.

User Interface
• Manual job chain execution. By popular demand, when manually starting a job that has other jobs chained to it via "After this job" scheduling option, you will now be offered a choice between starting the selected job alone, or starting the entire chain.
• Disable network traffic rules. Both traffic throttling and encryption can now be temporarily disabled without requiring you to delete the corresponding rule completely, which is useful for troubleshooting or for temporarily removing bandwidth consumption restrictions.
• Improved guest credentials test. Guest processing credential test will now additional verify the presence of mlocate, which is required for Linux guest file system indexing.

Setup
• Silent install. Starting with Update 2, updates will fully support automated silent install, including triggering update of remote backup infrastructure components. This should significantly reduce the time required to update Veeam deployments with a large number of backup servers.

PowerShell
• Support in Free Edition. Start-VBRZip cmdlet is now enabled in Veeam Backup™ Free Edition, allowing users to schedule periodic VeeamZIP™ backups using external task schedulers, such as Windows Task Scheduler. This should make Veeam Backup Free Edition a viable alternative to ghettoVCB and other similar script-based free backup solutions.

Integration with Veeam Endpoint Backup FREE

This update enables you to take full advantage of Veeam backup repositories as additional target locations for your Veeam Endpoint Backup jobs, with backups for each endpoint being stored in a separate folder named after the endpoint. To assign end-user permissions to individual Veeam backup repositories, use the new repository properties dialog that will appear in the repository's shortcut menu upon first connection attempt from Veeam Endpoint Backup™ FREE to the Veeam Backup & Replication server (and before that happens, this dialog can be accessed by Ctrl-right-clicking the backup repository). Global network traffic rules defined in Veeam Backup & Replication apply to endpoint backup jobs using Veeam backup repository as the target, allowing you to ensure that concurrent backups from multiple endpoints do not impact your available bandwidth, or that endpoint backup network traffic is encrypted.

Using a Veeam backup repository as the target for your endpoint backup jobs enables the following additional functionality:
• Centralized monitoring. Perform basic monitoring and management for all incoming endpoint backup jobs, including email notifications about endpoint backup status.
• Offsite protection. Get your endpoint backups off site to disk-based storage, tape or your Veeam Cloud Connect service provider with Backup Copy and Backup to Tape jobs.
• Backup encryption. You can choose to optionally encrypt your endpoint backups stored in Veeam backup repositories. All standard Veeam encryption features are supported for endpoint backups, including password loss protection.
• Application item-level recovery. Restore application items from backups of server machines with Veeam Explorers™ for Microsoft Active Directory, Exchange, SharePoint and SQL Server.
• Disk export. Export physical disk content from backup into VMDK/VHD/VHDX virtual disk files.

Update 2 is generally available for download starting April 28, 2015 (KB2024)
This update is cumulative and includes all fixes and enhancements from previous update below

[Gostev]

New Features and Enhancements

General
• Retention policy for replication jobs is now processed concurrently for multiple VMs in the job, reducing the overall job run time.
• Added email notification about File to Tape and Backup to Tape jobs waiting for user interaction (for example, when a tape needs to be inserted).
• VMware Tools quiescence should now log an informational event when VMware Tools are outdated, as opposed to a warning event.
• Independent virtual disks which are explicitly excluded from processing in the job settings should no longer log a warning during the first time when the given VM is being backed up.
• Virtual Lab proxy appliance was upgraded to virtual hardware version 7 to allow managing more networks out of box.
• Jobs should no longer appear to "hang doing nothing" in some circumstances, with the duration not ticking for any event in the job log.

Cloud Connect
• Cloud backup repository capacity limit of 63TB has been removed.

Linux support enhancements
• Added support for file level recovery from XFS 3.2.0 volumes (default file system in Red Hat Enterprise Linux 7.0 and CentOS 7.0 distributions).
• Added support for file level recovery from Btfrs volumes (default OS file system in SUSE Linux Enterprise Server 12.0).
• Added support for file level recovery from VMs with more than 10 LVMs.
• Improved Linux guest file system indexing performance for incremental backup job runs.
• FTP access mode of the file level recovery helper appliance now also displays friendly LVM volume names introduced in v8.
• To enable identity verification of the remote server, SSH key fingerprint is now displayed to a user when registering the new Linux server. Accepted SSH keys are stored in the configuration database to protect from MITM attack.

Performance enhancements
Added experimental support for direct data movers communication when both are running on the same server (for example, when backing up to a local storage on backup proxy server). If your local backup jobs report Network as the bottleneck, and you see high load on some backup proxy server NICs when the data was supposed to stay local to the server, your jobs performance is likely to benefit from this behavior modifier. To enable alternative data exchange path, create the DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:
0: Default behavior (no optimizations)
1: Data exchange through TCP socket on the loopback interface (faster)
2: Data exchange through shared memory (fastest)

EMC Data Domain integration
• Synthetic full backup creation and transformation performance has been improved significantly.

HP 3PAR StoreServ integration
• Remote Copy snapshots are now automatically excluded from rescan, both speeding up the rescan process and reducing storage load.

NetApp integration
• Added support for NetApp Data ONTAP 8.3.
• Added support for HA Pair 7-mode configuration.
• Added support for vFiler DR configuration.


Resolved Issues

General
• Restored incremental backup job performance back to v7 levels by removing unnecessary metadata queries from previous backup files.
• Multiple issues when parallel processing is disabled (high CPU usage on backup server, wrong task processing order, reversed incremental backup jobs with disk excluded failing to access backup files).
• Automatic network traffic encryption (based on public IP address presence) is enabled even when both data movers are running on the same computer, reducing job's processing performance.
• Quick Backup operation triggers jobs chained to the job it uses.
• Loss password recovery option does not function correctly when Enterprise Manager has multiple backup servers registered.
• Under rare circumstances, backup job with encryption enabled may fail with the "Cannot resize block data is encrypted" error.
• When multiple network traffic rules having the same source and target ranges are set up, changing settings to one of the rules also updates all other rules.

vSphere
• In vSphere 5.5, enabling virtual disk updates redirection to the VMFS datastore in Instant VM Recovery settings, and then migrating a recovered VM to the same datastore with Storage vMotion causes data loss of data generated by the running VM.
• Full VM restore process puts all virtual disks to the datastore selected for VM configuration file, ignoring virtual disks placement settings specified in the wizard.
• In certain environments, jobs fail to process VMs in Direct SAN Access mode with the "Failed to create processing task for VM <VM name> Error: Disk not found" error.
• Dismounting processed virtual disks from backup proxy server takes longer than expected in the Virtual Appliance (hot add) processing mode.
• Rescan of vCenter server containing hosts without HBA and SCSI adaptors fails with the "Object Reference not set to an instance of an object" error.
• Replication jobs to a cluster fail with the "The operation is not allowed in the current state" error if chosen cluster host is in the maintenance mode.
• Under rare circumstances, jobs with Backup I/O Control enabled may occasionally log the "Operation is RetrievePropertiesOperation" related errors.
• Enabling Backup I/O Control limits snapshot removal tasks to one per datastore regardless of datastore latency levels.
• VM Copy jobs always fail over to the Network (NBD) processing mode.
• VMs with virtual disks having the same names but with different capitalization, and located on the same datastore cannot be backed up.
• Backing up vCloud Director vApp to a CIFS share may fail with the "Object Reference not set to an instance of an object" error.
• Snapshot Hunter ignores backup window specified in the corresponding job.
• vCloud Director VMs created from linked clone template and with a user snapshot present incorrectly trigger Snapshot Hunter.

Hyper-V
• CBT does not track changes correctly on VHDX disks files larger than 2 TB.
• CBT may return incorrect changed blocks information for differential disks, when parent and child disks have the same name.
• Large amount of unmapped VHDX blocks may cause the job to failover to full scan incremental backup with the "Failed to update unaccounted changes for disk. Change tracking is disabled" error.
• Existing Hyper-V backup and replication jobs processing VM with SCSI disks start to consume x2 space on target storage after upgrade to v8.
• Removing a node from a cluster does not remove the corresponding node from the cluster object on the Backup Infrastructure tab of the management tree.
• Replication from backup fails with the "Virtual Hard Disk file not found" error if backup job that created this backup had one or more disks excluded from processing.
• Under rare circumstances, legacy replicas using non-default block size may fail with the "FIB block points to block located outside the patched FIB" or "Block offset [] does not match block ID []" error messages.
• Offhost backup from Windows Server 2012 R2 fails with the "Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown" error when processing a VM that can only be backed up in Saved or Crash-Consistent states, and native Hyper-V quiescence is enabled in the job settings.
• If a Hyper-V cluster node goes offline during backup from shared volume snapshot, and does not recover until the Veeam job fails, CSV volume backup infrastructure resource will not be released by the scheduler. As the result, other jobs waiting to process VMs from the same volume may remain hanging in the "Resource not ready: Snapshot" state.
• Replication job fails with the "Cannot assign the specified number of processors for virtual machine" error for VMs with core count larger than the one of the target Hyper-V host. Legacy replication job works fine in the same circumstances, however VM failover is not possible.

Application-aware processing
• VeeamGuestHelper process crashes when backing up Windows 2003 SP2 VM running Oracle.
• Application-aware processing hangs for extended time when processing a Windows VM containing one or more Oracle databases in the suspended state.
• Certain advanced guest processing settings lead to VM processing failing with the "Unknown indexing mode 'None' " error.

Multi-OS File Level Recovery
• Restore to original location does not work in certain LVM configurations.

Enterprise Manager
• Under certain circumstances, Enterprise Manager data collection may fail with the "Cannot insert the value NULL into column 'guest_os' " error.
• Restore scope rebuild operation tries to reach out to the hosts already removed from the corresponding backup server, resulting in login errors.

Built-in WAN Acceleration
• WAN accelerators fail to detect and populate global cache Windows Server 2012 R2 and Windows 8.1 OS data, which may result in reduced data reduction ratios.

Cloud Connect
• The Decompress backup data blocks before storing backup repository option is ignored for Cloud Connect backend repositories.
• Backup Copy jobs with encryption enabled fail with the "Attempted to compress encrypted data-block" error if backend repository has the Decompress backup data blocks before storing option enabled.
• Veeam Backup Service crashes with the "An existing connection was forcibly closed by the remote host" if network connection fails immediately upon establishing the initial connection to the Cloud Gateway.
• Cloud Connect quotas are rounded incorrectly in the service provider's interface.
• Tenant statistics data such as data sent/received is only shown for the last 24 hours.

Veeam Explorer for Active Directory
• "This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms" error is displayed if Use FIPS compliant algorithms for encryption, hashing, and signing Group Policy setting is enabled.
• Attempting to open single label domain’s AD database fails with the "No writable domain found" error.
• Attempting to open third level domain’s AD database fails with the "No writable domain found" error.
• Under certain circumstances, object restore may fail with the "Unable to cast object of type 'System.String' to type 'System.Byte[]' "error.

Veeam Explorer for Exchange
• Process of exporting email messages cannot be stopped.
• Copying or saving attachments from multiple different emails always copies or saves attachment from the first email for which this operation was performed.
• Opening public folder’s mailbox fails with the "Unable to find wastebasket" error.

Veeam Explorer for SQL Server
• Restoring to SQL instance with case sensitive collation enabled fails with the "The multi-part identifier "a.database_id" could not be bound" error.

Backup from Storage Snapshots
• Backup jobs from storage snapshots on iSCSI SAN take too long to initialize when large amount of iSCSI targets are configured on the backup proxy server.
• Under certain circumstances, backup jobs from storage snapshots on FC SAN may fail with the "Unable to find volume with Node WWN and LUN ID" error.

HP 3PAR integration
• Rescan of 3PAR array with empty host sets fails with the "Object reference not set to an instance of an object" error.

NetApp integration
• Under certain circumstances, storage rescan process may leave temporary storage snapshots behind.

Tape
• Backup to Tape jobs may appear to run slower after upgrading to v8 because of copying unnecessary backup files (VRB) to tape when source backup job uses reversed incremental backup mode.
• On standalone drives, tape jobs configured to eject tape upon completion fail the "Object reference not set to an instance of an object" error after 10 minutes of waiting for the user to insert tape.
• Tape job configured to use full and incremental media pools hosted by the different standalone tape drives fails when processing incremental backup with the "Value cannot be null" error.
• Tapes are formatted with the block size of tape medium, as opposed to the default block size of the tape drive.
• Tape job configured to use full and incremental media pools hosted by the different tape libraries fails while waiting for an incremental pool’s tape with the "Tape not exchanged" error
• Tape jobs fail with the "GetTapeHeader failed" error on IBM TS3100 tape library with Path failover feature disabled.

PowerShell
• Objects obtained with Find-VBRViFolder cmdlet have incorrect Path attribute.
• Get-VBRTapeJob cmdlet fails to obtain job object for tape jobs with incremental processing disabled.
• Get-VBRTapeLibrary cmdlet fails with the Standard Edition license.
• All built-in WAN accelerator management cmdlets have been made available in all paid product editions.

Patch #1 is generally available for download starting December 25, 2014 (KB1982)