backups writing to C:\Windows\SysWOW64

[mrt]

Why might my backups be suddenly writing to C:\Windows\SysWOW64 when this directory is not specified in the backup repo node? Other volumes specified in the node are available and have plenty of free space. Now C: is full and things are failing left and right. These jobs are also affected by the nbd failover bug.

The jobs are configured to make active fulls on the first Sunday of every month, which today happens to be.
I'll get in touch with support but am hoping for a quick answer here. Thanks

[kellison]

I upgraded my Veeam server to 6.0 over the weekend - it is a VM Windows 2008 R2 server. Upgrade went well but last night was the first rn of backups. I had about 10 GB of hard drive space available on my "C" drive when I left for the day. The Veeam is installed on this drive. This morning I found that over half my backups had failed. Looking at the event logs I found out the "C" drive had run out of hard drive space. I then shut down the server and added 20 more GB to the "C" drive, and proceeded to run a backup. It is at 29 percent (running VERY SLOW) on the first VM of 6 and it has already used up 10 GB of hard drive space. What is causing this and how to I fix this? My retention period for my backups is 7 days.
HELP !!!

[engagingit]

Happening to me as well, a copy of the backup file is being made at c:\windows\syswow64

I'm assuming there is a flag to make it write directly to the usb drive rather than save locally first but I can't seem to find it.

[real_tarantoga]

My veeam Backup Server is updated - and no more usable.
No way to disable excessive writing into %windir%\syswow64? (not resolved)
No way to chose the default repository while upgrading? (resolved)
A really annoying bug with the "synthetic full" check box! (resolved)

... and why a default repository on %systemdrive% ?!? What happens to veeam?

looking at your answer with anticipation, Stefan

[kellison]

Update - I had to stop my backup - it used up the entire 20 GB that I had added....

[Vitaliy S.]

Please contact our technical team as logs investigation is required. Thanks!

[Gostev]

and why a default repository on %systemdrive% ?!? What happens to veeam?

For consistency, and because usually, there are no other drives. The default repository is for evaluators only, of course we expect that for production usage, customers will create proper repositories pointing to the actual backup storage.

Regarding the actual problem discussed in this topic, we have received only a few reports like this, so this seems to affect very specific configuration of backup server only. Everyone who is affected should open a support case to let us figure out what is common.

Thanks!

[real_tarantoga]

short status update

the support recommandation to use another backup proxy was almost right, but ... also 64bit backup proxies didn't work.
i tried on a 32bit server - and it was running.
I will now create a backup proxy server pool on an dedicated esx host cluster (DRS disabled), with 2003 r2 & 2008 r1 enterpise servers, giving them a lot of vCPUs ... and i will update my results here.

for now it seems to be an explicit 64bit problem.

Stefan

[Sethbartlett]

You can also try deleting your current repository and creating a new one pointing to the same location and updating the job, this has seemed to work.

[Gostev]

Must be related to automatic creation of repositories during the upgrade.

[real_tarantoga]

Hello board,
the workaround with virtual machines used as backup proxies was very successful.
Even more I could run more concurrent jobs as anytime before! No real stress or heavy performance impact neighter on the VMs nor on the two ESX servers. So veeam jobs have processed the backup for 100 VMs over night - some uncounted terra bytes ... That's cool.

Other than mentioned above I could also use x64 machines.
The problem wasn't related to the architecture but to a mishandling of the existent backup jobs and/or backup files on the imported backup repository during the upgrade process.

The backup proxy on the upgraded veeam b&r 5.0.2.255:

• works for all newly created jobs - it is not possible to map old vbk files to the new jobs, so I have to start with a full backup.
• doesn't work for any job with exisisting vbk/vbr files, works after removing the backup files through the GUI
• doesn't work for old jobs with german vowels (umlauts) - even if the manually given file name doesn't include umlauts! - and works when the jobs will be renamed. That always worked in the older program versions ...
• doesn't work for jobs (vbm+vbk) moved from syswow64 to the backup repository, backups must be deleted

I wonder why the program continues to use the %systemroot%\syswow64 folder to put the meta file here for only some jobs, while other (new) jobs create subfolders on the repository and another group of jobs put the meta file right beside the existing vbk files ...

Finally, the problem seems to be resolved by working around step by step from issue to issue - and losing a lot of backups!!!
For the first time I'm deeply disappointed with the software and also with the support - not with the reaction time, but with the given answers and result (there's no solution or lossless workaround until now). I've got the impression they were "groping about in the dark" (?) as well.

Regards, Stefan

[Gostev]

Hi Stefan, sorry for the issue you've experienced. I have forwarded your feedback to our support and QC management. I've also verified that the issue is not wide spread (just a few support cases). With any software, there is always a risk with running into issues when making a decision to upgrade to the new major release immediately upon its availability. It is fair to say that no amount of testing by limited QC group can cover all the configuration and upgrade scenarios our 35'000 customers might have. Thanks.

[kellison]

I have to agree with Stefan - after going back and forth with support for a whole day - "try this - try this" I finally gave up and created a new datastore, added it as a new drive letter to my Veeam VM server, deleted the backups and repository and basically started from scratch again on my backups. Strange though that my other repository that was brought over from the upgrade is working fine. In fact, I had 2 different backups in the repository that I had deleted and the other backup worked fine. On my DR site that was also upgraded I am not having any trouble with the repositories (so far)... I'm not seeing any huge improvement in speed but guess will just be thankful that I have the backups working again.

[flavor4real]

Hello,
How is it going? All of my Veeam jobs worked fine until now. All of the sudden the one backup job creates their vbk in c:\windows\SysWOW64\

There were no prior issues with veeam and I'm kind of wondering. I already removed the VBK, verified the veeam job setting and tested again. Still does the same

Thanks

[Gostev]

Hi, what product version are you running? This sounds like known v5 upgrade issue that was resolved loooooooooooong ago... I believe, in 6.1 release.

[flavor4real]

We are running 6.0. We have to wait until we get approval for the 6.1 version.

I received a respond from the Veeam Support, pointing me to KB ID: 1475 and I have a question. the KB below says:
=====================================================================
1) The repository for the backup job(s) will need to be deleted however you cannot delete the repository with jobs linked to it so what you can do is create another repository pointing to the same location.

2) Move the Veeam files (VBK, VRB, VIB, VBM) from the SysWow64 folder back into their original locations with the rest of the backup files for your jobs.

3) As long as you have a VBM file for each set of backups you can then simply go into Backup Repositories (under Backup Infrastructure) and re-scan the repositories that you just created. Veeam should then see that the backup files reside in that directory.

4) Go into your Backup jobs and point back to the newly created repository and click the Map Backups link. Your jobs should pick up where they left off and you shouldn't have this issue again.

5) Delete the original repository which should now have no jobs linked to it.
=====================================================================

Point 2 outlines, that the VBM need to be located into their original locations with the rest of the backup files for your job. My VMB's are located in the C:\Windows\SysWOW64 .. Do I have to move them?

Thanks

[flavor4real]

I have another question.
If I have one mapped drive and a few backup jobs save the generated backup files (vib's/ vbk's) in the location. Does each backup job need to have their own folder structure within the location?
Example:
Z:\Backupjob1\ ...*.vbk,...*.vib
Z:\Backupjob2\ ...*.vbk,...*.vib
Z:\Backupjob3\ ...*.vbk,...*.vib

or can everything be dumped like
Z:\ ...*.vbk,...*.vib

I'm asking because I'm trying to get my issue resolved and I saw that under V6.0, it seems to be that veeam creates a folder with the Job name under the storage path and have the vbk, vib, and the VMB in it.

If so, do I have to create these folder manually and move the VMB's in it, then rescan the repository?
Thanks

[foggy]

Point 2 outlines, that the VBM need to be located into their original locations with the rest of the backup files for your job. My VMB's are located in the C:\Windows\SysWOW64 .. Do I have to move them?

Yes, move them along with all backup files.

If so, do I have to create these folder manually and move the VMB's in it, then rescan the repository?

The subfolder for the job is automatically created in the backup repository during the first job run following the upgrade to v6. If you do not have the folder for this job, please create it manually and place all the job's files in it.

[veremin]

Does each backup job need to have their own folder structure within the location?
Example: Z:\Backupjob1\ ...*.vbk,...*.vib
or can everything be dumped like : Z:\ ...*.vbk,...*.vib

As far as I’m concerned, such structured architecture is necessary. Otherwise, step 4 of the described method will definitely fail with the following error:

Code: Select all

Move all backup files to a new repository first.

Thanks.

[flavor4real]

Very Interesting ... thanks guys.

[flavor4real]

Hello,
I've search and found some posts but I have more questions.
I've upgraded from V5 to V6 and the backup jobs started to put their backup files to the ..\windows\sysWoW64 folder. I found many information but it's not really working out.

I have 4 existing repositories, pointing to 4 individual storage paths, and holding the vbk's/... for 3-4 backup jobs. Since previously all vbk's/vib's were simple dumped in each storage location.
I've therefore created the required folder structure Example:
Z:\Backupjob1\ ...*.vbk,...*.vib
Z:\Backupjob2\ ...*.vbk,...*.vib
Z:\Backupjob3\ ...*.vbk,...*.vib

and moved the jobs and the vbm in these locations. I rescanned the repositories and mapped each job to it. Now, Once I run the job it still put's the backup files in the ..\windows\sysWoW64 folder and creates a new vbm file.

so, what else do I need to do?
I've read that I need to create new repositories, I've done that.
Then I need to move the existing backups from the old repository to the new one. How do I do that?
Then I need to rescan
Then I need to map the jobs

Thanks,

[veremin]

I've read that I need to create new repositories, I've done that.
Then I need to move the existing backups from the old repository to the new one. How do I do that?
Then I need to rescan
Then I need to map the jobs
Thanks,

1.In your OS Just manually copy/paste existing folders (Backup Job1 and etc.) with backup files (.vbk,.vbm,.vib) stored in it to a new repository.
2.Then Rescan new repository.
3.After having done that, right-click the job, go to the “Storage” option and select newly-created repository from the list.
4.Map the job.

Additionally, If you still think that there isn’t enough information, you can easily use this topic as a source of knowledge.

That’s all.
Thanks.

[flavor4real]

does that mean that my storage admin has to attach 4 new storage to my Veeam Backup Server? So, then i would have 8 storage and i can move the data from the existing 4 to the new 4 and then remove the old 4.....

[foggy]

You do not need additional storage actually, just create new repositories pointing to the same location and move the backup files from the SysWow64 folder there, according to the new folder structure.

What you might be missing, is not just picking the new repository while mapping the job, but also pointing it to the existing backup files (using the Map backup link on the Storage step of the wizard).

Btw, here is another topic with the procedure outlined in detail: Moving To New Repository.

In case you're still stuck with getting the jobs writing to the required folders, I would recommend asking support to assist you via webex, this would probably be the quickest way to address that.

[flavor4real]

I see the new added Backup Repository in the storage selection mapping under the the job. however, once selected, I do not see any of the sub folders. I rescanned the repositories several times.

WebEx is not allowed in our environment ...

[flavor4real]

Version 6.0. We have not approval to use Version 6.1 yet.

Will a fresh install of Veeam 6.0 (instead of a upgrade) fix this issue? I could then just import the existing backup files.

[Gostev]

Hi, I'd like to remind that this is NOT a support forum, as explained when you click New Topic.

Please open a support case, and our technical support will be able to assist you whether or not webex is allowed. The team behind this forum is not involved in support, and we cannot assist you efficiently with technical issues via forum posts. Besides, most issues require debug logs research. Thank you for understanding!

[flavor4real]

Understood... sorry, just brain storming

[ENSFCa]

Hi, just wanted to chime in as I noticed this thread and, for me it is relevant.
We had an issue where this folder was filling our System drive on VM File Level Restore. We'd chose Windows FLR on our large data store VM and the system would hang for ages, eventually creating a huge file in this folder and requiring a services stop and restart (and cleanup of this folder and the C:\VeeamFLR folder too).
It's still not actually solved yet (case #155330, closed but awaiting fix) however the Veeam techies did find the cause - something called reparse points (http://en.wikipedia.org/wiki/NTFS_reparse_point).
We are using a Heirarchical Storage System which, as it should, archives old files and replaces the original file with a stub - these stubs are attempted to be verified by Veeam and each one takes a while to time-out - hence the hang and the massive cache file being produced as the system tries to deal with it.
Work-Around: use the 'Other' FLR option which mounts the VM in vCentre (via NFS) and allows you to browse the disk - it's slower and leaves bits behind in vCentre, but it's reliable.

[flavor4real]

I just wanted to give an update:

All the solutions did not worked. The Veeam Engineer WebEx'd in and tried a few things but there was no solution. so, I went and recreated one backup job and tested it. the Backup job created the folder structure based on the job names and put the *.vbm and backup file in. I therefore recreated all backup jobs and disabled the old ones. At first i received a slow backup speed of ~30-60 Mb/sec but after the initial backup, everything started to work again. I will now, slowly decommission the old backup jobs and remove the old backups.

So, the worst scenario is that you have to recreate the backup jobs, which is fine. You will not loos any data if you keep the old job and files intact and just slowly decommission then after your grace period.

The only issue I've experienced is that my backup storage locations disk space get pushed to full. I had to manually remove old backups one by one each day to ensure that I still can restore from the old data but have enough space for the new data to come in. After all, new jobs means a new retention period, so you need a bit of extra space. I managed.

Thanks for the support guys.