LocalDB gone wild backup hanging with 50% filling up c drive

Backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)

LocalDB gone wild backup hanging with 50% filling up c drive

Veeam Logoby DaStivi » Tue Jun 20, 2017 6:51 pm

Hi there,
i've a installation of Veeam Agent here, that have some serious issues... i noticed that my C Drive got filled up completely to 0byte free!
After that i started investigation, FileTree Tools like WinDirstat didn't showed me where the space was used, i really did found the space eaters 24h later, but first things first....

i did noticed that a veeam agent backup was running, it was at 50% and i didn't thought about too much about that, started to delete some things on my C Drive... like games from Steam sitting on the C Drive.... but soon i noticed that the drives starting to be filled up again.... after that i look'd further and saw that the SQL localDB read and write a lot of data somewhere... after that i looked in the agent gui again and still on 50%, then i noticed that the job was "running" for about 25h allready (i miss read this information first for minutes, but even that should have alerted me as normal backup take less than 5 minutes...) anyway i tried to cancle the backup now, then i got this nice error message:
Image
Image
after that i restarted the veeam agent service, still at 50%, canceling still getting the error message, reboot, still the same.. after that i decided to deinstall veeam agent and SQL localDB, btw. the amount of space was still used somewhere but i come too this later...

i also checked vssadmin list shadows, as i thought maybe some vss snapshot was hanging on C drive and consumes the space... but nothing left
i do have noticed in eventlog (after deleting it completly as it couldn't write anything more and crashed and didn't displayed anything more) that the SQL looks like to be crashing somehow... another reason why i deinstalled SQL localdb binaries too!
Image

after reinstall of veeam instantly the backup was running again at 50%! Canceling of course same error message... i deinstalled again and stopped investigation as it was allready middle in the night...
today i started my computer again and still saw the space usage on my c drive (i had hoped that somehow the localdb files got cleaned up after a full reboot)... i started to investigate where the files where on my drive... the issue still was that i didn't saw these files with WinDirstat or even with normal windows explorer properties of C-Drive...

i deleted every folder of SQL server i found in program files and all known folders where it could be, i even found after some digging the VeeamBackup_log.ldf and VeeamBackup.mdf files inside C:\Windows\System32\config\systemprofile and deleted it

then again i reinstalled veeam agent and guess what, instantly again the backup was on 50% same error... even after deleting the the SQL DB files...

C drive by the way started filling again, after some more digging in windows performancemonitor i saw that the crashing SQL service was writing some logs and dumps inside some subfolder in the path above:
C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint
Image
somehow this folder was not visible within windirstat and even normal windows explorer properties looks like did not parsed this subfolder, as when i navigated too it i could see the space taken there... allready about 50gig... i've delted again now everything there, got my space back now, but still strugling with veeam agent, as soon as i've installed it SQL is crashing starting too fill up again my c drive and veeam hangs on 50%

luckily inside this dumb folder is also an error.log file and i look into this right now, i saw allready some error:

CPerIndexMetaQS::ErrorAbort - Index corruption

i've to google now how too fix this... i gonna report back as soon as i've found something...
DaStivi
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Tue Jun 30, 2015 9:13 am
Full Name: Stephan Lang

Re: LocalDB gone wild backup hanging with 50% filling up c d

Veeam Logoby DaStivi » Tue Jun 20, 2017 7:30 pm

i've tried to get into the SQL localdb with SQL mgmt studio, but i only can connect to (localdb)\V11.0 and i don't see any veeamagent database there
also sqllocaldb i show only:
v11.0

C:\WINDOWS\system32>sqllocaldb i v11.0
Name: V11.0
Version: 11.0.6020.0
Shared name:
Owner: STEVE-PC\Stephan
Auto-create: Yes
State: Running
Last start time: 20.06.2017 21:27:32
Instance pipe name: np:\\.\pipe\LOCALDB#3F29707F\tsql\query


the pipe name from sqllocaldb cmd is not the same as it's in the error.log file in the folder... when trying to access the pipename from error log files with mgmt studio i get an auth. error!
DaStivi
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Tue Jun 30, 2015 9:13 am
Full Name: Stephan Lang

Re: LocalDB gone wild backup hanging with 50% filling up c d

Veeam Logoby Mike Resseler » Wed Jun 21, 2017 6:43 am

Hi Stephen,

WinDirStat: Did you try to start the program "as an administrator". That normally should give you insight in those files also. (You actually have to force this, even if you are an administrator on your machine)

For the agent problem. From what I can see there is a serious issue which I doubt we can solve over the forums. This will require reading logs and more. Can you create a support case (through the UI) and post the case ID and follow-up after that here?

Many thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3381
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: LocalDB gone wild backup hanging with 50% filling up c d

Veeam Logoby DaStivi » Wed Jun 21, 2017 7:07 am

Hi Mike, yes of Course i did run it as admin... anyway i found ah few different folders that does notshow up... basically these are all Folders where even as admin there is no access.. as soon as adding permissions (taking ownership) you cAn view the content and watch the properties...

I've openend a Case (02195343) allready and uploaded eventlogs and the SQL logs (error.log, SQLdump logs)
DaStivi
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Tue Jun 30, 2015 9:13 am
Full Name: Stephan Lang

Re: LocalDB gone wild backup hanging with 50% filling up c d

Veeam Logoby Mike Resseler » Wed Jun 21, 2017 7:10 am

Thanks,

Keep us informed about the progress.

Cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 3381
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler


Return to Veeam Agent for Windows



Who is online

Users browsing this forum: No registered users and 1 guest