vCenter Transaction Logs filled up from Veeam v5

[withanh]

The back story first:
I upgraded to Veeam v5 last weekend, as of now I left all my backup jobs as regular backup jobs, I haven't done anything with SureBackup yet. I also upgraded from vSphere 4.0U1 to vSphere 4.1.

The daily backups ran fine all last week, but they started failing this weekend. Investigation led me to the SQL Transaction Logs were full and that was causing the vCenter to shut down. Further investigation showed this to be size limited to 500MiB. This has worked just fine since we first installed our vCenter last year. Veeam backups (on v4) have worked great as well for the same 1yr plus time frame.

When we first installed our vCenter, I'm not sure if I set a limit on the transaction logs, or if it by default was size limited. This weekend (our first weekend of v5 backups) the vCenter kept shutting down causing Veeam to fail as well. I upped the limit to 1GiB, restarted the backups, they ran for about 8 hours then vCenter stopped again, causing Veeam to fail the jobs as well. I decided to remove the size limit and monitor it to see how large the size of the Transaction Logs get, by the end of the backups it had grown to 2.8GiB.

It seems kind of odd that the Transaction Logs were fine limited to 500MiB for over a year, then all of a sudden they ballooned up to 2.8GiB. I've placed a 5GiB limit on it for now, hopefully that should be fine.

This brings me to my questions.
1. Has anyone else seen similar behavior with Veeam v5?
2. Is it possibly related to vCenter upgrade?
3. How big is a reasonable size for the Transaction Logs? (I know, variables variables variables). I have 4 hosts and 46 VM's, the size of the database itself is 1.8GiB and the size of the transaction log is 2.8GiB.

Thanks!

h
<edited for formatting>

[Gostev]

I am 99.9% sure Veeam Backup has nothing to deal with this problem... our interaction with vCenter is very minimal, and did not really change from previous version. I would look for performance, reporting, management, automation tools or scripts you might have deployed recently. Large transaction logs tell me there are a lot of changes going in your vCenter configuration, any idea what may be causing all these changes? The only changes Veeam Backup does are snapshot creation/deletion, and logging vCenter events, and this was like that from v1. Other than that, Veeam Backup is read-only to infrastructure, just does not have the code to produce such a massive infrastructure changes (except SureBackup and temp VMs, but you do not have this yet).

My initial thought was that it is some performance monitoring tool, these produce really heavy load on vCenter. I've seen monitoring tools crashing vCenter (vCenter memory usage would sky rocket during performance data collection, eat up all available memory, then vCenter would crash). But they are mostly read-only, so again should not really grow transaction logs which serve DB writes...

[withanh]

The only changes that were made were upgrading vSphere from 4.0U1 to 4.1 and upgrading Veeam from v4 to v5. When I increased the Transaction Log size limit and watched it for an hour it did not change in size. The size did not start changing until I started the Veeam backup jobs again.

It is strange for sure! Hopefully someone with some vCenter expertise will chime in about the size and sudden growth spurt of the transaction log size.

Well, your comment about monitoring tools triggered some dim cells in the back of my head, I do have the Veeam Monitor and Enterprise Manager installed. I tried upgrading Enterprise Manager to v5 two weeks ago when I did the vCenter upgrade and Veeam BR upgrade, but am getting errors in the Enterprise Manager now. But still strange that the transaction logs did not bloat all week until the weekly backups were launched.

Maybe the issue is Enterprise Manager related???

[Gostev]

Actually, I think I know the reason. vSphere 4.1 added many new performance counters comparing to previous release. So each ESX(i) hosts now produces tons more real-time performance data than before, and that is a lot of realtime data that all of this goes straight into vCenter DB...

No, Enterprise Manager does not interact with vCenter at all.

[withanh]

That makes sense. I'm not surprised that it is vCenter related and not Veeam related. I asked here first instead of going to VMware Support for two reasons, first I wasn't sure if it was vCenter or Veeam, and most importantly that I knew I would get a clear answer faster than VMware Support.

Thanks Anton, your support and expertise is greatly appreciated!

Darhl

[Gostev]

It was just one idea, because our Monitor 5 also go affected by this (our own DB grows much faster now than in previous version due to all the new vSphere 4.1 performance counters we are collecting).

But, look what I found in VMware support KB:
Transaction log for vCenter Server database grows large after upgrading to vCenter Server 4.1

[withanh]

Awesome, I think we both found that article at the same time!
<edit>
Nope, not the same article, but they same the same thing!
http://kb.vmware.com/selfservice/micros ... Id=1026407
</edit>

[withanh]

And here is what MS says about recovery models: http://msdn.microsoft.com/en-us/library ... 90%29.aspx

[Gostev]

Great, looks like we nailed it

[withanh]

Found another article about it, it is fixable: http://www.virtualizationbuster.com/?p=914