168TB written to SSD drive

[AlexLeadingEdge]

Hi guys,

On our backup server, our C Drive has an SSD drive, while the backup drives are all 10TB rotating drives. The SSD drive shows 168TB had been written to this drive, nearly all in the last two months.

We don't do backups to the C Drive, so I'm trying to figure out if there is a temporary file being created on the C Drive before VeeamBR writes to the rotating drive.

The only other software of note is VeeamONE, but I doubt it would write that much in the way of information to the C Drive (?)

Any idea how I would identify what it doing all the writing? We need to stop this as it has triggered the SMART system in the drive that it is about to die. We will replace this drive ASAP, but first we need to fix this software configuration issue.

[AlexLeadingEdge]

I'm trying to open a ticket for this but it says I'm not a license or case administrator, and when I call Veeam Support it says I need to have an open ticket to talk to someone, so I'm going around in circles and getting nowhere.

[AlexLeadingEdge]

I have managed to figure out who the License Administrator is, and logged in as them and escalated my privileges.

Case #05660351

[AlexLeadingEdge]

Seems that sqlservr.exe has written the 186TB. Looking at Task Manager, I see that sqlserver.exe has written 294 gigabytes to the hard drive. I assume this is since the last reboot, which was 5 days ago.

[Mildur]

Hi Alex

#02 are contract numbers and not case numbers. Case numbers are starting with #05.
I removed the number from your post and replaced it with the correct case number.

As far as I see, the discussion is still ongoing with the support engineer. I suggest keeping working with our support.
If you have other disks, maybe it's worth to move the database (with the provided links) to this other storage and monitor the change rate on that device.

Thanks
Fabian

[AlexLeadingEdge]

Hi Fabian,

Thanks, I hadn't realised I used the wrong number.

As mentioned to the support engineer, moving the database will only move the problem to another disk. We need to identify the problem and fix it.

When I look at our other clients I don't see the high number of writes to the database. I suspect it is VeeamONE that is doing it, as the other backup servers don't share the drive with VeeamONE. I have uninstalled VeeamONE and will continue to monitor the change. At the moment the hard drive's expected life is dropping by ~1% per day so it shouldn't be too hard to see if this is the case within the next couple of days.

I've also noted that the PNY NVMe drive we used for the C Drive is most likely QLC, which has a low number of write cycles before it will trigger the SMART system, though according to the manufacturer's website we are at 203TB write cycles and it should be good for 850TB write cycles. Reading the information from the manufacturer (PNY) they've been very vague on the technology inside it. We will replace it with an Intel 1TB TLC drive once we know what is causing the high number of writes.

[Mildur]

Hi Alex

Of course, solving the issue is our main goal.
Moving the database to another disk (non-SSD) could help in not overusing the ssd.
But removing VeeamOne also works to find out if VeeamOne is the cause.

Best
Fabian

[david.domask]

Hey @AlexLeadingEdge,

I took a peek at the case and I suspect that the tooling used for monitoring the disk space usage isn't quite granular enough to really give a good picture of the usage.

I will update the engineer with some suggestions on more specific tooling you can use to get a better idea of the usage, but total throughput isn't so helpful as it just shows an amalgamation of all disk usage without a lot of specifics into the processes generating this. It's a good flag for "something is generating a lot of disk IO", but beyond that it's not so useful.

Within the case I can see some of the Task Manager/Resmon outputs, but it's still not so clear if the top offenders are really acting abnormally or if it's related to a specific Veeam operation.

Regardless, the Engineer should have a more substantial update for you soon.

[AlexLeadingEdge]

Thanks David,

I look forward to their reply.

[david.domask]

Thanks for your understanding and patience @AlexLeadingEdge.

Just my minor input, if it is logs from Veeam One, it might be the Veeam Intelligent Diagnostics: https://helpcenter.veeam.com/docs/one/m ... ml?ver=110

This does a log export and parses the logs looking for known issues you might be hitting and provides you with a convenient VID to give to support when we have a fix for a given issue. The log export happens on the Veeam server and that might be what you're seeing, but 168 TiB seems like a lot frankly speaking, as even with a fairly beefy 4 GiB of logs per export, that would be over 43000 exports; at one scan per day, that would be longer than VID has existed

But, it's possible maybe the export is significantly bigger. Else, I'd start to think towards Database behavior, but let's see the result of the above and also the more specific monitoring.