Block Size Question

[ekisner]

Hi there,

As I'm doing block alignments on my VMs (not a true block alignment, I'm repartitioning everything to use 32k blocks per recommendations from vendor), it comes to mind that I don't know how Veeam pulls data from them. We use Hot Add at the moment, however I'm contemplating putting Veeam on a physical box and plugging it into the storage array for Direct SAN.

So, for both, a question: how does Veeam determine what block size it is going to pull from the disk? It may be a stupid question, as it could very well just be information which is visible from either the LUN or the guest file system, I quite simply do not know.

And a follow-up question, if it doesn't determine it via either the guest file system or the storage array, is it possible to tweak it to what I know the LUN block size is?

[foggy]

Erik, block size used to process VM data can be specified in the job settings.

[ekisner]

Thanks, but I was referring to pulling from the source, not writing to the destination.

For example, the VM vm01, which I want to back up, has 1 VMDK. Said VMDK is stored on a LUN with a 32k block size. We'll say datastore_vm01 houses vm01.vmdk

Veeam hot-adds the VMFS datastore and the VMDK to the proxy.

How does it know what block size to do its I/O with.

[foggy]

That's exactly the block size that Veeam B&R uses to read data from the source datastore.

[ekisner]

Ah. Well in that case, is there any way to change it to a more applicable value? Particularly, 32k to 64k?

After a support call with our storage vendor, it was identified that the array which operates in 32k blocks (or 64k blocks for things like SQL data) is getting a bit overwhelmed when a guest makes a bunch of requests for small block sizes (specifically the default windows block size of 4k). Since our jobs are all set up using local target, this is 4x worse than even a 4k block size.

I recognize that dedup is affected by this, as it's harder to dedup larger blocks, but performance is more important to us than a bit of wasted disk space.

[dellock6]

The underlying VMFS is always creating blocks at 1MB each (sub blocks at 8k are for files smaller than 8k itself). The 32K block is the amount of blocks retrieved per single IO, while "local target" set at 1MB is because final blocks are compared at this size for the dedupe and compression activities.
But if the storage is striped at 32k and Veeam retrieves data at 32k, I'd say the alignment between the two is correct. Or I'm missing something.

Luca

[ekisner]

I could be wrong, but I believe I read a document which says that the VMFS block size is more or less irrelevant; it will not pull a 1MB block for each guest IO, rather it will forward the guest IO to the storage array as-is. A 32k IO from a windows guest for example will hit the storage array as a 32k IO.

My question, and so far how I've interpreted answers, is less about alignment (my fault for using that word) and more about matching block sizes.

It sounds like it reads the data dependent on the previously mentioned Job settings. With a max block size of 8K, they will not be misaligned (it'll still be a 1:1 read operation with no single guest IO requiring more than one storage IO) yet the storage array will still be retrieving 4x the size of the requested block.

This wouldn't be a huge issue with a traditional spindle-based storage array, but this one being a flash hybrid array, makes it a bit of an issue. It will cache data, based on some algorithm. In the case of an 8k read from the B&R server, it's only technically caching 25% of the data - the other 75% of what it's caching is junk in that block.

If Veeam does 1MB of IO at 8k, it's going to need 128 IOs.
The storage array conversely does 4MB of IO (128 IOs * 32k block sizes), caching 4MB into flash/memory so that Veeam can get the 1MB.

To make the numbers look scarier, consider Veeam doing 1TB of IO at 8k - the example holds true at any size.

[dellock6]

Probably I need to re-quote this:

That's exactly the block size that Veeam B&R uses to read data from the source datastore.

So, as I said, Veeam requests blocks at 32k. If the storage reads blocks at 32k, it's the same size, and there's no read amplification.
The 1MB block size of Local Target option is the size of the block used in Veeam to do deduplication and compression, but it happens after it's retrieved from the production storage.

[ekisner]

Oh! I misunderstood then.

I was under the impression that if the job storage optimization settings were set to Local target (16 TB + backup files) it would read and de-dup in 8k chunks.

If Veeam is indeed reading at LUN block size, then that answers my question. Thank you Luca.

[taurus1978]

Hello all,

does anybody know how many read i/O VEEAM produces on the "source" (Production) storage systems ?
I only found Information about Repository Performance and Sizing. There we have a nice Guide from Luca on this...
But what about Read I/O and Performance ?

Which mechanism decides what blocksizes are used to perform a read operation from the production storage. ?
Is this the same Blocksize i set up in the Backup Jobs for Loacal, LAN or WAN Target ?

Then additionally how many I/O can VEEAM pull out of the production storage during read operations ? i know i can set Storage Latency Controls. But this only refers to latency...

Maybe this depends on the Backup Mode (DirectSAN, LAN or Hot-add) ?

Simple example:
When i set up a Backup Job with Forward Incremental. Let us say we hav 10 VMs with 2 .vmdks each.
With 100 GB per .vmdk. So there is a source capacity of app. 2000 GB (2TB).


i will be happy for any information to this.

Thank you folks.

[foggy]

Hi Patrick, you can get some details from the thread above.

[crackocain]

Hi Everyone

I want to ask write io size for target storage sizing. I create Disk Magic report but write io size change the throughput a lot.

For example if 32K write io selected, storage throughput is 300MiB
64K write io selected, storage throughput is 700MiB

If Veeam direct access this storage through SAN (ReFS or NTFS 64k formatted) what is the exact IO size?
Storage option (Local target)

[Gostev]

There's no exact I/O size as Veeam does not work with the storage directly. Instead, all I/O goes through the OS cache, which will do additional I/O grouping in order to do bigger writes, as it helps performance. But in the worst case scenario of a single block being updated in a backup file, I/O size for "Local target" option will be whatever 1MB of source data compresses too (or around 512KB on average, as typical compression is 2x).

[crackocain]

Thanks Anton.

Actually bigger io size is great. I'm afraid io sizes are down by the OS to 128k or 64k. Backup throughput performance is painful this scenario. Worst case scenario 512k is great