Lotus Notes leads to big incremental files.

[galiral]

Hi everyone,
i was looking for a solution for an offsite backup replication using rsync over a wan link (shdsl 4mbit/s) but the size of the incremental size of the lotus notes mail server is just too big, even using the wan target optimization.
This is the sizing of my backups with veeam 5.0.2.230:



aren't the incremental too big?
maybe i have to ask to reduce the indexing on the database?

i just don't know how can i squeeze them enough to pass these files during the night in a reasonable time, maybe i can replicate the synthetic fulls during the weekend with the hardlink technique posted here http://forums.veeam.com/viewtopic.php?f=2&t=8634, not considering that i have another vm which is a domain controller + sql server + file server and that the situation is quite similar.



i could use some advice on what to do, or maybe i'm just missing something

[ThomasMc]

How much RAM have the servers got?

[Gostev]

Related discussion > backup and replication files are huge

[galiral]

How much RAM have the servers got?

4 gb and 2 cores for every vm, as the veeam vm which have 4gb and 4 cpus

[tsightler]

I would say your results are pretty much to be expected. Even on WAN target mode Veeam uses 256K blocks. Mail servers have a tendency to make many small changes across the entire disk, which leads to large incremental sizes. I typically tell people to expect a 30% change rate on Exchange, and I'd expect Notes to be very similar.

You can mitigate the impact of the block sizes by pairing Veeam with a caching WAN accelerator. We used to see anywhere from 70-90% acceleration on our Exchange server replication when paired with Cisco WAAS, and I've seen higher improvements from customers with Riverbed.

[joergr]

Completely agreed with Tom, especially Mailservers produces huge amounts of db changes, thus, what you see is totally expected behavior.

best regards,
Joerg

[dkvello]

What Domino version ?
Do You use:

1) DAOS ?
2) Transaction Logging ?
3) Full Text Indexing ?

[dkvello]

Some tips to avoid backing up changes that You don't need on Domino


1)

Set the DAOS Base path to alternate virtual disk drive/path outside the Domino\Data directory. This i set in the Server Config Document.
You would want to back up this virtual disk of course, but it will keep the attachment-changes away from Your ordinary data.



2)

Set the Transactaion Logs to a separate Virtual disk. If You use Round Robin (Run-time optimized) You won't need to back up this virtual disk



3)

Put the View Rebuild working directory on separate virtual disk. You don't need to back up this drive:
This will keep You from backing up the vast amounts of temporary changes related to View index updates.

Changing the temporary folder used for view rebuilds

When IBM® Lotus® Domino® rebuilds views -- for example, when you use updall -R or when a user opens a view whose index has been deleted -- it may
generate temporary files to sort the data in order to rapidly update the views; Domino deletes these files after rebuilding the views. By default, these temporary
files are located in your system's temporary folder -- for example, C:\TEMP. If your system doesn't have a temporary folder, then Domino puts the files in
the Domino data folder.

Depending on the amount of memory available during rebuilding, the space required in the temporary folder for each view being rebuilt is approximately two times
the size of the largest view or two times the size of all the data in documents, whichever value is greater. It is recommended that you change the location
of the temporary files to a different drive from the Domino data folder. Putting the temporary folder on a different drive distributes disk I/O and ensures
that there is enough space to rebuild views. Domino is very conservative when estimating the amount of disk space needed for optimized view rebuilds so that it
won't spend unnecessary time sorting data only to discover that there's inadequate disk space. Make sure that the temporary folder you specify has plenty of disk
space available.

To change the temporary folder used for view rebuilds, add the setting View_Rebuild_Dir to the server's NOTES.INI file and specify a new location. For example, add:

View_Rebuild_Dir=D:\REBUILD



4)

If You have Domino 8.5.3 put the FT Indexes on separate virtual disk. You don't need to back up these (You'll save a lot of backup space) as they will be rebuilt after a restore.

Full Text Index on separate Drive

Since 8.5.3 (available as of today) we have the option to put the Full Text Index on another drive. We have asked about it many times for many years and finally got this functionality.
The notes.ini parameter e.g. FTBasePath=d:\full_text can be used to switch the starting directory for the full text index from data directory to a different directory or drive.

It makes a lot of sense to separate the FT Index from the NSF data.
There are a couple of reasons

- FT Index causes a lot of file-system fragmentation
- At some point even with DAOS on larger environments the file-system for NSF data can be quite big and separating FT and NSF would make sense.
- When using snap-shot backup reducing the size of the file-system containing NSF would make a lot of sense
- From I/O point of view on larger servers with a lot of FT data this can also improve performance

To migrate your FT index to a separate drive you need the following steps:

- Set the notes.ini FTBasePath=d:\full_text
- Run updall -f to rebuild all FT indexes.

This will automatically delete the old FT index and re-create it on the new drive.