copy backups to usb-disks

[Jambari]

Hi,

i'm switching to a new backup-solution... i'm trying out Veeam Agent for Windows Free.

does any one has experience with backups to a internal drive and then every now and then copy these files to several external usb-drives? are there any known issues?

my previous backup software created .tmp files and renamed them after completion. so i could run "ROBOCOPY.exe d:\internal-stored-backups e:\external-stored-copy /MIR /XF *.tmp"

i found out that veeam does not do tmp files... it writes directly the vbm (metadata), vbk (full), vib (incremental) files to my internal d: drive.

the vbk and vib files are readable from robocopy while a veeam-backup is in progress. such a partial written vbk/vib file can be copied by robocopy and robocopy will not throw any errors. but in the end there are incomplete backup files on the usb drive.

does this lead to any errors? in my tests veeam did not let me choose partially written vib incrementals... i think because veeam updates the metadata/catalog at the end and only files in this catalog are presented for restore.

is there any other "best practice" to copy/mirror all the local backups to external drives?

is there any command line tool from veeam itself to copy a backup folder to another drive?

[Mildur]

Hi Jambari

Is this for a home lab/home computer or for a production environment?

If you want to use RoboCopy or any other tool, do not copy any data before the backup is finished. You can't use incomplete backup files for a restore.

is there any other "best practice" to copy/mirror all the local backups to external drives?

Option 1
With Veeam Agent, you can create a second backup job to another location, but only with the server edition. Workstation Edition can send a second copy to a Cloud Connect Provider.

Option 2
You can also use Veeam Backup & Replication Community Edition (free of costs until 10 workloads), which lets you create an agent backup of your workstation/server and then do a backup copy Job to an usb disk (rotated Drive).

is there any command line tool from veeam itself to copy a backup folder to another drive?

Without a second backup job configured, the command line cannot help you a your goal.

Thanks
Fabian

[Jambari]

Hi Fabian

i'm currentyl playing around in some VM's at Home... but when it then finally works, i'd like to use that for production in the end.

i like to backup all my machines (linux, windows) to a folder (network share) and for every machine i use a subfolder named after the machine. we do not use one backup solution... we use mutliple... some machines only get their mariadb's exported. some machines get block-based backups of all the drives they have. some machines get database and rsynced-file backups. it depends on the need and it also depends if we can automatically rebuild the vm's ourself and only need to restore a database in that machine. and on some machines we only have access as a user (shared web-servers), so we cannot install software there. some production servers are considered critical... and there we only use debian on these... and only packages that are really needed, nothing else... no external repository is wanted there. so we cannot install veeam or anything non-debian there.

so it is a mix of diffrent tools we use. but all of them write to these folders... and periodically we insert an usb-drive and sync all the backups (folders) to the usb drive... this drive gets air-gapped and the next time another usb-drive will be used... so we have multiple usb-drives with no connection to the internet or any computer in diffrent places.

i will try option 2... Veeam Backup & Replication Community Edition... but i somehow doubt that it will well integrate with our current solution. maybe i can copy all our folders except the ones with veeam backups with robocopy and after that i could run the backup copy job to my external drive... somethin like this... we will see how this works...

[Mildur]

Hi Jambari

Your welcome.
My recommendation is to keep the installation simple. Using multiple backup methods and applications can make the backup environment really complicated to manage. If you can consolidate into one tool, it would safe you time.

If all this machines are VMs (vSphere, HyperV, Nutanix AHV or RHV), you can use Veeam Backup & Replication to create a backup of the filesystem and the application. No Veeam Agent required. Veeam currently has integration into MySQL, Oracle, Exchange and SharePoint. Here is the link to our quick guide.
Backup the VM to the network share, use a Backup Copy Job to copy the backup to the USB Disk.

and for every machine i use a subfolder named after the machine

With VM Backups, you won't get Folder per Machine backups, but you can activate Backup Chains per Machine. What is the goal in using a different folder per machine? If Veeam is doing all the magic (backup job + backup copy job), the folders doesn't really matter.

some machines only get their mariadb's exported.

This still needs to be done by your own tools. You could run a script in the veeam backup job todo it before the veeam backup.

i like to backup all my machines (linux, windows) to a folder (network share)

Just FYI, we don't recommend using network shares as a backup repository. We see bad experiences from support cases when using Network share as a primary backup target.
We recommend general purpose servers with build in disks. You could use a standalone installation. VBR and Backup storage for the primary backup job on this system.

[Jambari]

Hi,

we have no problems to use multiple backup methods. consider your router... how do you backup your router? we just cron an ssh into and grab the config. of course, a unique tool, but it does excactly what it should do. we have many diffrent systems, also raspberry pi's and cnc-machines... seperate syslog servers, we just need the logs... we can rebuild them automatically if they die... and so on... i dont think any single tool will or should support everything.

for vm's we use also multiple systems... proxmox, kvm and virtualbox... it depends what the usecase for that particular system is.

i like to keep it simple... thats what these folders are for... one folder for one machine... you need to know the machine name and then you can get your data... and you have the documentation also in that folder so you know how to restore or what maintanance tasks to start after restore. one machine has 20 maria db's... so we backup these db's to subfolders... it is really an easy to use system.

why are network shares a problem? just when the share is from a windows system? or also from a linux/samba share? we use network shares all the time for diffrent things (windows/samba and nfs) for lots and lots of years and never had any issues.

maybe you should consider hardening the software against connections losses. as long as one cannot use transactions on a filesstem we have to do stuff like to name the things .tmp while writing and when everything is complete rename all the things... like these moderns browsers do with the .incomplete extension or something. or you put a file like "forcefsck" before you begin and delete when done... so if your application find this file in the repository you know it needs a check... like linux does this. network shares can disapper at any time...


when i open the Veeam Console i have this Menu "Files" where i can copy folders... i copied the repository to another drive... that sould do the trick... it seems that not windows is copyin the files but veeam is. so veeam knows how to copy own files... is there a way to automat this? command line? like veeam.browser copy "src" "dst" or some powershell thing?

i would like to copy all the backups in the veeam repositiory to the usb drive from a script... not from the gui. and do i really have to create two copy jobs? one for all the windows machines and one for all the linux machines? can i not only copy the whole repository?

[Mildur]

Hi

we have no problems to use multiple backup methods. consider your router... how do you backup your router? we just cron an ssh into and grab the config. of course, a unique tool, but it does excactly what it should do. we have many diffrent systems, also raspberry pi's and cnc-machines... seperate syslog servers, we just need the logs... we can rebuild them automatically if they die... and so on... i dont think any single tool will or should support everything.

Yes, that makes sense. I assumed you are talking about different modes for VMs and physical windows/linux workstation/servers.
For special hw appliances like firewall or switches normally other backup methods are required.

why are network shares a problem? just when the share is from a windows system? or also from a linux/samba share? we use network shares all the time for diffrent things (windows/samba and nfs) for lots and lots of years and never had any issues.

In many cases where customers have opened a support cases for corrupted backup files, some cheap nas was used as a backup repository. And file share based backup repositories can most likely not leverage our FastClone technology. Without FastClone, synthentic full backup will use a large amount of storage.
Our recommendation is to use general purpose servers with local disks and an enterprise raid controller for Veeam backup and Replication. I understand, that not all customers are willing to pay for something like this, but it's the most stable backup repository you can get. And you can use it as a Linux Hardened Repository to protect your backups against ransomware attacks. That's not possible with fileshare based repositories.

maybe you should consider hardening the software against connections losses. as long as one cannot use transactions on a filesstem we have to do stuff like to name the things .tmp while writing and when everything is complete rename all the things... like these moderns browsers do with the .incomplete extension or something. or you put a file like "forcefsck" before you begin and delete when done... so if your application find this file in the repository you know it needs a check... like linux does this. network shares can disapper at any time...

Thanks for the request. Currently we don't have plans to optimize Fileshare based backup repositories.
For all windows and Linux based backup repositories with local disks (and a few dedup appliances), we have our data mover which does check sum calculations, if every block was received without corruption. If not, the block will be send again. We can't install our data mover on NAS Appliances.

when i open the Veeam Console i have this Menu "Files" where i can copy folders... i copied the repository to another drive... that sould do the trick... it seems that not windows is copyin the files but veeam is. so veeam knows how to copy own files... is there a way to automat this? command line? like veeam.browser copy "src" "dst" or some powershell thing?

You can use File Copy Job, but this job won't remove older files from the source. You can also use our NAS Backup Job Feature to backup the files.
Copying Veeam Agent Backup Files with a File Copy Job is not best practice. But if you want todo it, do it. Just wait until the Backup Job has finished. A Veeam Backup Copy Job would automatically wait.

i would like to copy all the backups in the veeam repositiory to the usb drive from a script... not from the gui. and do i really have to create two copy jobs? one for all the windows machines and one for all the linux machines? can i not only copy the whole repository?

With V12, a single Backup Copy Job can be used for Veeam Agent Backups.

Thanks
Fabian

[Jambari]

Hi

We have here some Development-Machines that do hourly backups and they have their own GB databases... in my tests i found out that Veeam Agent is always reading at least 2.2GB of Disk-Data even if the last backup is only 1minute old and we changed no file (written are just between 8mb and 12mb) and when we do a single change in a 10gb file it will read the whole 10GB of the file + the 2.2GB (i guess mft and ohter maybe internal stuff).

i guess that V12 will also not have support for CBT-Driver installation on non-server Windows machines? so that we could do fast, efficient, hourly small backups?

some dev-machines have multiple 100gb+ vm's so they can test diffrent os/software combinations, create snapshots and revert the systems. then it gets even worse when veeam reads the whole vm-disks every hour... CBT-Driver on Worksations would be nice to have. is that planned?