-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
DR help & next version of Veeam Questions
Hi
We are currently running a trial of VBR. We require it mainly for DR to our remote site over a 10mb WAN, and we are wondering the best way to approach this. Just use replication, or run backup and then move the backup to the remote site using something like rsynch? If anyone can post past experiences or any guidance that would be great.
1 Any DR advice (backup or replication or both) preferably only using VBR for DR across a 10mb WAN (2 live sites, each site is DR location for each other)
2 The current version of Veeam replication creates a Thick disk on the backup site, will the new version allow to replicate to a Thin disk instead?
3 Is there anyway we can seed a replication job by using a backup of vm's then move the backup file across and restore in the backup site?
I have read a lot of posts here and they have gave me a few ides, but i think i have now read too many so am looking for any advice
thanks in advance
We are currently running a trial of VBR. We require it mainly for DR to our remote site over a 10mb WAN, and we are wondering the best way to approach this. Just use replication, or run backup and then move the backup to the remote site using something like rsynch? If anyone can post past experiences or any guidance that would be great.
1 Any DR advice (backup or replication or both) preferably only using VBR for DR across a 10mb WAN (2 live sites, each site is DR location for each other)
2 The current version of Veeam replication creates a Thick disk on the backup site, will the new version allow to replicate to a Thin disk instead?
3 Is there anyway we can seed a replication job by using a backup of vm's then move the backup file across and restore in the backup site?
I have read a lot of posts here and they have gave me a few ides, but i think i have now read too many so am looking for any advice
thanks in advance
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: DR help & next version of Veeam Questions
Hello Michael,
1. I would recommend doing backups locally to perform Guest file level restores or VM file restores. Also I would do replicas to the DR site (in case your main site goes down) to lower the downtime for your production servers. We already have a topic called What I use Veeam for, please have a look, as it is very informative.
2. Yes, our upcoming release will have thin to thin replication
3. For this you need to use replica seeding option which can be found at advanced job settings.This option was exactly made for what you've described above. Please have a look a the user Guide for more information:
http://www.veeam.com/veeam_backup_4_1_user_guide_pg.pdf (page 55)
Thank you!
1. I would recommend doing backups locally to perform Guest file level restores or VM file restores. Also I would do replicas to the DR site (in case your main site goes down) to lower the downtime for your production servers. We already have a topic called What I use Veeam for, please have a look, as it is very informative.
2. Yes, our upcoming release will have thin to thin replication
3. For this you need to use replica seeding option which can be found at advanced job settings.This option was exactly made for what you've described above. Please have a look a the user Guide for more information:
http://www.veeam.com/veeam_backup_4_1_user_guide_pg.pdf (page 55)
Thank you!
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Vitality
thanks for the reply
1 Yes i have read this link and Larry is using a 100MB link, i was wondering if anyone has been using a 10mb link to see if this is still feasible?
2 Good to hear )
3 Yes i know we can seed it onto a USB drive and ship the drive, but it still puts the VM's in thick format on the USB Drive. What i was wondering is can we backup the VM locally using the backup part of VBR which will create a deduped file of say 10 servers which is for example a 10th of the size of the thick provisioned vm's, then put that onto a usb drive and then restore onto the DR site's ESXi servers then point the replication job to the newly restored vm on the DR site and start replicating. Basically i want to know if i can use backup 1st to seed the replication job, or i have only 2 options, copy across WAN or seed via a USB drive (thick provisioned)?
4 Is there anyway you can use VBR to backup locally (dedupe etc) then use VBR to replicate the backup files to another site or does this have to be 2 separate jobs and 2 separate tasks to achieve this? So in the event of a DR i would just restore the VM's using the backup file in the remote DR site? or would i have to use rsynch\double take for something like this?
As far as i can see it i have 2 options that both achieve backup and DR (DR is the most important by far):-
1 Use VBR just for backup locally, then use rsync or double take to replicate the backup to DR site
2 Use VBR to backup locally for local restores and retention to tape + run a separate job to replicate the VM's to the remote DR site for DR
I don't need continuous protection, i have a 4 hour downtime and 24 hour data-loss to meet. As long as i can achieve this, anything better is just a bonus
again if anyone can help, share their knowledge or give experience with rsynch\double take or something similar, or even new suggestions\methods i would greatly appreciate it
thanks again
thanks for the reply
1 Yes i have read this link and Larry is using a 100MB link, i was wondering if anyone has been using a 10mb link to see if this is still feasible?
2 Good to hear )
3 Yes i know we can seed it onto a USB drive and ship the drive, but it still puts the VM's in thick format on the USB Drive. What i was wondering is can we backup the VM locally using the backup part of VBR which will create a deduped file of say 10 servers which is for example a 10th of the size of the thick provisioned vm's, then put that onto a usb drive and then restore onto the DR site's ESXi servers then point the replication job to the newly restored vm on the DR site and start replicating. Basically i want to know if i can use backup 1st to seed the replication job, or i have only 2 options, copy across WAN or seed via a USB drive (thick provisioned)?
4 Is there anyway you can use VBR to backup locally (dedupe etc) then use VBR to replicate the backup files to another site or does this have to be 2 separate jobs and 2 separate tasks to achieve this? So in the event of a DR i would just restore the VM's using the backup file in the remote DR site? or would i have to use rsynch\double take for something like this?
As far as i can see it i have 2 options that both achieve backup and DR (DR is the most important by far):-
1 Use VBR just for backup locally, then use rsync or double take to replicate the backup to DR site
2 Use VBR to backup locally for local restores and retention to tape + run a separate job to replicate the VM's to the remote DR site for DR
I don't need continuous protection, i have a 4 hour downtime and 24 hour data-loss to meet. As long as i can achieve this, anything better is just a bonus
again if anyone can help, share their knowledge or give experience with rsynch\double take or something similar, or even new suggestions\methods i would greatly appreciate it
thanks again
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: DR help & next version of Veeam Questions
Michael,
1. Yes, we do have customers who have 10 or less MB links between Main and the DR site, but you may consider using WAN acceleration tools, such as HyperIP for example.
3. Oh.. now I got it, unfortunately, you cannot use restored VMs to continue replication. But with thin to thin replication coming out with our next release, you wouldn't have such a problem.
4. Yes, you can backup your VMs locally, and then create a copy job to take those files offsite. But note that copy job doesn't track changes, so there are no incremental runs. In this case I would recommend doing backups locally and then using Rsync or similar tools to take those files to the DR site. Here is an example of Rsync usage:
http://www.veeam.com/forums/viewtopic.p ... ilit=rsync
Yes, both scenarios looks good for me. If the restores of the main production VMs take less than 4 hours, then I would use the 1st scenario. But the second variant offers less downtime for your production environment, which is a good bonus.
Thanks!
1. Yes, we do have customers who have 10 or less MB links between Main and the DR site, but you may consider using WAN acceleration tools, such as HyperIP for example.
3. Oh.. now I got it, unfortunately, you cannot use restored VMs to continue replication. But with thin to thin replication coming out with our next release, you wouldn't have such a problem.
4. Yes, you can backup your VMs locally, and then create a copy job to take those files offsite. But note that copy job doesn't track changes, so there are no incremental runs. In this case I would recommend doing backups locally and then using Rsync or similar tools to take those files to the DR site. Here is an example of Rsync usage:
http://www.veeam.com/forums/viewtopic.p ... ilit=rsync
Yes, both scenarios looks good for me. If the restores of the main production VMs take less than 4 hours, then I would use the 1st scenario. But the second variant offers less downtime for your production environment, which is a good bonus.
Thanks!
-
- Veteran
- Posts: 259
- Liked: 8 times
- Joined: Sep 18, 2009 9:56 am
- Full Name: Andrew
- Location: Adelaide, Australia
- Contact:
Re: DR help & next version of Veeam Questions
We're effectively using a 2Mbit link (with WAN Acceleration), if it helps:Hasslehogg wrote: 1 Yes i have read this link and Larry is using a 100MB link, i was wondering if anyone has been using a 10mb link to see if this is still feasible?
http://www.veeam.com/forums/viewtopic.p ... 607#p15480
http://www.veeam.com/forums/viewtopic.p ... 968#p15687
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Bunce
thanks, i will speak to our comms team to see price difference between more bandwidth vs WAN acceleration. Nice figures )
Vitality, thanks for the help
thanks, i will speak to our comms team to see price difference between more bandwidth vs WAN acceleration. Nice figures )
Vitality, thanks for the help
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jun 06, 2010 9:35 am
- Full Name: Magnus Elgebrandt
- Contact:
Re: DR help & next version of Veeam Questions
Hi,
When is the new version due? Thin -> Thin replication is the feature we are missing.
Thanks for a great product,
When is the new version due? Thin -> Thin replication is the feature we are missing.
Thanks for a great product,
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: DR help & next version of Veeam Questions
Hi Magnus,
We are aiming for Q3
We are aiming for Q3
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jun 06, 2010 9:35 am
- Full Name: Magnus Elgebrandt
- Contact:
Re: DR help & next version of Veeam Questions
Thanks for the quick respond. Then I have to contact the EMC guys, we need more disk .
I have just started to test the product. In a DR scenario I understand that you need at least 2 replication points to make sure that you always have one consistent version of the VM at your DR site, is this correct?
Regarding Licensing. If I have 3 host * 2 CPU (at production site) but only replicating to 1 host*2CPU (at DR site) will the product licens to 4*2CPU even when if there are several more host at DR site (used for running the VM but not target in a replication)?
I have just started to test the product. In a DR scenario I understand that you need at least 2 replication points to make sure that you always have one consistent version of the VM at your DR site, is this correct?
Regarding Licensing. If I have 3 host * 2 CPU (at production site) but only replicating to 1 host*2CPU (at DR site) will the product licens to 4*2CPU even when if there are several more host at DR site (used for running the VM but not target in a replication)?
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: DR help & next version of Veeam Questions
Magnus,
I would recommend you to have few restore points to avoid situations when you replicate already corrupted source VMs (infected by some sort of malware). Having rollbacks will make sure you can always restore to a proper VM state. But if you don't want to keep any rollbacks that doesn't automatically mean that your VM might be not in consistent state. This scenario would also work.
You should license only source hosts, not target ones. Assuming you have 3 hosts with 2 CPU each at your production site, so you need to have only 6 sockets license.
Thanks!
I would recommend you to have few restore points to avoid situations when you replicate already corrupted source VMs (infected by some sort of malware). Having rollbacks will make sure you can always restore to a proper VM state. But if you don't want to keep any rollbacks that doesn't automatically mean that your VM might be not in consistent state. This scenario would also work.
You should license only source hosts, not target ones. Assuming you have 3 hosts with 2 CPU each at your production site, so you need to have only 6 sockets license.
Thanks!
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Guys
I need some help please. I want to replicate a vm to our Dr site which at the minute is only over a 2.5mb line. I have seated the vm by replicating onto a usb hard drive then copying it to the exact folder location at the DR site. i have then ran the replication task over the weekend starting friday night and monday morning it is still going? The server is static, hardly anything at all changes with it, maybe 100mb at most, how is the job not complete after 3 days? am i configuring this wrong? i have enabled VSS etc,
if you have any ideas please let me know
thanks
I need some help please. I want to replicate a vm to our Dr site which at the minute is only over a 2.5mb line. I have seated the vm by replicating onto a usb hard drive then copying it to the exact folder location at the DR site. i have then ran the replication task over the weekend starting friday night and monday morning it is still going? The server is static, hardly anything at all changes with it, maybe 100mb at most, how is the job not complete after 3 days? am i configuring this wrong? i have enabled VSS etc,
if you have any ideas please let me know
thanks
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: DR help & next version of Veeam Questions
Michael,
Please make sure CBT is enabled at advanced job settings and check that VMs you replicate meet the requirements below:
http://www.veeam.com/forums/viewtopic.p ... k+tracking
Thanks!
Please make sure CBT is enabled at advanced job settings and check that VMs you replicate meet the requirements below:
http://www.veeam.com/forums/viewtopic.p ... k+tracking
Thanks!
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Hi Vitality
CBT was enabled and i am using the latest version of vSphere ESXi 4 and only Version 7 VM's
any ideas? possible bad seating?
on a server with 100mb of changes over a 2.5mb line whats the kind of timeframe am i looking for? under 1 day right?
i will try and seat again if you think that will help, but basically it was still doing file 6 of 10 when i cancelled it
CBT was enabled and i am using the latest version of vSphere ESXi 4 and only Version 7 VM's
any ideas? possible bad seating?
on a server with 100mb of changes over a 2.5mb line whats the kind of timeframe am i looking for? under 1 day right?
i will try and seat again if you think that will help, but basically it was still doing file 6 of 10 when i cancelled it
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: DR help & next version of Veeam Questions
Michael,
It seems like your job was doing a full run. You should also check for the snapshot presence on the source VMs before enabling CBT, if you do have snapshots, please delete them and re-run the job.
You may try "seeding" only one VM to see if you did everything correctly with your previous attempt. Additionally, you may send all the logs to our support team, so they could tell you if you were running Full or Incremental run last time.
It seems like your job was doing a full run. You should also check for the snapshot presence on the source VMs before enabling CBT, if you do have snapshots, please delete them and re-run the job.
You may try "seeding" only one VM to see if you did everything correctly with your previous attempt. Additionally, you may send all the logs to our support team, so they could tell you if you were running Full or Incremental run last time.
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Vitality
thanks, i will configure 3 seperate replication jobs each only seeding 1 VM each (different VM's) and will let you know how it goes in a few days. I will of course remove any snapshots if they are present
thanks for the help
Michael
thanks, i will configure 3 seperate replication jobs each only seeding 1 VM each (different VM's) and will let you know how it goes in a few days. I will of course remove any snapshots if they are present
thanks for the help
Michael
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Hi Vitality
After your comments, i tried 4 separate server replication jobs, of 4 different servers. I seeded these via replicating to USB 1st using the wizard and shipped the USB to the DR site then followed the intructions in the readme. I created a folder called VeeamBackup on my datastore on the DR site and then copied the 4 different vm folders into this folder (HSTN20(vm-321), HSTN21(vm-449), HSTN26(vm-325) and HSTN32(vm-451). I then tried to run the HSTN20 job at 2am and the HSTN21 job at 5am. i cam in this morning and 20 was at 14% and 21 was at 5%
To tell you a little bit about my setup
We have hosts in Houston and hosts in Chicago (the DR site), all running the latest version of ESXi, we have 1 vCenter managing this environment, which is based in Houston. I have Veeam installed on a physical HP DL360 with 4GB ram and 2 quad core CPU's
These jobs are running for hours and although we only have a 2mb WAN, they should be completing much fast than this, it looks as if its not seeding correctly and doing a full copy? Can you help me please, as we dont have much time left with the trial
thanks
Michael
After your comments, i tried 4 separate server replication jobs, of 4 different servers. I seeded these via replicating to USB 1st using the wizard and shipped the USB to the DR site then followed the intructions in the readme. I created a folder called VeeamBackup on my datastore on the DR site and then copied the 4 different vm folders into this folder (HSTN20(vm-321), HSTN21(vm-449), HSTN26(vm-325) and HSTN32(vm-451). I then tried to run the HSTN20 job at 2am and the HSTN21 job at 5am. i cam in this morning and 20 was at 14% and 21 was at 5%
To tell you a little bit about my setup
We have hosts in Houston and hosts in Chicago (the DR site), all running the latest version of ESXi, we have 1 vCenter managing this environment, which is based in Houston. I have Veeam installed on a physical HP DL360 with 4GB ram and 2 quad core CPU's
These jobs are running for hours and although we only have a 2mb WAN, they should be completing much fast than this, it looks as if its not seeding correctly and doing a full copy? Can you help me please, as we dont have much time left with the trial
thanks
Michael
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: DR help & next version of Veeam Questions
You did seeding correctly, otherwise the job would not start.
I think the primary issue is link speed, while it looks like first increment is coming over pretty large. Overall 2Mb link without any accelerators is definitely not enough for image-level replication of multiple VMs to ESXi. Results would have been much better if you had fat ESX in DR site, this way Veeam can perform network traffic compression, and perform VMDK rebuild locally versus over slow WAN.
I think the primary issue is link speed, while it looks like first increment is coming over pretty large. Overall 2Mb link without any accelerators is definitely not enough for image-level replication of multiple VMs to ESXi. Results would have been much better if you had fat ESX in DR site, this way Veeam can perform network traffic compression, and perform VMDK rebuild locally versus over slow WAN.
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Gostev
thanks for the reply, have you got any recommendations? Will this problem still exist in the next version of Veeam due out this year?
I was worried that the VM was Thin and then when moved to the USB drive that it auto turned into Thick as its on NTFS and then copied to DR site, so it looked like it had changed a lot and thats why it was taking forever.
We are looking to have a 10mb WAN, is this also not good enough? would you recommend we backup and use a tool like rsynch instead?
any help and recommendations would be greatly appreciated
thanks
thanks for the reply, have you got any recommendations? Will this problem still exist in the next version of Veeam due out this year?
I was worried that the VM was Thin and then when moved to the USB drive that it auto turned into Thick as its on NTFS and then copied to DR site, so it looked like it had changed a lot and thats why it was taking forever.
We are looking to have a 10mb WAN, is this also not good enough? would you recommend we backup and use a tool like rsynch instead?
any help and recommendations would be greatly appreciated
thanks
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: DR help & next version of Veeam Questions
No, unfortunately next version of Veeam will not improve your link speed which is the main problem here.
While the next version will support thin to thin replication, this won't affect the replication traffic anyhow. There will be some significant improvements (up to 2x) in reduction of incremental traffic, but looking at your current performance numbers, this won't really help you much.
My recommedations are kind of listed already in the previous post, but I can re-iterate.
Anything listed below will improve the situation:
- increase WAN link speed
- use WAN accelerators
- replicate to "fat" ESX target with our service console agent, instead of ESXi
While the next version will support thin to thin replication, this won't affect the replication traffic anyhow. There will be some significant improvements (up to 2x) in reduction of incremental traffic, but looking at your current performance numbers, this won't really help you much.
My recommedations are kind of listed already in the previous post, but I can re-iterate.
Anything listed below will improve the situation:
- increase WAN link speed
- use WAN accelerators
- replicate to "fat" ESX target with our service console agent, instead of ESXi
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Gostev
thanks again, few more questions if you dont mind
have you got any idea on the percentage increase i would see if i moved to FAT ESX at the destination site?
Does the next version of Veeam fix the only local rebuild on fat ESX and allow ESXi to do it, or will this always be the case as there is no service console?
would the rebuild over the WAN decrease in time if we didnt set any compression\deduplication on the job?
thanks
thanks again, few more questions if you dont mind
have you got any idea on the percentage increase i would see if i moved to FAT ESX at the destination site?
Does the next version of Veeam fix the only local rebuild on fat ESX and allow ESXi to do it, or will this always be the case as there is no service console?
would the rebuild over the WAN decrease in time if we didnt set any compression\deduplication on the job?
thanks
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: DR help & next version of Veeam Questions
No problems
1. I expect to see WAN traffic reduced up to 3 times when replicating to fat ESX w/service console agent, comparing to ESXi target.
2. For v5, architecture of replicating to ESXi will remain the same (but we are considering some enhancements for future release). Note that some customers with vSphere chose to install Veeam Backup server to the target site, next to ESXi. This way, you keep VMDK rebuild traffic local, while using CBT only changed blocks are transfered from source. The implication of this approach is that you are unable to perform replica seeding.
3. No, disabling compression\deduplication will not have any positive effect.
Thanks!
1. I expect to see WAN traffic reduced up to 3 times when replicating to fat ESX w/service console agent, comparing to ESXi target.
2. For v5, architecture of replicating to ESXi will remain the same (but we are considering some enhancements for future release). Note that some customers with vSphere chose to install Veeam Backup server to the target site, next to ESXi. This way, you keep VMDK rebuild traffic local, while using CBT only changed blocks are transfered from source. The implication of this approach is that you are unable to perform replica seeding.
3. No, disabling compression\deduplication will not have any positive effect.
Thanks!
-
- Enthusiast
- Posts: 40
- Liked: never
- Joined: May 28, 2010 2:10 pm
- Full Name: Michael Hogg
- Contact:
Re: DR help & next version of Veeam Questions
Gostev
what about this idea.
Say i replicate locally across the LAN from 1 datastore to another. The rebuild of the vmdk's will be done locally obviously. I then copy the 2nd datastores files onto a usb hard drive, ship to the DR site and copy onto the Dr sites datastore. I then change the job to point to the new location
would this work? would i avoid the rebuild of vmdk as it already done that locally, or does it rebuild the vmdk's everytime it replicates?
thanks
what about this idea.
Say i replicate locally across the LAN from 1 datastore to another. The rebuild of the vmdk's will be done locally obviously. I then copy the 2nd datastores files onto a usb hard drive, ship to the DR site and copy onto the Dr sites datastore. I then change the job to point to the new location
would this work? would i avoid the rebuild of vmdk as it already done that locally, or does it rebuild the vmdk's everytime it replicates?
thanks
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: DR help & next version of Veeam Questions
Michael, unfortunately this again would only work with "fat" ESX being source and target, which can communicate directly during replication due to the presence of service console agent. With ESXi, there are no such agents and all replication traffic has to go through Veeam Backup server (which is located in DR site in this scenario).
Who is online
Users browsing this forum: Baidu [Spider], EskBackupGuy23 and 99 guests