Comprehensive data protection for all workloads
steve21
Novice
Posts: 8
Liked: never
Joined: May 17, 2012 5:35 pm
Full Name: Stevin Truijen
Contact:

[MERGED] bacup repository

Post by steve21 »

Hello,

i am currently looking at different NAS devices to use as a backup repository. i was looking at the Drobo's, Synology, and QNAP. what would you recommend for a NAS repository. i would like it to hold 4-5TB of backup space. Value for the Dollar will be a major selling feature. the though put is limited to 150MB/s so i would like to take advantage of all the speed that i have over the network. i would mainly like to know what you are using and what you might recommend.
jpeake
Enthusiast
Posts: 88
Liked: 25 times
Joined: Sep 25, 2012 7:57 pm
Contact:

Re: [MERGED] bacup repository

Post by jpeake »

steve21 wrote:Hello,

i am currently looking at different NAS devices to use as a backup repository. i was looking at the Drobo's, Synology, and QNAP. what would you recommend for a NAS repository. i would like it to hold 4-5TB of backup space. Value for the Dollar will be a major selling feature. the though put is limited to 150MB/s so i would like to take advantage of all the speed that i have over the network. i would mainly like to know what you are using and what you might recommend.

Stay away (far, far away) from any of the SOHO models that use Marvell CPU. Other than that, i think those companies all offer decent products.
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: recommendations for veeam backup storage

Post by habibalby »

Hi, why don't you consider 1TB Near-Line SAS drives 7,200 rpm with IBM DS3500/DS3512 iSCSI SAN Storage. they won't give you high read/write but low cost and good since you will use them for Repository only.

Thanks,
dellock6
VeeaMVP
Posts: 6155
Liked: 1968 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: recommendations for veeam backup storage

Post by dellock6 »

Hussain, I think all depends on the budget.
The DS storage is anyway more expensive than "soho" storages like Qnap/Sinology, also given the fact those can be filled with off-the-shelf sata disks, while usually storage systems from main vendors require dedicated disks (sometimes with modified firmware, and high priced), and even if they are the same disks you cannot plug disks you have not bought from the vendor itself.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
steve21
Novice
Posts: 8
Liked: never
Joined: May 17, 2012 5:35 pm
Full Name: Stevin Truijen
Contact:

Re: recommendations for veeam backup storage

Post by steve21 »

Thank you for the Advice i will take it all into consideration on my hunt for a NAS device. i was looking at the IBM recommendation and that is a little overkill for what i need my backup job to do currently. if i was going with dedupe then i would consider something higher end. i will also make sure that i stay clear of the Marvel CPU's.

thank you for all of you input.
Cokovic
Veteran
Posts: 295
Liked: 59 times
Joined: Sep 06, 2011 8:45 am
Full Name: Haris Cokovic
Contact:

Re: recommendations for veeam backup storage

Post by Cokovic »

I've recently used a Thecus NAS at one of my customers and was very satisfied with the quality and performance of this NAS. Used a N8800 Pro V2. Quite similar pricing to QNAP and Synology.
NetWise
Influencer
Posts: 13
Liked: 3 times
Joined: Aug 26, 2009 5:52 pm
Full Name: Avram Woroch
Contact:

Re: recommendations for veeam backup storage

Post by NetWise » 2 people like this post

Something to keep in mind, that hasn't been mentioned. A few people have commented that the speed of their backups/backup repository aren't really what they're worried about, slow storage will do, etc. This is certainly true for writing, and when doing incremental/differentials with CBT nightly. But on the day when you need to do restores, and more than just something like a 1VM SureBackup, please make sure that the speed of your backup repository will support your RTO. I've seen a lot of people backup to slow disks, because a 3 hour per night Reverse Incremental is fine. If a SAN or LUN failed, and you had to restore 10, 20, 50 VM's, the speed of that backup repository might start to matter, very quickly.

In that same vein, something that is important as well is that if you are on say 10GbE for ISCSI, and you give your Veeam server a 10GbE NIC to see the SAN quicker for your backups, it might be wise to do the extra work to confirm and test that your vPower NFS is kicking in on the 10GbE link. If you try to boot up 40 VM's direct from backup, and do svMotion to bring them back to primary storage, across your single 1GbE link from the VM, it could be much much slower than you expect.

Just saying to remember that you're building a "recovery solution", NOT a "backup solution".
mlinders
Veeam ProPartner
Posts: 26
Liked: 8 times
Joined: Apr 22, 2012 10:42 pm
Full Name: Mark Linders
Location: the Netherlands
Contact:

Re: recommendations for veeam backup storage

Post by mlinders »

We mostly use Synology for our backup repositories, connected using iSCSI.
From our experience this has been perfectly stable in 20+ customer environments.
Most used are the RS2212RP+ and RS811RP+ (small environments perform well on DS213+, DS412+, etc.)

Important to mention:
We only use Enterprise class disks, so no desktop sata drives.
Only issue we have seen is errors in VMware vSphere due to HDD hybernation, disabling the sleep feature in Synology fixes this issue.

Recently we swapped out our HP MSA storage solution for a Synology RS3412RPxs and seen a massive increase in backup speed.

Some of the posts here saying that iSCSI on Synology is not reliable, is something I absolutely do not see in any of the environments we are using. Only issue we had with iSCSI was due to a faulting NIC firmware in the server.
guru_reggie
Influencer
Posts: 14
Liked: 1 time
Joined: Nov 15, 2012 9:44 am
Full Name: Michael Hoare
Contact:

[MERGED] DAS Storage recommendations

Post by guru_reggie »

Hi all,

Currently have Veeam B&R 6.5 running on a VM (2008 R2) with a few 2TB vmdk's attached as separate drives...

Recently I've been getting close to my thresholds when backing up.. Rather than consider moving to RDM's or Dynamic Disks across vmdk's, I'm thinking of making my server physical (2012 maybe) and attaching some storage to it.. The SAN I've inherited doesn't support replication of more than 2TB per lun, and has no dedupe available..

So I guess I'm open to suggestions on vendor etc for a DAS..! I've used IBM DS3512/3524 before, but my current company are mainly HP or Dell people.. I've had a look online (without wanting to speak to resellers just yet) but don't see anything at a glance that supports maybe 20TB or so and maybe including it's own dedupe as well...

Thoughts would be most welcome :)

Many Thanks,
Reg..
dellock6
VeeaMVP
Posts: 6155
Liked: 1968 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: recommendations for veeam backup storage

Post by dellock6 »

Really? I'm not so informed about Dell offering (apart knowing the name of the appliance is DR4000), but HP has the StoreOnce series of appliances, and they just launched the new platform in these days at HP Discover. They can start with small versions and then scale when needed. Also, there are DataDomain from EMC, or ExaGrid.

But, before going to products, you'll be better to design your requirements, and then try to map these requirements to the different existing solutions, so you can be sure the chosen one is that beeing nearesst to your needs.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
eysfilm
Novice
Posts: 3
Liked: never
Joined: Dec 17, 2012 8:24 pm
Full Name: Shawn Peters
Contact:

[MERGED] Recommended Storage Units

Post by eysfilm »

I'm just putting together a proposal to get Veeam backing up all our schools but I'm wondering what you all would recommend using for storing all the backups to our board office. Been reading all the forum's and it's all over the space (de-dup appliances, linux boxes, etc.). I'm just getting confused now.

We have 40 schools averaging about 400-500GB of data each, all running ESXi 5.x. I'd like to allocate about 1TB of data per site for future expansions and backup revisions. All the backups would be coming back to our central board office over our WAN (don't worry about bandwidth). Once the backups are at the board office we don't need to push them offsite anymore. The ability to expand the storage in the future would be nice, but it's not a requirement. Looking around the $15,000 mark (give or take) :P

What would be a good solutions to use to store all the data? What are other using, and what's your experience with the devices.
If your using any of these devices, what are your average de-dup/compression ratios?

I've been looking at the following, but not sure what our best option is.
- Dell DR4000 (this looks pretty nice)
- DataDomain DD160
- Synology RackStation RS3413xs+
- QNAP TS-EX1679U
- Netgear ReadyNAS 4200
- or a Windows Storage 2012 server with as many drives that will fit. Biggest unit HP, Dell or other vendor can offer.

Suggestion? Thoughts?

Thanks in advance....
Gostev
Chief Product Officer
Posts: 31606
Liked: 7097 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: recommendations for veeam backup storage

Post by Gostev »

Hi Shawn - I've merged your post into main discussion thread since you list multiple models, but also be sure to search the forum for specific models listed, as many of them have a dedicated discussion with more details. Thanks!
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: [MERGED] Recommended Storage Units

Post by chrisdearden »

eysfilm wrote:I'm just putting together a proposal to get Veeam backing up all our schools but I'm wondering what you all would recommend using for storing all the backups to our board office. Been reading all the forum's and it's all over the space (de-dup appliances, linux boxes, etc.). I'm just getting confused now.

We have 40 schools averaging about 400-500GB of data each, all running ESXi 5.x. I'd like to allocate about 1TB of data per site for future expansions and backup revisions. All the backups would be coming back to our central board office over our WAN (don't worry about bandwidth). Once the backups are at the board office we don't need to push them offsite anymore. The ability to expand the storage in the future would be nice, but it's not a requirement. Looking around the $15,000 mark (give or take) :P

What would be a good solutions to use to store all the data? What are other using, and what's your experience with the devices.
If your using any of these devices, what are your average de-dup/compression ratios?

I've been looking at the following, but not sure what our best option is.
- Dell DR4000 (this looks pretty nice)
- DataDomain DD160
- Synology RackStation RS3413xs+
- QNAP TS-EX1679U
- Netgear ReadyNAS 4200
- or a Windows Storage 2012 server with as many drives that will fit. Biggest unit HP, Dell or other vendor can offer.

Suggestion? Thoughts?

Thanks in advance....
Hi Sean,
Thats quite an interesting setup you've got there. Are you going for a local backup repository as well or will the HQ storage be the only location ? Have you done any small scale testing ? if so, how much data and at what kind of rate does your backup run at ?

In terms of repository performance, I wouldn't expect it to be a the bottleneck in your design - ultimatly the WAN is going to determine how fast you can write to that storage.

How many jobs did you want to run at a time ? if you wanted to run them all at once, you'll need a good amount of RAM on that repository server to hold the dedupe hash table.

How much retention did you want ? If you allocate backup space to your production storage space in a 1:1 ratio you should get at least 14 restore points based on a reverse incremental, however this can go up or down based on data change rate.

For all of that I think you would be looking at a windows Storage server - the other options will still require some form of proxying server at your HQ side to get efficient wan utilisation.
eysfilm
Novice
Posts: 3
Liked: never
Joined: Dec 17, 2012 8:24 pm
Full Name: Shawn Peters
Contact:

Re: recommendations for veeam backup storage

Post by eysfilm »

Initial backup would be local then we would transfer files to core backup unit, but subsequent backups would all be done over the WAN. Each school would be running it's own local proxy host to help with backups. On high range were maybe looking around 1-3GB of changes nightly per school, so WAN backups are looking around the 30-45 min mark.
We need to run the 40 school backups nightly, but they could be split into chunks as the backup window in pretty large (8pm-6am). Retention would be as long as possible. We would have to play with this as more and more time passed, but we would be looking for at least 1 months worth of data. This is what are current local backups at the schools are getting.

Thanks.
Alaerus
Lurker
Posts: 2
Liked: never
Joined: Dec 13, 2011 4:00 pm
Full Name: Stephen Dye
Contact:

Re: recommendations for veeam backup storage

Post by Alaerus »

I use a Drobo filled with WD red drives, got about 21TB for 5k, plus the Drobo can be used as a SAN if ever needed.
andersonts
Veteran
Posts: 307
Liked: 31 times
Joined: Mar 21, 2012 9:56 pm
Full Name: Tim Anderson
Contact:

Re: recommendations for veeam backup storage

Post by andersonts »

Hi Sean for the price point you are looking for you can probably rule out a Data Domain or DR4000 as they are a bit more than that I believe. I like your option of using DAS behind the Windows server personally because for 15k you can get a pretty nice chunk of space...not sure if you will be able to get 40TB usable for 15k but you will get close. Dell R520 is a nice platform but HP and IBM have similar offerings (I am more familiar with Dell but really doesn't matter as they are all basically the same thing with different cases). I would assume that of your 40 sites there will be quite a bit of duplicate data with forms and policy letters etc. so you might even be able to use the Windows 2012 Dedupe on top of what Veeam gives you. I haven't seen anyone out there with 40TB yet using a Windows 2012 dedupe policy but I know there was at least one post where someone had maybe 15 or 20TB and reported it was working well...based on your averages it sounds like you have about 20TB of data total currently which could compress down to 5TB or less so in that case you wouldn't need much storage space at all! As Chris mentioned don't skimp on processor or memory if you go that route...not just for the hash table but also for the Windows dedupe processing.
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: recommendations for veeam backup storage

Post by habibalby »

Hi,
Happy new year in advanced.

Just I would like to through a hint in here so it might be considered with selecting a Repository Storage.
[*]Remember, Veeam gives the flexibility to run your backup/VM from disk.
[*]Remember the IOPS requires by your VMs if you want to run it from the repository Storage.
[*]Remember type of failure you have, complete VM crash, one VM, entire production storage, plan for worse case scenarios otherwise you will be blamed by your boss that you didn't implement it carefully, blah blah blah of fu**.
[*]How fast you want to recover, that depends on your RTO... So, disk speed is important as well and that differ between organization to another.

Writing this from an experience point of view, as I'm in trouble I can't run all of my VMs from my repository as it's IBM DS3512 with NearLine SAS 7200 disks... My boss has gone with this over EMC equal to our production SAN just because of price:) I'm clever enough that I wrote an email to him stating all of the above. SO, I'm protected :)

Hope it helps.
lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: recommendations for veeam backup storage

Post by lohelle » 2 people like this post

An example of possible backup storage.

SC847E26-R1400LP case, 36 3,5" slots, 2 x 6G SAS exp $1900 http://www.supermicro.com/products/chas ... 1400LP.cfm
Supermicro 2-way MB $500
2 x CPU $800
32 Gig DDR3 ECC REG MEM $500
LSI MegaRAID SAS 9271-8iCC, 8-port (2x8087) + CacheC/FP $900
36 Seagate 3TB 7200 RPM ($130 each) $4680

Total would be in the $9k range.

This would give 51TB of RAID10 storage (2 x hotspare), 96 TB of RAID6 storage (2x hotspare)
I think RAID10 would be best if using Instant Recovery. 36 disks should give good IOPS.

I have one of these cases (trays are included on these by the way), a Supermicro MB with IPMI/KVM, 32gig of RAM and a combination of 3,5" SATA-drives and Samsung 830 SSD's
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: recommendations for veeam backup storage

Post by Yuki »

Lohelle
- i'm hearing more and more companies warning not to use desktop grade drives without TLER.

Anyone tried SansDigital products?
Just stumbled onto these guys with a monster 72x4TB iSCSI NAS - http://store.sansdigital-shop.com/elen8u72bayw.html with 10G available as an option and built-in replication features.

Of course something like this 24 bay with 2x4 1Gbit Active/Active controllers is a more reasonable option - http://store.sansdigital-shop.com/acar4u24bay8.html

It's not as cheap as building one, but probably more reliable with dual controllers, all the components are guaranteed to work together and will be clean on the inside (which can be an issue with custom built units).
ddayton
Enthusiast
Posts: 46
Liked: 19 times
Joined: Nov 12, 2012 6:40 pm
Full Name: Don Dayton
Contact:

Re: recommendations for veeam backup storage

Post by ddayton » 2 people like this post

I also needed to get started on a backup storage solution with little to no budget last year. I have some money allocated this year to add the second VMware server in the DR site, but plan to continue using the cheap NAS storage I have in place for at least the near future.

Several years ago I needed a few TBs of NAS storage that was needed to archive video from our security surrveillance system. I did a lot of comparisons of products and features and stumbled upon an OpenSource project called FreeNAS. Unbelievable what you can get for free. I installed FreeNAS V2 64-bit on a used desktop PC with a quad core CPU (Q8400), 4GB RAM, 4GB USB Flash Drive (boot device) and four 750GB SATA drives. I set this up as RAID 5 volume shared using CIFS protocol. The unit has been online and stable for three plus years, so when I needed NAS storage for VMware and Veeam B&R I decided to use the same basic setup substituting the 750GB drives with 2TB drives. With only 4GB of RAM you cannot use the four 2TB drives in a RAID 5 because they recommend 1GB per 1TB of disk space, so I went with two 2TB drives as RAID 1 with two volumes. After benchmarking CIFS verses iSCSI protocol with FreeNAS I found that iSCSI was a little faster and more reliable with Veeam B&R. I have two FreeNAS units running offsite connected via 1Gb fiber. One is an iSCSI target for the SQL Server dump files and the other is an iSCSI target for Veeam B&R. This has been our configuration for a few months and all is going well (as far as NAS storage). Having a little issue with SQL Server Veeam Backup where the snapshot consolidate is causing us problems, which hoping during our next outage the VMware updates we apply will help.

FreeNAS has all the bells and whistles to support network link aggregation too, so utilizing several NIC connections can improve target access. I have experimented with three 1Gb connections but the FreeNAS server locked up and I think it was due to the test PC only had 1GB RAM, but it was really fast up until it froze several hours later. While on a single NIC is didn't lock up.
jorgene
Enthusiast
Posts: 77
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: recommendations for veeam backup storage

Post by jorgene »

The Netgear Ready NAS 2100 i pretty cheap compared to the storage and performance you get from other vendors in same price range.
But use ISCSI NOT SMB on it. SMB performance suck big time, but ISCSI rocks !
There are different HDD types, 2100 have 4slots so with 2TB HDD you get about ~5.5TB data store with raid. Now i think there are supported 3TB hdd's also.
You can buy the device without HDD's and buy supported HDD's and install (the most cheapest way) or buy the unit with HDD's installed.
rciscon
Influencer
Posts: 21
Liked: 3 times
Joined: Dec 14, 2010 8:48 pm
Full Name: Raymond Ciscon
Contact:

[MERGED] Recommendations for an Inexpensive NFS Array Backup

Post by rciscon »

We've been a VMware and Veeam shop for several years now with vSphere 4.1 and 5.1 installations at our company HQ and satellite offices.

At Tier 1 locations we have vSphere production running on low-end NetApp SANs and we're using SATA disks on the ESX host as a target for Veeam backups.

At Tier 2 locations we have both vSphere production and backup target running on SATA disks on the same ESX host--FAR from an ideal solution, but better than no backups at all.

We're approaching budget time and I'd like some advice and recommendations on a low cost (under $1000 without disk) NFS-based external array that I could connect to my ESX host as a target for Veeam backup jobs.

In the past I've tried Drobo devices, but their low performance and non-existent management has removed them from consideration. I've played with a NetGear ReadyNAS Pro 4, but performance seems quite slow and I had some configuration issues that required working with NetGear support that did NOT give me the warm fuzzies.

Does anyone have any recommendations?

I could be sold on the NetGear ReadyNAS, but I'd like to hear some word from people who have them in production.

I'm also thinking of another option that I'm going to outline in another post.

Thanks in advance,

Ray Ciscon
Senior Systems Engineer
MacLean-Fogg Company
foggy
Veeam Software
Posts: 21124
Liked: 2137 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: recommendations for veeam backup storage

Post by foggy »

Ray, you've been merged into the corresponding topic, please review and feel free to ask any additional questions. Thanks.
HDClown
Enthusiast
Posts: 45
Liked: 6 times
Joined: Dec 27, 2012 12:25 pm
Contact:

Re: recommendations for veeam backup storage

Post by HDClown » 2 people like this post

Check out the Synology DS1513+ and DS1813+ (both fit your budget). Both use the Intel Atom D2700 Dual Core 2.13ghz process and they can saturate a GigE link. I will be purchasing a DS1813+ as a backup target at my off-site location.

I personally own a DS1512+ which has the same CPU as these two I mentioned and I max out a GigE link to it all the time on large file transfers.

I don't think you'll be disappointed with Synology for this need.
robbyde
Novice
Posts: 4
Liked: never
Joined: May 22, 2013 11:07 am
Full Name: Rob Delbridge

[MERGED] New SAN for Veeam Backups?

Post by robbyde »

Hi All,

I'm looking for a cheapish SAN which will specifically used for Veeam backups.

I've been looking at QNAP SANs and would probably need at most 8TB storage at most but potentially with the capacity to upgrade. Sorry should have mentioned, RAID 5 would be great in a ideal world too.

Does anyone have any recommendations?

Thanks
foggy
Veeam Software
Posts: 21124
Liked: 2137 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: recommendations for veeam backup storage

Post by foggy »

Rob, I've merged your post into existing discussion that could give you some advises regarding your question. You can also search for other similar topics on these forums for more information. Thanks.
PaulNSW
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 14, 2012 10:48 am
Contact:

[MERGED] What's your Backup Target?

Post by PaulNSW »

We're currently backing up the data from our EqualLogics to an old MD3000i. The MD is becoming more and more unsupported, the last VMware was 4.1. We have it running on 5.1, but an upgrade to 5.5 introduced some strange errors.

We currently have around 30 days of recovery points in 18TB of storage

Just wondering what you are all using, what VMware version you're on, and if you would recommend it as a target!
foggy
Veeam Software
Posts: 21124
Liked: 2137 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: recommendations for veeam backup storage

Post by foggy »

Paul, some recommendations can be found in this topic above. Thanks.
dellock6
VeeaMVP
Posts: 6155
Liked: 1968 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: recommendations for veeam backup storage

Post by dellock6 » 2 people like this post

Damn Foggy, I just wrote a pretty long reply and I lost all my text because you moved the thread :P

Anyway, Paul, there are indeed many different solutions for your needs, but I'd like to check first of all with you, what are the pain points of your actual solution? Because if it's only the unsupported state, I'm not getting it: backup jobs do not use ESXi as targets, so a storage that is not supported by VMware, can still be connected as a direct attach to a physical server with Windows or linux on it, and be used as a Veeam repository. This is common practice, I did it myself at several customers when they upgraded their production storage with a new solution, the old one often becomes a new backup target.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
foggy
Veeam Software
Posts: 21124
Liked: 2137 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: recommendations for veeam backup storage

Post by foggy »

Sorry, Luca, I'll like your post, by way of apology. ;)
Locked

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], JaySt and 97 guests