I think you are providing the correct level of communication to your users. I appreciate the updates along the way especially when all my friends say, “Why is foursquare down”. I just keep RT the updates and then everyone knows. Good luck with your updates, hope it streamlines your database issues and makes 4sq king!

Thanks for all the work you put in for keeping people informed – helps keep peace of mind. I respect that and hope you avoid all of these issues in the future. Just the fact that you’re going to all this trouble to keep your readers informed shows that you care. Keep it up!

This is getting really annoying. 2 days in a row!

Your a fabulous company and site, keep up the hard work

While it’s understandably frustrating to have multiple days in a row server errors, I’m so glad such a great team is working on getting it resolved and communicating the problem to the users. The GetSat forums have been flooded with people asking about what’s going on, and I continue to point them towards the multiple 4SQ twitter accounts, as well as this blog and the new status page.

Issues like this are going to pop up when you have as massive amounts of growth as Foursquare has seen. Do we like the downtime? Not really. But is it ultimately worth it to see the site/service continue to grow and expand? Certainly. Keep up all the hard work. You are all awesome and we appreciate the continued effort everyone is putting in to get things running smoothly.

While it’s understandably frustrating to have multiple days in a row server errors, I’m so glad such a great team is working on getting it resolved and communicating the problem to the users. The GetSat forums have been flooded with people asking about what’s going on, and I continue to point them towards the multiple 4SQ twitter accounts, as well as this blog and the new status page.

Issues like this are going to pop up when you have as massive amounts of growth as Foursquare has seen. Do we like the downtime? Not really. But is it ultimately worth it to see the site/service continue to grow and expand? Certainly. Keep up all the hard work. You are all awesome and we appreciate the continued effort everyone is putting in to get things running smoothly.

I think you should compensate your users for the downtime. Perhaps an “I survived the outage x2″ badge? Just kiddin’. This kind of stuff sucks when you get bit by it. I’m in the business and I have a lot of sympathy for anyone who has to go through a prolonged outage.

best,

r

Um….. status.foursquare.com is down now too………

seems to be up for us

I think the amount of communications is just about right. While I’d like to have a better idea of “how long”, working in IT myself I realize that you’re far better off working to fix the problem correctly than you are trying to figure out how to give an accurate “when” estimate. I think the status updates (which for the most part were 60-90 minutes apart) were appropriate.

Good work guys – open communication = loyalty

lol, if it was unacceptable, then it wouldn’t have happened in the first place. Nice try though.

How are you partitioning shards such that you are seeing such a skew in assignment as to require manual rebalancing?

The downtime in a way is great for everyone involved… it just means Foursquare is experiencing the massive growth we have all been dreaming about. Foursquare is going mainstream, and as it continues to do so, outages like this are to be expected.

You guys did a great job of communicating the problem and keeping us updated yesterday. Communication was a little subpar on the first day, but you did an excellent job of making up for that yesterday. Keep up the great work!

Well, now that foursquare is aware of your annoyance, I’m sure they’ll try to fix the problem. Man, if it wasn’t for you, we may have had to deal with these outages forever! Thanks a million, mikee. We all owe you, big time!

I would definitely agree with David, you cannot put an estimated time on technical work. Even for me in my business I always go for the longer end of telling clients the most time I may be in something. That way if your done sooner you know you achieved a happy client and goal at the same time!

I like the twitter updates

I think you guys are doing a fabulous job of keeping us in the loop as things happen. Yeah, downtime sucks, but it gives us a chance to miss you. Absence makes the heart grow fonder, yes?

I think you should give all current users an honorary badge that states “I Survived The Great Foursquare Blackout of October 2010″.

Yeah, I definitely pay attention to your Twitter updates and RT them to people when I notice they’re wondering what’s going on, and also forward them to my mom, but the one time I checked the ‘status’ link, that was ‘down’, too. But I think you communicate very well and I echo the positive things others have said. Good job! Now get some sleep! : )

Can we hope to see some technical explanations about the problem with MongoDB sharding? I’m going to move from master-slave to sharded setup and this sharding story is freaking me out a bit :

That is a great idea. It would shed some bright light on an otherwise dark situation. It would make the whole thing something that all users could just laugh about in a few months time when comparing badges with their friends.

That is a great idea. It would shed some bright light on an otherwise dark situation. It would make the whole thing something that all users could just laugh about in a few months time when comparing badges with their friends.

How to be better? Just stop crashing. I hit 2 new places last night and couldn’t check in, Grrrr!!!

No, that’s a stupid idea, because it would be pandering to people who want an easy badge. “Survived?” It was a website outage, not an earthquake. If it matters that much to you, you need to get a life.

For a FREE service, 99.999% up-time is unrealistic. You guys are doing an awesome job with such little staff. Communication is key and your new methods for doing so are perfect.

It’s just a badge.

For those people wondering about the details of issues in Mongo, see this post on the Mongo mailing list: http://groups.google.com/group/mongodb-user/browse_thread/thread/66752f49af68619

In particular, they say:

“
- If your shard key has any correlation to insertion order, I think you are
ok.
- If you add new shards very early [such that] the source shard doesn’t have high
load, i think you are ok.
- If your objects are fairly large (say 4KB+), i think you are ok.
- If the above don’t hold, you will need the defrag enhancement which we
will do asap.
”

I’m not happy that all of my check-in history has been erased. For example at work, where’s I’ve checked in more than 28 times in a month.. now says I’ve checked in 3 times total.

NOT HAPPY.

So you’ve thought about it I’m sure, and I know when you’re starting out costs need to be capped. But what about re-architecting around some RDBMS i.e. Oracle for one. I’d love to see some statement about your actual volumes and transaction rates that would show it wouldn’t work – from what I can tell I reckon it would.

When your scale reaches 4M users and 200M checkins (at who knows what rate, but you gotta figure it’s a sharp curve), you should start worrying Before that, intelligent configuration of MongoDB will serve you well!

I do understand that
But having 58M items in one collection and couple of thousands added every minute forces me to think in advance… that’s why I’d like to know how to avoid this kind of problems, when intelligent configuration just won’t be enough.

It’s still there! We had a temporary glitch displaying some users’ histories after the downtime but we restored all check-ins. See here: http://foursquare.com/user/-522585

it would be fun … why make everything so serious?

You say everything is back up, but my days out and checks ins are still wrong. I am losing mayorships, and I’m very upset. Can someone please respond to my comments and let me know when you can fix my account!!!!!

My days out and check ins are still wrong!!!

For a subset of users, the count is not showing as correct. As soon as we make some more changes to ensure site stability (tonight), we’ll be restoring these. Don’t worry, they aren’t gone forever!

It’s more than just a badge. It would actually bring back some of the users lost to other geo-location services like Yelp, Gowalla and F*ceb**k.

I’d really like to see something in the app. I know there’s a generic “couldn’t connect to foursquare” message, but it’d be nice if it could pull a small bit of text from somewhere (status.foursquare.com, or Twitter) mentioning what’s going on. Enough to fit in the error message.

That way, when I’m checking in and it can’t connect, I know it’s not something I can fix. Sometimes I tend to try again in a minute, or walk around for better service, or reboot my iPhone – but if there’s no way I’m checking in, it would save some time to tell me up front.

Would cleaning up the database of all duplicate entries help solve these problems?

Hang in there guys! Growing pains. Your transparency and communication is great! Thank you.

Mine still haven’t been restored.

I am happy that now Foursquare is ok, but I have an annoying problem: since I installed the Android app foursquare_2010093000.apk, I cannot edit venues, even those venues I created: every time the error “A surprising new problem has occured. Please try again!” appears.
And from the desktop website editing is not possible.
There a lot of venues near me or where I go which need corrections, but it is not possible for me

all check-ins were restored this morning!

counts were restored as of this morning.

we agree, and this is something we’re working on!

so i dont know if this has anything to do with this post but several of my friends including me are having problems with out checkins not going to FB… whats the deal… it was working one day then the next – nothing… i am frustrated cuz i have done everything that help has told mem to do I even deleted my FS account losing everything and started over but the same thing is happening… also it shows that i am linked with FB but there is no name next to it but it pulls up my friends in FB …. kinda confused… please help!!

Still wondering why so many of us are not able to have our check-in’s show up on fb…..and why no one is answering the question as to WHY it is happening.

200 Millions check-in’s?? And how many are actually posting??????

I’ve got the same problem checkins not on FB/Twitter. I’ve searched the web, searched the forums, reviewed my settings a thousand times, mailed 4sq. but i get zero reply. I’m gonna delete the app and am done with it. ;-(

Foursquare est tout simplement génial, ce réseau social est sans doute un futur acteur majeur dans le monde du marketing.
Un bien jolie blog.

http://groupe4-supdepub.blogspot.com

Vincent rédacteur du blog Buzzmania

we made some changes last week that should fix this for you! see here to fix:

http://foursquare.zendesk.com/entries/302121-updates-to-facebook-connect

we made some changes that should fix this for you. see here: http://foursquare.zendesk.com/entries/302121-updates-to-facebook-connect

bonjour excusez moi je me suis inscris mais je me suis tromper de site j’était à regarder votre blog et au lieu de m’inscrire au site j’ai fait le contraire et je n’ai aucun iphone ni tel portable non plus désolez est-ce que l’administrateur peux supprimer mon compte svp
bien à vous
Cordialement

Just cruising through your blog and came upon this. What a great example of openness and communication with your users. I just got my foursquare Ambassador cards today and I’m hopeful I can add some venues to your growing base of Vendors who realize the value you offer to them. IMO, only a fool wouldn’t take advantage of your service.

Thank you for this topic

دردشة صوتية
شات كتابي
شات سعودي كول
دردشة سعودي كول
شات صوتي
دردشه صوتيه
سعودي كول
دردشة سعودي كول
شات سعودي كول
سعودي كول
دردشة صوتية
دردشة سعودي كول الصوتية
سعودي كول
سعودي كول 6666
سعوي انحراف
سعودي انحراف 2010
شات صوتي
دردشة صوتية

“A surprising error KEEPS occurring” and no amount of refreshes or reinstalls will fix it.

Hi it is a great article.We had a great time in reading your article. Thanks for sharing your information.

android developer
android developers

Pretty section! It feels great reading such post like this, very informative and interesting. I’ve learned. Here’s another source that worth a look about this also. Thanks for sharing this info!

Wow nice information you have shared here. Actually Google made searching of information easy on any topic. Well keep it up and post more interesting blogs.Thanks for sharing.  designer mens clothes

Great discounts on clinique product by using clinique bonus time today. Also get your cheap car insurance cgu car insurance to save your monthly expenses on auto insurance.

 
I must show my thanks to the writer just for rescuing me from such a crisis. As a result of looking throughout the world-wide-web and seeing thoughts which were not productive, I assumed my entire life was well over. Being alive devoid of the solutions to the difficulties you’ve resolved as a result of your entire site is a serious case, as well as the kind which could have in a negative way affected my entire career if I had not discovered the website. Your personal natural talent and kindness in dealing with almost everything was helpful. I’m not sure what I would have done if I hadn’t come upon such a point like this. I am able to at this point relish my future. Thanks so much for your high quality and effective help. I will not hesitate to propose your web site to any individual who needs to have counseling on this problem. pest control brisbane

 
I must show my thanks to the writer just for rescuing me from such a crisis. As a result of looking throughout the world-wide-web and seeing thoughts which were not productive, I assumed my entire life was well over. Being alive devoid of the solutions to the difficulties you’ve resolved as a result of your entire site is a serious case, as well as the kind which could have in a negative way affected my entire career if I had not discovered the website. Your personal natural talent and kindness in dealing with almost everything was helpful. I’m not sure what I would have done if I hadn’t come upon such a point like this. I am able to at this point relish my future. Thanks so much for your high quality and effective help. I will not hesitate to propose your web site to any individual who needs to have counseling on this problem. pest control brisbane

سعودي كول
سعودي كول 6666
سعوي انحراف
سعودي انحراف 2010

سعودي
كول       سعودي كول  6666    
 شات سعودي كول 6666       سعودي كول 24       سعودي
كول 94

    
    
    سعودي كول   
      
    سعودي كول  6666    
     شات
    سعودي كول 6666
         
    
    
    سعودي انحراف
          سعودي
    كول 94

tankio is my free free god  is your drink is my food pliesa call ok
lilke ?

سعودي انحراف

سعودي انحراف 2010

سعودي انحراف 6666

انحراف

سعودي كول

سعودي كول 6666

سعودي كول 94

شات سعودي كول الصوتي

سعودي انحراف

سعودي انحراف 2010

سعودي انحراف 6666

انحراف

tankio is my free free god  is your drink is my food pliesa call ok
lilke ?

سعودي كول

سعودي كول 6666

سعودي كول 94

شات سعودي كول الصوتي

سعودي انحراف

سعودي انحراف 2010

سعودي انحراف 6666

انحراف
سعودي انحراف

سعودي انحراف 2010

سعودي انحراف 6666

انحراف

tankio is my free free god  is your drink is my food pliesa call ok
lilke ?

سعودي كول

سعودي كول 6666

سعودي كول 94

شات سعودي كول الصوتي

سعودي انحراف

سعودي انحراف 2010

سعودي انحراف 6666

انحراف
سعودي انحراف

سعودي انحراف 2010

سعودي انحراف 6666

انحراف

I would definitely agree with David, you cannot put an estimated time on technical work. Even for me in my business I always go for the longer end of telling clients the most time I may be in something. That way if your done sooner you know you achieved a happy client and goal at the same time!

Wow nice information you have shared here. Actually Google made searching of information easy on any topic. Well keep it up and post more interesting blogs.Thanks for sharing.

This intimidation often stems from the rituals and Latin used during
portions of Catholic services, both of which are integral to the
religion but may be confusing to non-Catholics possibly interested in
the faith. Santa Cruz Real Estate

overloaded database? awww sorry to hear that. Good thing you managed to fix it.
Boston Market Coupons

True! I couldnt agree more
Outback Steakhouse coupons
Ruby Tuesday coupons

It was good that you were able to fix the problem at once. Good luck! I hope that you would not encounter any problems in the future. After all, I am a big fan of Foursquare!

Nifty Files

Having an area product is mostly a danger to promote it wider. This is because an area product is a novel product that cannot be obtainable in different place. This product shall be a pleasant souvenir. wedding flowers

I liked how the thoughts and the insights of this article is well put together and well-written. Hope to see more of this soon. http://www.hllyfmtransmitter.com/

I liked how the thoughts and the insights of this article is well put together and well-written. Hope to see more of this soon. http://www.hllyfmtransmitter.com/

A good blog always comes-up with new and exciting information and while
reading I have feel that this blog has really all those qualities that
qualify a blog to be a good one.

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts.Any way I’ll be subscribing to your feed and I hope you post again soon.

From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! Keep up the good work…Property Investment

The Most Helpful Restoration Service in America – Affordable Services in Your Area – 24/7 Hotline. water removal

I love your iphone app.
Clinique Bonus Time

Your iphone app is so user friendly.

Check in feature is quite popular in smartphones.

Keep in mind that there are many options for CC licenses that are much
less-restrictive than the version used at D.O.M. Visit the Creative
Commons website for an explanation of the different licenses. 

This blog has definitely changed my perspective on this subject. There’s
no way I would’ve thought about it this way if I hadn’t come across
your blog. All I was doing was cruising the web and I found your blog
and all of a sudden my views have changed. Good on you, man!

your downtim i acceptable. Dunkin Donuts Coupons

[link=http://www.googe.com]Google.com[/link]

Geat websit i love foursquare

 I’d made my mind up on this one – or that I had – after reading about
this in a specialist magazine. Now your writer has convinced me
otherwise … and in far fewer words! Thanks for sharing an excellent

Please tell me that you’re
going to keep this up! Its so good and so important. I cant wait to
read more from you. I just feel like you know so much and know how to
make people listen to what you have to say. This blog is just too cool
to be missed. Great stuff, really. Please, PLEASE keep it up! 

 Well that’s very Nice work! I would like to appreciate the great work done by the web master and would like to tell everyone that they should post their interesting comments and should make this blog interesting. Once again I would like to say keep it up to blog owner!!! i enjoyed to read this post.

Good informative post. I will visit your site often to keep updated.
Awesome!
Some really helpful information in there Bookmarked.

I am glad i found ur blog.Not everyone can provide information with proper flow. Good post. I am going to save the URL and will definitely visit again. Keep it up.

This is a great inspiring article. I am pretty much pleased with your good work. You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.

This is a fantastic website and I can not recommend you guys enough.
Full of useful resource and great layout very easy on the eyes. Please
do keep up this great work.

I’ll most definitely be bookmarked and shared with all my friends. I have
been looking for something like this for a long time and I can’t
believe I finally stumbled on it when looking for a basic code. Thanks
for putting together such a comprehensive site.

To tell you the teuth I was passing around and come across your site. It is wonderful. I mean as a content and design. I added you to my list and decided to spent the rest of the weekend browsing. Well don

The difference between the right word and the almost right word is more than just a fine line! it’s like the difference between a lightning bug and the lightning!

Thanks

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine. 

I admit I have not been on this blog in a long time however it was joy to find it again. It is such an important topic and ignored by so many even professionals! I thank you for helping to make people more aware of these issues. Just great stuff as per usual!

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

Flower delivery in dubai online,by dubai flowers,we send cake and gifts to dubai by same day flower delivery services in UAE.

The post is absolutely fantastic! Lots of great information and inspiration both of which we all need! Also like to admire the time and effort you put into your blog and detailed information you offer! I will bookmark your blog!

There are plenty of good points here, but I’m not sure that under real-time search being discarded. I agree that it is very important, but not the point of that to show what people is saying today about a topic. medium length hair styles

The difference between the right word and the almost right word is more than just a fine line! it’s like the difference between a lightning bug and the lightning!

Mongo Mailing list is interesting concept. Antonio

It seem like someone is posting these weired comments over and over again..

“The difference between the right word and the almost right word is more than just a fine line! it’s like the difference between a lightning bug and the lightning!

“

200 million!? that’s crazy

I just followed you on twitter!

 it’s like the difference between a lightning bug and the lightning!

Thank you
mister for me

Still it does some CDs on here 

All properties meet comfort, equipment and location criteria. Most villas find themselves close to beaches and shopping centers.  For pleasant holidays, trust Cote d’Azur villa rentals team to help you find your South of France holiday rental.

Wow! 200 million checks? That’s truly a huge number!

شات صوتي -
دردشة صوتية -
ahj w,jd 
- شات كتابي -
دردشه كتابية -
ahj –
يوتيوب -
utube 
- دليل مواقع -
منتديات نسائية 
- منتديات انسجام 
- العاب -
العاب بنات -
استضافة

This is very interesting, You are a very skilled blogger. I have joined your feed and look forward to seeking more of your excellent post. Also, I have shared your site in my social networks!

This blog Is very informative , I am really pleased to post my comment
on this blog . It helped me with ocean of knowledge so I really believe
this site will do much better in the future.

 A very nice page. I think the effort has passed, we have to thank you:))
Dis estetiği

 Very glad to see everything has turned to normal and congrats for the efforts to make the things alright again!

 Too much followers of Foursquare! That’s a point though…

And I’m wondering what this has to do with the blog at hand. Anyhow, congratulations OP

Great points really quite interesting..I visit often to have nice information.

Thanks for a very informative and useful post….

Keylogger
Zip Repair Tool
keylogger
Data Recovery

Your blog is
great, We thank you for providing this information!

I can’t even get 11 visits in one day

Thanks for this post!
It seems to be really a good idea.
It’s definitely more than just a badge;)
BTW in case you haven’t tried this crossword puzzle maker. A good way to train your memory!

Hey that’s the right way to celebrate.

Cheers from manu.

I was seeking a post like this. Really refreshing take on the information. Thanks a lot.

I always wondered why websites go down for long periods of time. Is this usual when my site is hosted elsewhere? ers911.net

Hi,
abundant Blog. The way you explained it is absolutely alarming and makes
anybody to apprehend till the end. Accumulate posting. 

Award winning advertising agencies