Comments on: Amazon’s S3 Web Service, our #1 cause of failure http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/ Wed, 07 Dec 2011 22:49:55 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: A Wikileaks Supporter http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-3614 A Wikileaks Supporter Sat, 11 Dec 2010 00:41:02 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-3614 This whole Wikileaks fiasco is pretty crazy. You should check out http://voteonwikileaks.com. It's a recently launched website that seems to be going viral. They got something like 50,000 visitors in the first 24 hours of launch. It's sort of like a crowdsourced collection of arguments against and for Wikileaks. As a blog owner, you'd probably find some of the opinions there a good read. This whole Wikileaks fiasco is pretty crazy. You should check out http://voteonwikileaks.com. It’s a recently launched website that seems to be going viral. They got something like 50,000 visitors in the first 24 hours of launch. It’s sort of like a crowdsourced collection of arguments against and for Wikileaks. As a blog owner, you’d probably find some of the opinions there a good read.

]]> By: myhotrevenue.com1 http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-3612 myhotrevenue.com1 Tue, 07 Dec 2010 22:07:29 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-3612 i tried blog.crowdflower.com in the past and i like it, just keep it up! i am sorry if i wrote in wrong section and please admins to move this to another place. i am find ways to make money on the net. what posibilities we have? have you ever tried survey sites? i t seems to be easy to make money. i found myhotrevenue dot com in my searches. IS this legit? anyone know? i tried blog.crowdflower.com in the past and i like it, just keep it up!

i am sorry if i wrote in wrong section and please admins to move this to another place.

i am find ways to make money on the net. what posibilities we have? have you ever tried survey sites? i
t seems to be easy to make money. i found myhotrevenue dot com in my searches. IS this legit? anyone know?

]]> By: Святослав Маминов http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-1429 Святослав Маминов Sat, 22 Aug 2009 22:03:21 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-1429 Интересная тема, продолжайте. Иногда нахожу ответы, которые получить самому просто реально не хватает времени. Благодарю! Интересная тема, продолжайте. Иногда нахожу ответы, которые получить самому просто реально не хватает времени. Благодарю!

]]> By: Артем http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-1418 Артем Tue, 18 Aug 2009 23:56:49 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-1418 Вот это да... Куча комментов и практически ни одного по теме :) Вот это да… Куча комментов и практически ни одного по теме :)

]]> By: codediva http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-399 codediva Fri, 25 Jul 2008 02:48:44 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-399 I understand your frustration, however, anytime you used hosted services (even those from a trusted provider) you should plan on implementing some redundancy. I know this can be costly for small orgs, but it could have minimized the impact of this occurance. I understand your frustration, however, anytime you used hosted services (even those from a trusted provider) you should plan on implementing some redundancy. I know this can be costly for small orgs, but it could have minimized the impact of this occurance.

]]> By: LusciousPainGeek http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-395 LusciousPainGeek Thu, 24 Jul 2008 08:48:32 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-395 Isn't this one of the risks you take in exchange for the convenience of using a centralized system like S3? Unfortunate, but bound to happen and something you need to look into when performing risk assessment. Isn’t this one of the risks you take in exchange for the convenience of using a centralized system like S3? Unfortunate, but bound to happen and something you need to look into when performing risk assessment.

]]> By: Ted Z http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-394 Ted Z Tue, 22 Jul 2008 19:15:35 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-394 Look into Akamai, they'll do this correctly (disclaimer: I've worked for them). Or implement their system (quickly deteriorating DNS endpoint lookups) yourself, it's not hard if your scope is limited. You can just have S3 be your primary and when it fails, switch DNS over to your backup. Look into Akamai, they’ll do this correctly (disclaimer: I’ve worked
for them). Or implement their system (quickly deteriorating DNS
endpoint lookups) yourself, it’s not hard if your scope is limited.
You can just have S3 be your primary and when it fails, switch DNS
over to your backup.

]]> By: Ryan Haney http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-393 Ryan Haney Tue, 22 Jul 2008 15:14:43 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-393 Have you considered using S3 as a backup provider with a local disk as cache? I can envision software that emulates a hard disk with a caching mechanism that copies to and from S3 in the background, as files are requested / uploaded. I do think though that you haven't load tested your own storage solution. I have a friend that works for Seagate and hard drives fail ALL THE TIME. It's just nature of the beast. Have you considered using S3 as a backup provider with a local disk as cache? I can envision software that emulates a hard disk with a caching mechanism that copies to and from S3 in the background, as files are requested / uploaded.

I do think though that you haven’t load tested your own storage solution. I have a friend that works for Seagate and hard drives fail ALL THE TIME. It’s just nature of the beast.

]]> By: Joe http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-392 Joe Tue, 22 Jul 2008 03:05:39 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-392 Get a grip !!! It's not like you're running a stock brokerage system on Wall St ! My experience is that it's not hosting computers that are the reliability issue, but the ISP who provides the connection. Try getting any level of service guarantee is next to impossible. When I build "five nines" systems for clients, we end up putting data centers in multiple cities using multiple carrier connections. It costs a heck of a lot more than what Amazon charge for S3. It's like you paid for a cheap car, so don't whinge about the rattles. Get a grip !!! It’s not like you’re running a stock brokerage system on Wall St !

My experience is that it’s not hosting computers that are the reliability issue, but the ISP who provides the connection. Try getting any level of service guarantee is next to impossible. When I build “five nines” systems for clients, we end up putting data centers in multiple cities using multiple carrier connections. It costs a heck of a lot more than what Amazon charge for S3. It’s like you paid for a cheap car, so don’t whinge about the rattles.

]]> By: Striddy http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-391 Striddy Tue, 22 Jul 2008 02:06:19 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-391 Should've been: Also, I don’t get the sense that FaceStat’s load is so heavy as to ... stress your other hosting service to the point of failure much. Should’ve been:
Also, I don’t get the sense that FaceStat’s load is so heavy as to … stress your other hosting service to the point of failure much.

]]> By: Striddy http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-390 Striddy Tue, 22 Jul 2008 02:04:56 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-390 I agree that this deep and long of an outage is not acceptable, even for a service with tempered service level. However, your "this months FaceStat downtimes by cause" chart is disingenuous and useless. Foremost because looking at the last weeks of data immediately after a huge outage skews the picture severely. Why not show us since launch? Or several other months' charts for comparison? Also, I don't get the sense that FaceStat's load is so heavy as to > It’s astonishing that serving content off our own boxes can be more reliable than serving content off of Amazon. Hypothetically, yes. Necessarily? No. Are you going to get much more reliable at serving content in the next few years? No. Is going Amazon S3? Probably. 99.99% is a primary design goal, and unless I'm misinformed, they've been hitting that most months. This is obviously a disaster for them, but be real. Or, you know, jump ship and be sure to also write posts about how unreliable or expensive self-hosting is when it eventually bites you in the ass or becomes too costly to sustain. I agree that this deep and long of an outage is not acceptable, even for a service with tempered service level.

However, your “this months FaceStat downtimes by cause” chart is disingenuous and useless. Foremost because looking at the last weeks of data immediately after a huge outage skews the picture severely. Why not show us since launch? Or several other months’ charts for comparison? Also, I don’t get the sense that FaceStat’s load is so heavy as to

> It’s astonishing that serving content off our own boxes can be more reliable than serving content off of Amazon.

Hypothetically, yes. Necessarily? No.

Are you going to get much more reliable at serving content in the next few years? No. Is going Amazon S3? Probably. 99.99% is a primary design goal, and unless I’m misinformed, they’ve been hitting that most months. This is obviously a disaster for them, but be real.

Or, you know, jump ship and be sure to also write posts about how unreliable or expensive self-hosting is when it eventually bites you in the ass or becomes too costly to sustain.

]]> By: Alex Ranger http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-389 Alex Ranger Tue, 22 Jul 2008 01:45:17 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-389 Yes. Did you read the Amazon SLA information before you started using their service? S3 going down is the fault of Amazon's engineering team. Your site going down is the site of yours. You signed up and used a service that gives you tiny rebates for downtime rather than true uptime guarantees and you got what you deserved. Quit whining. Yes. Did you read the Amazon SLA information before you started using their service? S3 going down is the fault of Amazon’s engineering team. Your site going down is the site of yours. You signed up and used a service that gives you tiny rebates for downtime rather than true uptime guarantees and you got what you deserved. Quit whining.

]]> By: Danny Thorpe http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-388 Danny Thorpe Mon, 21 Jul 2008 21:55:56 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-388 <strong>Amazon S3 Down for 7 Hours; S3 Clients Looking for Exit...</strong> Lukas Biewald lays bare his frustrations with Amazon’s S3 service, particularly after the recent S3 service outtage that left his FaceStat business offline for more than 7 hours recently.  Actually, Lukas has double posted on this issue - he has... Amazon S3 Down for 7 Hours; S3 Clients Looking for Exit…

Lukas Biewald lays bare his frustrations with Amazon’s S3 service, particularly after the recent S3 service outtage that left his FaceStat business offline for more than 7 hours recently.  Actually, Lukas has double posted on this issue – he has…

]]> By: Tim http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-386 Tim Mon, 21 Jul 2008 17:46:24 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-386 Guys, cloud computing is still what should be called beta. Massively parallel systems are difficult and this is going to take a while to get right. If you put all your data on one of these sites, or pin your business model to theirs, you are asking for trouble. Don't put mission critical applications on new, unproven technology with no failover plan and then start whining that you aren't getting what you wanted. It's new, it's cool, it's going to be the future, but right now it's still going through growing pains. Guys, cloud computing is still what should be called beta. Massively parallel systems are difficult and this is going to take a while to get right. If you put all your data on one of these sites, or pin your business model to theirs, you are asking for trouble. Don’t put mission critical applications on new, unproven technology with no failover plan and then start whining that you aren’t getting what you wanted. It’s new, it’s cool, it’s going to be the future, but right now it’s still going through growing pains.

]]> By: jeremy http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-385 jeremy Mon, 21 Jul 2008 16:41:49 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-385 This is something any online business deals with, it's always better to host it yourself but not really cost effective for a business that's just starting. As a web business owner currently experiencing inexplicable down time I feel your pain. If you're not responsible for getting it to work again it does give you an excuse to go outside for a little while. This is something any online business deals with, it’s always better to host it yourself but not really cost effective for a business that’s just starting. As a web business owner currently experiencing inexplicable down time I feel your pain.

If you’re not responsible for getting it to work again it does give you an excuse to go outside for a little while.

]]>