3 stories
·
0 followers

Never Blame the User

1 Share
At the time when you're taking courses in school, you have no way to predict what will become useful later in life. I took a lot of courses on advanced mathematics, but I haven't needed to solve a complex cosine derivative in decades. And that traveling salesman problem that was covered in multiple computer science classes? In the real world, most problems are either complicated or they're not; we usually don't have time to dwell on the worst-case scenarios and theoretical complexities. (If something looks extremely complicated, then you either brute-force it or you find some additional information to augment the data that makes the problem easier.)

One of the courses that I took back in college was on usability ("HCI" or "human-computer interaction"). Honestly, I took it because I needed a relaxing no-brainer class between vector calculus and denotational semantics (if you know one computer language, you know them all). I liked that the HCI course valued creativity rather that rote memorization.

For example, one of the homework assignments was to create a new metaphor for a music player. We're all used to a triangle meaning "play", a square meaning "stop", and a circle for "record". The assignment forced us to come up with something completely different -- where the user could just look at the icon and intuitively know what it does. My metaphor was based on a dog. "Play" was a dog wagging his tail. "Stop" was the dog sitting. "Pause" was him holding his paw out to shake hands. "Seek" was the dog sniffing the ground, and "rewind" was the dog running in the other direction. (I'm sure I still have the icons somewhere...) A user could just look at the picture and tell what the function did.

I took the course back when the Web came first appeared (1993-1994). That HCI course has influenced every user interface and web design that I have created since then. It's also where I adopted the motto "Never blame the user." If the user clicks on the wrong thing or cannot find a feature, then it's a design issue and not a user mistake. (It's also where I realized that every question in an "FAQ" is synonymous with a design failure.)

From One Mistake to Another


Over at FotoForensics, the volume of traffic has nearly doubled in the last week. Until the last few days of July, a typical load had 800-1,000 unique image uploads per day. But then something happened... the load shot up. On July 29th, the load went to nearly 2,000 unique uploads. July 30th was 3,141. July 31st was 2,586. In fact, July was lining up to be the 4th busiest month, but those last few days shot it to the busiest month on record.

I'm used to periodic, short spikes. That's usually when a new forum learns that the site exists, or maybe a front-page posting on Reddit mentions FotoForensics in the comments. Spikes usually peak after 2 days and last 3-4 total. Right now, the load hasn't dropped. It's been a week and I'm seeing the site level off at 1,500 to 2,000 unique uploads per day.

When there's a spike, I can usually look at the web logs and identify the source from the HTTP "referers" (the links that pointed to FotoForensics). Reddit, Facebook, UFO forums, conspiracy sites... they stand out as a large number of references from the same few web pages. However, this time there is no single source.

I was trying to find the cause... and then I realized... I did this to myself.

Web Page Uploads


FotoForensics permits people to upload content two different ways. You can either upload a file directly from your you computer, or you can submit a URL to an online image. This provides flexibility for users, but it also led to one of the early problems. While technical people know how to find the URL to a picture, most of the people on the web are not technical. The site quickly received lots of URLs to web pages (HTML) rather than to the pictures referenced by the HTML.

I don't blame the users for uploading HTML instead of images in the URL field -- that's a design flaw. My initial solution was to identify the most common sites and just grab the picture that I thought they wanted. For example, Imgur pages typically have one main image, but then a ton of little pictures. I assumed that the user wanted the big main image. This was a good first-effort, but it wasn't ideal.

There's a common programming library called "magic" that identifies files based on some simple rules. For example, every PDF document begins with "%PDF" and GIF images begin with "GIF". Magic has hundreds of rules for matching different file formats. My second parser used magic to identify the file type. If it looks like "text/html", then I would parse it for images. This solution listed all of the pictures found on the page and prompted the user to select the image to upload.

My second solution worked really well, but it occasionally failed to parse HTML files. Last month, I took a closer look at this problem. It turns out, sometimes magic is wrong. Specifically, the rules to match source code (C and C++) come before the rules for identifying HTML. If the HTML began with JavaScript (a language similar to C and C++), then magic would report "text/x-c" or "text/x-c++" (or sometimes text/x-php, text/x-ada, or other text format).

For my third revision to this problem, I rewrote the parser and am no longer dependent on magic. Now the HTML images are extracted every time. I released this solution on July 28th -- and that afternoon the system load began to increase.

As far as I can tell, fixing that small bug enabled a small percentage of people. Those people, in turn, told their friends. And their friends told friends. This created the significant rise in volume without any single forum referring users. It just goes to show: a little usability goes a long way.
Read the whole story
Share this story
Delete

How to Survive a Bear Attack: An Illustrated Guide

4 Shares

How to Distinguish a Grizzly (Brown) Bear Stay Prepared

Illustration by Ted Slampyak



Read the whole story
Share this story
Delete

Three Months to Scale NewsBlur

49 Comments and 86 Shares

At 4:16pm last Wednesday I got a short and to-the-point email from Nilay Patel at The Verge with only a link that started with the host “googlereader.blogspot.com”. The sudden spike in NewsBlur’s visitors immediately confirmed — Google was shutting down Reader.

Late night at the office

I had been preparing for a black swan event like this for the last four years since I began NewsBlur. With the deprecation of their social features a year ago I knew it was only a matter of time before Google stopped supporting Reader entirely. I did not expect it to come this soon.

As the Storify history of the Reader-o-calypse, NewsBlur suffered a number of hurdles with the onslaught of new subscribers.

A few of my challenges and solutions

I was able to handle the 1,500 users who were using the service everyday, but when 50,000 users hit an uncachable and resource intensive backend, unless you’ve done your homework and load tested the living crap out of your entire stack, there’s going to be trouble brewing. Here’s just a few of the immediate challenges I faced over the past four days:

  • My hosting provider, Reliable Hosting Services, was neither reliable, able to host my increasing demands, or a service I could count rely on. I switched to Digital Ocean and immediately got to writing new Fabric scripts so I could deploy a new app/task server by issuing a single command and having it serve requests automatically within 10 minutes of bootstrapping.
  • It didn’t take long to max out my Amazon Simple Email Service (SES) account’s quota of 10,000 emails a day. So a few hours into the melee I switched to Mailgun, which unfortunately resulted in emailing myself 250,000 error reports. If you tried to email me and couldn’t get through, it’s because 50,000 emails about lost database connections made their way ahead of you in line.
  • Eventually, I was just plain blacklisted on SES for sending too many emails.
  • Fortunately, when the PayPal fraud department called because of an unprecedented spike in payments, I was prepared.
  • HAProxy would Having HAProxy serve errors (site is down, maintenance, timeouts, etc) with a 200 OK status code instead of the proper 500 Exception status code because of a ridiculous undocumented requirement to include HTTP Headers at the top of the error template. When your webapp uses status codes to determine errors, you get extremely strange behavior when it loads utter crap into your DOM.
  • The inevitable file descriptor limits on Linux means that for every database connection you make, you use up one of the 1,024 file descriptors that are allocated to your process by default. Changing these limits is not only non-trivial, but they don’t tend to stick. This is responsible for bringing down Mongo, PostgreSQL, and the real-time Node servers, all at different times of the night.
  • The support queue is enormous and I’ve had to spend big chunks of my 16 hour days reassuring paying customers that eventually Stripe will forgive me and my unresponsive servers and will send the payment notification that is responsible for automatically upgrading their accounts to premium.

The sad extent of my St. Patrick’s Day

As a one-man-shop it has been humbling to receive the benefit of the doubt from many who have withheld their judgment despite the admittedly slow loadtimes and downtime NewsBlur experienced. Having the support of the amazing NewsBlur community is more than a guy could ask for. The tweets of encouragement, voting NewsBlur up on replacereader.com (If you haven’t yet, please tweet a vote for “#newsblur to #replacereader”), and the many positive comments and blog posts from people who have tried NewsBlur is great.

It has also been a dream come true to receive accolades from the many who are trying NewsBlur for the first time and loving it. Since the announcement, NewsBlur has welcomed 5,000 new premium subscribers and 60,000 new users (from 50,000 users originally).

NewsBlur users are intelligent, kind, and good looking!

The next three months

Over the next three months I’ll be working on:

For those of you who are still trying to decide where to go now that you’re a Reader refugee let me tell you a few of the unique things NewsBlur has to offer:

  1. Radical transparency. NewsBlur is totally open source and will remain that way.
  2. It still feels like RSS, just with a few more bells and whistles. NewsBlur provides actual list of posts, as opposed to the more curated magazine format of some of the other popular replacements. This clean interface makes it easy to see the stories you want. One innovation however is the four different view options you have. NewsBlur can show you the original site, feed, text or story view.
  3. It has training. NewsBlur hides stories you don’t want to read based on tags, keywords, authors, etc. It also highlights stories you want to read, based on the same criteria. This allows you to find the stories you care about, not just the stories that the hive Hive cares about. And best of all, NewsBlur will show you why stories are either highlighted or hidden by showing the criteria in green or red.
  4. NewsBlur has rebuilt the social community that Google had stripped out of Reader. Users can share stories through their Blurblog and discover new content by following friends’ Blurblogs. The People Have Spoken is the blurblog of popular stories.
  5. Because NewsBlur is entirely open-source, if you don’t want to pay you can host your own server. Instructions are on GitHub, where you can also find the source code for the NewsBlur iPhone + iPad app and Android app.
  6. Most importantly, NewsBlur is not entirely a free app. The immediate benefits of revenue have been very clear over the past few days. Not only are NewsBlur’s interests are aligned with its users, but as more users join NewsBlur, it makes more revenue that can be used to directly support the new users. Not convinced that paid is better than free? Read Pinboard’s Maciej Ceglowski’s essay Don’t Be a Free User.

Shiloh during better times. Your premium subscription goes to both server costs and feeding her

With NewsBlur’s native iOS app and Android app, you can read your news and share it with your friends anywhere. And with the coming improvements over the next three months, you bet NewsBlur will be the #1 choice for Google Reader refugees. survivors.

Join NewsBlur for $24/year and discover what RSS should have been.

Read the whole story
popular
2812 days ago
reply
Share this story
Delete
45 public comments
ksw
2801 days ago
reply
Excellent write-up. I'm growing more confident about my decision to use NewsBlur in place of Reader.
Manhattan
shanel
2800 days ago
This is what brought me here.
datavortex
2803 days ago
reply
Great tale from the DevOps battlefield
Stockbridge, GA
parisferra
2807 days ago
reply
Excellent stuff. I go straight for premium membership. Good products deserve support.
heliostatic
2808 days ago
reply
Best news reader around, and the best team building it. Plus, you always know what features are coming next if you check the various branches on github.
Williamstown, MA
sigvei
2809 days ago
reply
_Slik_ snakker du med kundene dine. Enmannsforetaket NewsBlur tjuedoblet brukermassen over natta.
Oslo
maistrack
2811 days ago
reply
Simply awesome!!
keri
2811 days ago
reply
might not be for everyone, but it's for me
chapel thrill, nc
htakeshi
2812 days ago
reply
Google Rederがサービス停止を突然、発表。試行錯誤しながら、私も環境移行をしています。
その後の展開は、ビジネスモデルのケーススタディとしても、考えさせられることが多いです。
andycwb
2812 days ago
reply
Excellent opportunity. Just signed up for the super-premium extra dog food option.
gienahghurab
2812 days ago
reply
Actually just testing my blurblog... *'.'*
BLueSS
2812 days ago
reply
I love NewsBlur, and this was a nice read.
KevinMMeredith
2812 days ago
reply
Nice summary of what happened
Victorville, CA 92392 USA
smadin
2812 days ago
reply
Some interesting detail on @NewsBlur's Readerpocalypse.
Boston
pastorwalters
2812 days ago
reply
Interesting story of NewsBlur's rapid growth after Google's Reader shut down announcement.
Two Rivers, WI
tfield
2812 days ago
reply
Post Mortem on Readerpocalypse glitches from Newsblur which I switched to from Google Reader a while back. Great service.
Alexandria, VA
robferrer
2812 days ago
reply
Really interesting story of a sudden spike in demand for @newsblur
Leamington Spa, UK
to7
2812 days ago
reply
Newsblur news:
indra
2812 days ago
reply
fun times :)
Canberra
lpmpessoal2
2812 days ago
reply
Como a vida está a mudar ao NewsBlur com o fim do Google Reader. História interessante. Na primeira pessoa.
TheRomit
2812 days ago
reply
I hope that doesn't mean those API end points requested by the W8 developer are going to take a back seat :-(
santa clara, CA
superiphi
2812 days ago
reply
love the transparency at #newsblur :)
Idle, Bradford, United Kingdom
CliffS
2812 days ago
reply
Thanks for the update Samuel! This is why I prefer to support smaller operations--my input actually counts for something.
Wahiawa, Hawaii
trepidity
2812 days ago
reply
Excellent story
adamgurri
2812 days ago
reply
Notes from the front
New York, NY
agonist
2812 days ago
reply
Trial by fire is always an exciting challenge. I paid up and like what I see so far. Good luck to you.
the dark side of the sun.
mithrandir
2812 days ago
reply
I'm super excited to see NewsBlur take off like this. Congrats!
sredfern
2812 days ago
reply
Congrats Sam! (awesome name btw)
Sydney Australia
danatnr
2812 days ago
reply
Fascinating story of NewsBlur programmer whose demand increased tenfold when Google Reader announced closing.
Ohio
nickoneill
2812 days ago
reply
5 days and @samuelclay already has time to write blog posts, amazing.
san francisco
smilerz
2812 days ago
reply
Kinda wished I had found NewsBlur before becoming a Reader refugee. Great work and hang in there - all of the hard work will pay off!
Chicago or thereabouts
fredw
2813 days ago
reply
Great read on scaling exploding web apps. Been there, done that.
Portland, OR
pberry
2813 days ago
reply
Love the run down and appreciate the time it took to update us.
Chico, CA
samuel
2813 days ago
Ok, I'm going to admit it right now. My girlfriend wrote a good chunk of the post as a template, which I then rewrote with specifics. Seriously, she's the absolute best.
jbloom
2813 days ago
Hire your girlfriend to do PR. She is awesome! :D
jbloom
2813 days ago
reply
Love this detailed account of NewsBlur after the Google Reader closing announcement. NewsBlur rocks!
Columbus, Ohio
jbloom
2813 days ago
Going to get my dad a premium account.
DracoLlasa
2813 days ago
reply
been very impressed with the service and the ramp up. only took a few days and things are running pretty smooth. Glad i made the move here
mgeraci
2813 days ago
reply
Congrats, Sam, and keep up the amazing work!
New York, NY
tedder
2813 days ago
reply
all hail Samuel Clay! Great work. I reached out in case I could help in devops, but I didn't know the extent of your personal inbox disaster.
Uranus
samuel
2813 days ago
Don't know if you noticed, but I accidentally sent myself 250,000 error emails, so yeah, my inbox is currently jammed.
tedder
2813 days ago
yes- that's why I said "personal inbox disaster" :) spam cannons are always fun.
jzsimon
2813 days ago
reply
Great story!
blueminder
2813 days ago
reply
"Paypal’s fraud department just called, asked me what’s going on. Asked the rep from Omaha if she’s heard of Reader, and then a big Ohhh."
San Francisco, CA
pyrona
2809 days ago
Heeeey, when you share something you should always comment something so we can reply to you! Newsblur has a, uh, "feature" that makes it so others can't comment on your shares unless you do first. (I'm only telling you this here, because it's the most recent thing you shared that you commented on ^_^)
blueminder
2809 days ago
Ah, I didn't know that's how it works on this end of the RSS world. http://sighroll.com/wp-content/uploads/2012/10/themoreyouknow.jpg
SuzeVich
2809 days ago
BTW: You TOTALLY got me with Shiloh ... I would have paid even if you hadn't stopped the free subscriptions. ;D
iross
2813 days ago
reply
Holy crap. Am I reading that right? NewBlur went from 1,500 users to 60,000+?

Any chance those shirts make a comeback? I'd buy some NB swag.
Madison, Wi
CliffS
2812 days ago
I think it's 50k to 110k.
kyleniemeyer
2813 days ago
reply
Hey, I'm in this post! Perhaps not my most flattering picture...
Corvallis, OR
kyleniemeyer
2813 days ago
Also, I think that NewsBlur has pretty much recovered from the Great Reader Exodus.
samuel
2813 days ago
Nope, not even close to recovered. While the servers are stable, the load times are now 9X where they should be. I have charted some remedies, but they're going to take time. And I still have yet to go 24 hours without a server meltdown.
kyleniemeyer
2813 days ago
Yeah, I did notice the outage this morning. At least, things seemed better today than they had all last week.
jbloom
2813 days ago
Samuel, at least I am able to use my iPad now. And I am glad to see from all of this that you know your stuff with databases and such. You know WAY more than I do! :D
satadru
2813 days ago
reply
Excellent work.
New York, NY
BiG_E_DuB
2813 days ago
reply
This guy is awesome. Premium is basically confirmed for me...might do it Monday
Charlotte, NC, USA
samuel
2813 days ago
reply
Here it is, my last five days in blog form. Biggest lesson learned: take more photos!
Cambridge, Massachusetts
gazuga
2813 days ago
Epic post from a prescient fellow. It's plain you've been mulling over the question of free users, with good cause. For my counterpoint to Maciej Ceglowski, see "Transitioning back to freemium" in the ideas section of the support forums.
gazuga
2813 days ago
Loving my t-shirt, by the way!
ChrisHamby
2813 days ago
Thanks for all your hard work! I didn't send in a photo of my tshirt because it's a few sizes too big... but that's the last thing I'd want you to worry about now!
sashae
2812 days ago
thanks for working so hard to keep all of us poor news junkies happy... you've built a really great thing!
InKunKa
2812 days ago
I'm thrilled with Newsblur, and happy to be a new paying member. I wish I had known about it before :)
mocker
2812 days ago
Keep up with the transparency of what's going on with any site issues. As someone who already went premium, I appreciate it.
sredfern
2812 days ago
Your dog may gain some weight with all those premium subscribers...
trekkie
2812 days ago
Wore my shirt all day but forgot to take a picture.
2812 days ago
You won me over, I completely ditched google reader and deleted all of the data over there. I need some confirmation on if the price will go back to $12 and when exactly it will do this, I understand why you did it but I was there when it changed and that obviously made me do a double take. I NEED the features of the premium account so please let me know if I should wait or if the price will never go back down.
mfitzhugh
2812 days ago
You're doing a great job, Samuel. When I checked out Newsblur last week, your dedication to building a great service was evident from the get-go. It made signing up for premium an easy choice. Keep up the good work, and don't worry about breaking a few things along the way.
PunkRockWarlord
2812 days ago
I joined a few days ago and have been very impressed. I'd been wanting an RSS client that would sync between my laptop and iOS devices (and wasn't tied to Google), and NewsBlur delivers on all fronts. Thanks for putting in all the work, Samuel!
pdp68
2812 days ago
Speaking as a Reader refugee, I have been very impressed with NewsBlur so far - so much so that I am starting to wish that Google had shut down Reader sooner. In many ways, NewsBlur strikes me as being everything that Reader could have been.
kworr
2812 days ago
Maybe you need some help? I have some skills on nginx/freebsd/pgsql and making sites go stable on the high load (not just extremely high, like 2kk request per day).
raorn
2812 days ago
You did a great job, Samuel! dev.newsblur.com is something I always wanted. Already fell in love with "intelligent training" feature.
sylvaingaffie
2812 days ago
Man, you are AWESOME !! I still wonder how I would react in your case. I probably would have sank into madess :o).
stavrosg
2812 days ago
I live in Europe, so I woke up one morning with NewsBlur already in meltdown mode, and found about Reader a while later, so I just decided to sit out the mayhem and return to my feeds a couple of days later. You handled the situation as best as you could, congratulations and be assured that my premium subscription will be renewed once it ends. I am also glad that I quit Reader away ahead of the announcement of its shutdown and found this service.
nebkor
2812 days ago
As someone said on HackerNews, your first hire should be a good sysadmin; making changes to ulimits is easy and easy to make stick :)
cmn
2811 days ago
Here's 24$, love it!
Alpha_Cluster
2813 days ago
reply
Interesting post about the the aftermath of the Google Reader announcement on the awesome NewsBlur service!
DaftDoki
2813 days ago
reply
What one person has built is impressive. Success disasters are always better to deal with than normal disasters.
Seattle
ocrammarco
2812 days ago
Agreed, as a developer I have nothing but admiration and sympathy.
katiegirl
2812 days ago
This is exactly why I have given this guy my $24. He seems pretty up and up.