Podcast: Quad Core
Faster, Hotter, Scalier
Jill
David
Jason
Joyent Weblog
Public Betas Are a Sham
A little over a year ago, I met with Dave Young to talk about joining him at Joyent. I was intrigued from the start, but he hooked me with a 15-minute diatribe against the inanity of seemingly never-ending “public betas” for web sites. It was a rant that could have come out of my own mouth. In short, we both agreed that permanent “betas” are bullshit – as are any public betas that last more than a few weeks.
You’re either ready to ship, or you’re not. Slapping a “beta” badge on your logo doesn’t cover for any actual flaws or shortcomings in your software. It’s this decade’s equivalent of the 90’s “under construction” GIF animation.
This week, we officially launched our hosted Joyent Connector service, and, as promised, we did it without ever slapping a “beta” label on it. It’s far from perfect and our to-do feature list is still quite long, but real software ships. If it’s good enough for your customers to use, you shouldn’t need a bogus “beta” label to excuse its deficiencies.
One way we avoided a public beta was by soft-launching. We started taking money and turning on real accounts early in February, a couple of weeks before making any official announcements. We spread the word quietly to friends and to our loyal customers at TextDrive (most companies would kill for a customer base like TextDrive’s), people who, we hoped, would be somewhat forgiving of any unforeseen hiccups as we scaled the service.
There’s nothing wrong with real betas – controlled, scheduled periods of testing for unproven software still in development. For some “social” web apps that are expected to scale to many thousands of users, it might even make good sense to ship as a public beta – for a short period of time. Exhibit A regarding how to do this right: Mike Davidson’s NewsVine, which also shipped for real this week, after a vigorous eight-week public beta.
The difference is this: in a real beta, your users are testers, and they know that they’re testers, and there’s a light at the end of the tunnel where everyone – developers and testers both – knows that the testing period will be over. But if your users are customers and see themselves only as such, then you shouldn’t be taking their money for beta-quality software.
We took a little bit longer to ship our hosted Joyent Connector service than we originally planned (that laughter you hear is because of the “little bit” part), but when our original ship date came and the software wasn’t ready, we took more time to get it right, rather than give customers software that wasn’t ready but slathered with “beta” cream. If you go for perfect, you’ll never ship. But “good enough” is possible, and beta-quality software isn’t good enough.
On the other side of the coin are companies like Google and Flickr, who use the “beta” label for web apps that are renowned as some of the best-designed, best-engineered, most-useful, most-fun in the world. What exactly is “beta” about Gmail or Flickr?
Do the millions of people using these services consider themselves testers, or users?
Commenting is closed for this article.
http://writely.com has an interesting spin on the beta thing. While you’re logged in, there’s a link to a one-click survey where you can say whether or not you think writely is ready to come out of beta. They call it the “Beta Meter,” and they show you a running tally of how they’re doing, as measured by their own (self-selected) beta testers.
— Zach A. Thomas 1219 days ago #We are going to be in beta for quite a while near the end of march. The good part about the web is that you actually have the freedom to stay in beta as long as you wish. If you develop hardware or produce printed media you know how incredibly frustrating it is to finally see your finished product and then finding a small error, misprint or bug.
On the web, your product is never finished. On the web, every day is beta day. Long live the web, long live the beta!
— Boris Veldhuijzen van Zanten 1218 days ago #Boris: I think Gruber’s trying to say that what you’re describing is a bad thing. True, a perpetual beta is great for developers, but it’s terrible for customers since you’re basically asking them to pony up cash (or to look at ads) for something you don’t consider “done” enough to remove the beta label. ‘Beta’ is a way of ducking responsibility.
— David Demaree 1218 days ago #David: That might be what he means (and I understand) but it’s not what he writes (and others might understand). True: asking cash for something that isn’t finished isn’t cool. But my point would be that there is no ‘Finished’ on the web anymore. Therefore the term ‘Beta’ isn’t that bad after all.
— Boris Veldhuijzen van Zanten 1218 days ago #Boris: I totally disagree with you. It’s true, there is no “finished” on the web—there never was any such thing as a “finished” website. There is, however, such a thing as being “done” enough for the general public, which for me means a couple of things:
* Basic features are present and accounted for. The more ambitious the app, the longer this list will be, but “shipping” software isn’t missing anything really important. That’s not to say that it has everything it should or could have, just the minimum list of features needed for people to get into using it.
* Those features been tested and found to work consistently on current release versions of the Big Three browsers. (And you know what? A lack of Safari support needn’t be a deal-breaker, so long as it’s forthcoming. Us Mac users know where to find Firefox if your app is useful enough.)
* The site is open for business. Signups (if needed) are being taken, and invite codes are only required if the normal, intended function of the app requires them.
* The site scales so that whoever wants to use it can use it most of the time.
* There’s a help page, a contact form or just some resource to help new users get acclimated.
There is no such thing as ‘finished’ on the web, and there never was. Forgive me for saying so, but duh—we all already knew that. “Shipping” or “beta” are descriptors not of the state of the software (which will change constantly) but of the developers’ confidence that it’s ready for visitors. It’s a state of mind, not a technological fact. The technological fact is that the difference between a ‘beta’ site and a ‘shipping’ one is often invisible to the end user, except that calling something ‘beta’ sets the expectation that it might break and calling something ‘shipping’ or ‘version 1.0’ sets the expectation that it might break but not as much.
Shipping means a product has been tested and found to be generally reliable. Shipping means it’s supported by its maker to whatever extent they choose to support it. Shipping means accountability—not necessarily in a practical sense, but definitely in terms of reputation. Shipping means you can ask for the consumer’s confidence that your product is good.
Beta is the absence of those things. Beta is not something to be aspired to—it’s a temporary state that should last as briefly as possible. Of course websites evolve, of course there’s no such thing as finishing a website and having done with it. We all know that, and I’d assume your customers know that.
— David Demaree 1218 days ago #The web makes updating something easy, but it doesn’t remove a developer’s responsibility to finish their product to a level “good enough” to ship.
Then, like most other software, you release patches and updates after it’s gone to market.
Interestingly, in the industry I work there are two pre-release levels:
Alpha => feature complete, with bugs
Beta => feature complete, no bugs
Alpha is of a pre determined length (of course this can change but generally it’s been budgeted at a certain duration) and once you hit Beta you’re saying “This is it, it’s ready to ship”. You then have a whirlwind period of testing to ensure that’s the case, fix any last minute issues found (which is reasonably inevitable), then you ship it.
And you better have made it “good enough” because you can’t patch it after it’s released.
:)
— Ryan 1218 days ago #It seems to me that all this beta bashing is based on the assumption that people are not mature enough to make judgements on their own. (Does anyone believe people will continue to use (or pay for) some software which constantly sucks just because it has a beta label attached? Or that people will believe some webapp is great, because it says ‘never ever have been in beta’?) Language is a game, and the term beta has adopted various meanings – with a well defined state in the software development cycle being only one of them.
— Saurier 1218 days ago #I think this post is awesome… So I dugg it. I hope that is OK.
— Leo 1218 days ago #http://www.digg.com/software/Public_Betas_Are_a_Sham
Saurier: I can think of many, many products or apps which manage to suck without clinging to the beta label. No one is trying to say that apps should never be in beta—that would be dumb. The objection is against open-ended public betas—when you release something as beta, solicit customers to come use it and essentially treat it as a released product, then why can’t the “beta” product be version 1.0?
— David Demaree 1218 days ago #How do you propose people test their web apps with impossibly high numbers of people to simulate then?
— df0notfound 1218 days ago #Private beta. Invite them in, keep it quite, correct the problems, launch.
— David Young 1218 days ago #I always figured that GMail and Flickr were trying to be ironic in their use of “Beta.”
— EV 1217 days ago #I think that Flickr and Google use “beta” status in a very different way from most companies. For some companies, beta means incomplete or work in progress. For Google, it seems to be more an expression of the company’s philosophy: The product is feature complete and basicly bug free, but it isn’t perfect yet. To get it there, we’re going to open it up publicly, let millions of people use it, and then mine the data to analyze user behaviors. How do you think Gmail got a delete button?
— Jonathan Dobres 1215 days ago #I just wrote a post in the similar vain to this, titled Your Beta Better Be Brilliant
It’s more along the lines of not disappointing users with lacklustre beta’s but still the same jist.
— Blaze 1207 days ago #I think for a proper beta you need to have a formal (and easy) method for people to provide feedback on the application/product. Do you know how to provide feedback in live mail, google stuff? Even better is if you don’t close user tickets with no explanation like they did with the windows onecare beta
— Andy 1207 days ago #Google started this and most of their stuff remains in beta. Look at Gmail. How long does this need to be “beta”? Is that so in the event it breaks, they can say “hey it’s just in beta, you can’t complain”? It’s silly.
— Marquee 1180 days ago #I agree with Jonathan . In addition to the ‘Delete’ button, Google didn’t even have the e-mail chat feature that they do now. And even that is just one of the many new features since GMail was launched. It would seem like they’re constantly adding to their application, based on user behaviour — so why isn’t the ‘beta’ tag justified for GMail?
— Arun Kale 1129 days ago #I disagree with Arun. Look at long-lived applications such as Photoshop. Sure, they release a new version every year or two, but none of the versions are a “beta” version. That doesn’t stop them from improving the next release or even releasing a point release. Beta is for something new that needs to be tested.
— Matthew Smith 961 days ago #Um, isn’t that what version numbers are for?
— David Ramos 961 days ago #Companies that stick to betas for long periods of time are actually short changing themselves. I never actually knew that gmail had the new features that were mentioned. I thought it was still the same product that I tried years ago. Had there been a new version number, instead of the same beta label, I would have known something changed and at least went back to try out what’s new.
One of the smartest things Micro$oft ever did was to equate the year to the version. Heck, if I tried gmail 2004 and didn’t like it, maybe gmail 2007 will be better.
— Mike 961 days ago #A beta is a finished product that needs more testing to discover and fix bugs. A public beta is intended to test a wider variety of hardware, software and workflow options than in house testing could accomplish in a reasonable length of time and at reasonable cost.
If you want to add features put them into a new version number or have the class to admit that your product is still in the development phase. When it hits feature complete call it the alpha it really is.
— David Walker 961 days ago #