Notes from SCALE10x — More I/O for the Cloud, Please

A crew of Joyeurs went to Los Angeles to attend SCALE 10x – the 2012 Southern California Linux Expo over the weekend.

We went for a number of reasons. First, we were invited. Two of our engineers, Brendan Gregg and Robert Mustacchi, were on the speaker schedule. Second, we have been eagerly working with other members of the illumos community to grow the base of developers building applications and tools for this descendant of OpenSolaris (you probably already know by now that Joyent maintains and uses extensively SmartOS, an illumos distribution). A few people asked what we were doing at a Linux conference but after we explained that illumos / SmartOS are completely open sourced, they were happy to learn more. And we got some great feedback from people who are interested in developing on illumos that will help guide our community building.

Read More »

2012: The Year the Cloud has to Grow Up


Virtual Strategy Magazine asked a number of players in the cloud computing industry for their predictions for 2012. Our CTO and Co-founder Jason Hoffman offered his thoughts on what Joyent is looking forward to in the coming year.

In the cloud, we are rapidly moving from providing development environments to running critical applications in production. This shift means application must run seamlessly and predictably well.

So what does it then mean to run applications “predictably well?” And how do we augment and improve running application “well” through time?

Read More »

Photo by Flickr user Creativity103.

Events: Joyent CTO Jason Hoffman Speaking at MongoDB Los Angeles

A quick note about MongoDB Los Angeles, a conference put on in LA by our partners at 10gen (the initiator, contributor and continual sponsor of  the non-relational open source database MongoDB) on January 19th. Our esteemed CTO, Jason Hoffman, is going to be presenting a talk “N2M: Node.js and MongoDB as the modern stack for the real-time web.” Here’s a quick synopsis:

The combination of node.js and mongodb has emerged as the framework of choice for people building real-time applications, and is on the path to becoming the replacement for the common LAMP stack. This talk will suggest some reasons why and discuss some of the observed patterns in these types of applications.

Joyent Cloud General Manager Steve Tuck will be down there manning the booth and can answer any technology questions you might have. The MongoDB Los Angeles event has proven so popular that 10gen had to lift the previous attendance limit to add capacity.

Read More »

Where Does Your Node Program Spend Its Time?

Originally posted at Dave Pacheco’s Blog.

Performance analysis is one of the most difficult challenges in building production software. If a slow application isn’t spending much time on CPU, it could be waiting on filesystem (disk) I/O, network traffic, garbage collection, or many other things. We built the Cloud Analytics tool to help administrators and developers quickly identify these sources of latency in production software. When the app is actually spending its time on CPU, the next step is figuring out what it’s doing: what functions it’s executing and who’s calling them.

Read More »

A Bit More About the New Joyent Cloud API

As part of SmartDataCenter 6.5, we’ve introduced something I’m pretty excited
about sharing, which is an open signing protocol for HTTP. We’re using it in
what we call CloudAPI, which is the REST API that powers our customer portal, and is
exposed to you to create your own applications. In this post I’ll go over what the signing
protocol is and why you should care, how SmartDataCenter is using it, and some
ideas we have in mind for the future.

If you’re not an RFC nerd, HTTP defines a “pluggable” framework for
authentication so that any number of authentication mechanisms can be used, and
goes on to define Basic Authentication. Basic Authentication is nothing more
than sending your username and password across the wire, so it’s not ideal from
a security perspective. Following Basic Authentication, HTTP also goes on to define
Digest Authentication, which involves sending a hashed form of your password
across the wire, but it’s hard to use and doesn’t see much use these days.

However, for the best security properties, you really want to be leveraging
public key cryptography to authenticate. HTTPS (well, really SSL) has a mode
of allowing this called “client-auth”; basically your client has a certificate
and private key, and in addition to validating the server’s certificate, the
server asks you (the client) to present your certificate. Sounds great, and
does do what we want from a security perspective. However, it’s plagued by poor
usability in browsers, is more or less a programming nightmare (especially with
“custom” CAs), and can incur significant server-side cost, since it’s usually
a load balancer (i.e., HTTPS termination point) that is handling the
cryptography validation. All of this means that we want the benefits of public key crypto
without the burden of client-auth SSL.

To solve this problem with SmartDataCenter, we’re introducing the HTTP Signature
authentication scheme, which is already released as open source. The HTTP
Signature scheme allows for digital signatures to be leveraged as your
credentials, and does not imply any particular key management scheme. It should
be suitable for just about any REST API out there, so we’re certainly hoping if
you’re an API vendor you take a look at it, but rather than jump into the
technical details of the spec itself, I thought it would be interesting to
show you some CLI candy, so lets walk through some examples.

Read More »

Node on Azure and SDK Release

Earlier this week you may have seen word from Corporate Vice President Scott Guthrie or the Interoperability Team at Microsoft, but today the Azure team released their announcement regarding Node on Windows Azure and the Windows Azure SDK for Node.js. After many months of hard work between the Azure team and the core team here at Joyent on the port of Node and npm, we’re seeing Node’s first appearance in Microsoft’s products. For Joyent and the Node community, this is yet another strong statement regarding the maturity and enterprise-readiness that Node has achieved. Providing a first-class experience for Node developers on Windows is like strapping space shuttle solid rocket boosters to a technology that was already shattering airspeed records for community growth and enthusiasm. You will have the opportunity to hear more about Node from Microsoft’s perspective from Scott Guthrie himself in late January at the Node Summit.

Ryan Dahl, Engineer at Joyent and creator of Node, also chimes in on the importance of this achievement.

Scaling WordPress on Joyent Cloud: Part Three

The following is a continuation of our series on scaling WordPress and a repost of Peter Yorke’s original at peteryorke.net. Peter is a Solution Architect at Joyent with a passion for performance and scalability — he’s the man who knows how to make websites go to “11.”

You have hit the big time with your WordPress blog, GigaOM and Nikki Finke think you are the most influential blogger on the planet.

Congratulations, your mysql database is about ready to crash or worse, it already has.

In this post I discuss what you need to do with your database to keep this beast running smoothly and your blogging empire humming along.

Besides your users reporting the blog is slow, what objective measures are there? I use the NewRelic Std feature that lets me see database latency. Here an example of a database that needs help.

Read More »

Presenting File System Latency

Part 5 of 5 on Examining File System Latency in Production, by Brendan Gregg, Lead Performance Engineer at Joyent and author with Jim Mauro of “DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD

Previously I have explained why disk I/O metrics may not reflect application performance, and how some file system issues may be invisible at the disk I/O level. I then showed how to resolve this by measuring file system latency at the application level using MySQL as an example, and measured latency from other levels of the operating system stack to pinpoint the origin.

The main tool I’ve used so far is DTrace, which is great for prototyping new metrics in production. In this post, I’ll show what this can mean beyond DTrace, establishing file system latency as a primary metric for system administrators and application developers. I’ll start by discussing the history for this metric on the Solaris operating system, how it’s being used at the moment, and what’s coming next. The image below provides a hint:

Read More »

A 2000x Performance Win with DTrace Analytics

Note: This post originally appeared on the DTrace blog of @brendangregg

I recently helped analyze a performance issue in an unexpected but common place, where the fix improved performance of a task by around 2000x (two thousand times faster). As this is short, interesting and useful, I’ve reproduced it here in a lab environment to share details and screenshots.

Issue

In a production SmartOS cloud environment, a script is used to count entries in an active log. This script is executed from cron(1M), and the grep(1) command it uses is running very slowly indeed:

# time grep done log | wc -l
1492751

real    8m56.062s
user    8m12.275s
sys     0m0.218s
# ls -l log
-rw-r--r-- 1 root root 18M 2011-12-08 21:59 log

That took almost nine minutes. The grep(1) command is processing this file at about 34 Kbytes/sec, which seems awfully slow. Dropping the “wc -l” and using “grep -c” instead didn’t make much difference.

Hypothesis

I have a method for attacking performance issues (which I’ll explain in a later article); it starts with checking for errors, because they do cause performance issues and are a fast area to check.

The slow time sounds like it could be disk I/O related: either slow reads caused by disk errors, or random disk reads. I first checked the error counts:

# iostat -En | grep Hard
sd0              Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
sd1              Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Which were fine, and then for random reads:

# iostat -xnz 1
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.2    0.3    1.3    1.0  0.0  0.0    0.0    0.0   0   0 ramdisk1
   12.3   81.6  355.4 3005.9  0.0  0.1    0.0    0.7   0   2 sd1
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    7.0    0.0  245.7  0.0  0.0    0.0    0.1   0   0 sd1
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    4.0    0.0   59.5  0.0  0.0    0.0    0.1   0   0 sd1
[...]

Which are not present. This doesn’t look disk related.

Read More »

A Brand New MongoDB Appliance with Pre-Installed, Tuned Up Mongo Goodness

We’re down in Santa Clara for the annual MongoSV show and simultaneously announcing the launch of our brand new SmartMachine for MongoDB. This new virtual machine comes pre-loaded with a fully tuned, optimized, and ready to deploy instance of MongoDB, the NoSQL database that is becoming a key player in mobile, e-commerce, game and other responsive, low-latency applications. The growth of the MongoDB community has been extremely rapid as more people have come to embrace NoSQL data stores for modern Web application and DIRTY (data-intensive, real-time) uses.

In discussions we also heard many devs talk about how MongoDB and Node.js (the application environment of which Joyent is a corporate steward) play so well together with javascript, JSON, and other nice matches that make mating the two ridiculously easy to pull off. MongoDB running on top of Joyent’s SmartOS SmartMachine brings a lot of additional goodies including:

  • On-the-fly resizing of the database with no reboot or service interruption
  • Bursting of up to 800% at not extra charge to keep the database responding smoothly, even under extreme loads
  • DTrace-driven analytics for mapping latency outliers in four-dimensional heat maps rather than fighting through logs or staring at graphs or charts that hide the bad actors

You can get your feet wet with MongoDB+Node.js with this simple blog by our friends at MongoLab explaining how to set up a basic Node.s Web server running on a MongoLabs database as a service. Also we have some details of putting both MongoDB and Node.js on a Joyent SmartMachine virtual server.  MongoHQ, like MongoLab, is another viable PaaS that allows you to play around with MongoDB in a simple to manage software-as-a-service environment.

Most importantly, we’re interested in what the Joyent Cloud community is going to do with this new SmartMachine purpose-built for MongoDB apps. Please share and let us know what you think and what improvements can be made. Thanks!
Follow

Get every new post delivered to your Inbox.

Join 49 other followers