Aug 10

Easy Twitter Streaming API Testing with Mockingbird

My company makes two applications that focus on Twitter: CheapTweet and TweetReach. Lately, we’ve started transitioning parts of these apps to the Twitter Streaming API. To do that, we’ve been developing the flamingo project to encapsulate all the best practices of working with Twitter streams so that others won’t have to solve all the same problems. I’m not going to talk about flamingo today, though (I’ll do that soon). I’m going to talk about testing.

Testing code that talks to the Twitter Streaming API is hard. The are 8 documented failure HTTP response codes, random disconnections, and second-to-second rate changes. And, since this is Twitter we’re talking about, basically anything can happen at any time – including complete downtime.

So, let me introduce you to mockingbird. Mockingbird is a simple streaming HTTP server written in Ruby that you can use in place of the server at stream.twitter.com when you’re testing your Streaming API code. With mockingbird, all it takes is a few lines of code and you can simulate all the issues I described above and more. Here’s an example config:

Mockingbird.setup(:port=>8080) do
  wait 1
  5.times do
    send '{"foo":"bar"}'
    wait { rand }

So what does this do?

  1. Starts a mockingbird server on port 8080 on your localhost
  2. When there’s a connection, it starts by waiting for 1 second before sending any data
  3. Then, it sends ‘{“foo”:”bar”}’ 5 times with each time followed by randoom wait of between 0 and 1 seconds
  4. After that, it closes the connection and waits for more

You can do lots of other things with this simple configuration style. See the README and code examples at github.

Mockingbird provides a simple setup/teardown mechanism so it’s easy to start a server and shut it down during your unit tests, specs, or whatever. Here’s an example Test::Unit test:

class TestConnection < Test::Unit::TestCase
  def setup
    # Starts the server running a separate process
    Mockingbird.setup(:port=>8080) do
      status 500, "Error"

  def test_handle_500_errors
    # Your test code here for connecting to localhost:8080
    # and handling a 500 error here

  def teardown
    # Stops the server process

We’ve already found mockingbird to be extremely useful for testing flamingo. If you’re using the Streaming API and need to be confident your code can withstand the crazy stuff Twitter might throw at it, I think you’ll probably find it helpful, too. Of course if you have questions, comments, ideas or (even better) pull requests, hit me up.

Jul 10

ActiveSupport Considered Harmful

This an open letter to my fellow ruby gem authors and maintainers.

Hi Everybody,

It’s been a while. You’re doing awesome stuff; all this great code out there making Ruby do all sorts of nifty things. It’s a big reason – maybe the reason – why being a Rubyist is so great. But yeah, there’s just one thing that’s been bugging me… we need to talk about ActiveSupport.

ActiveSupport is great, don’t get me wrong. I love try and tap and sum as much as the next guy. It sure saves me some lines of code. But the problem is it’s, well, a little invasive. Many of us love Ruby precisely because it’s more flexible than a Russian gymnast. But just because you can reopen Object and add

class Object
  def your_great_method

doesn’t mean you always should. As we all know, ActiveSupport is one giant ball of I’M CHANGING OBJECT (and Enumerable and Array and String and, well, you get the picture). This is a pretty awesome idea in a controlled environment like Rails. Rails is all-encompassing. When you write Rails, you’re not really in Ruby anymore, you’re 3 layers deep, Inception style, inside DHH’s dream of web programming. In that world, Object can do magic things and Strings can inflect right back on themselves. And so, of course, ActiveSupport is an integral part of Rails (more on that in a moment).

The problem comes when you decide you just can’t  live without tap or a few of these other modern conveniences and so you throw in the ol’ ActiveSupport dependency in the gemspec. The first effect of this is that whatever project your users are working on now much bring in this giant ball o’ Ruby changin’ with its various invasions, idiosyncrasies and occasional incompatibilities.

But, you say, most people are using Ruby for Rails and Rails has ActiveSupport baked right in. We’re good right? Well, no, actually. This is where it gets really painful. When you’re gemspecin’ and you decide to throw in that latest version of ActiveSupport, well, you’re basically telling every Rails user out there that’s not on the latest version that they need to upgrade their entire environment just to use your gem. Now, the Ruby community is made up of people who are pretty early adopters so many people are on the latest and won’t even notice. However, as Ruby gets more mature and lots of people have lots of versions of things out there, not everyone can just upgrade at the drop of a hat. You’re cutting off those people from your gem awesomeness unless they either a) upgrade (which isn’t always immediately possible) or b) resort to horrifying monkey patches. And all this because you just love sum so much.

So, my fellow gem authors, I ask you to think twice next time you really want to use that convenience method that’s only in ActiveSupport N.N.N. Perhaps a little helper method would do? Maybe one that’s in your gem’s own namespace perhaps? Your users will thank you.



Mar 10

Using MongoDB and a Relational Database Together

Below is the presentation I gave to the Austin MongoDB Day on 3/27/2010. I talked about the design decisions one needs to make when determining whether to introduce a document database like MongoDB into an existing application built on a relational database. I give a few examples of how we’ve made MongoDB and our MySQL database live together in harmony inside CheapTweet.

Thanks to Lynn Bender of GeekAustin, CoSpace, and the folks at 10Gen for putting on a great event.

Jul 09

Introduction to Sphinx and Thinking Sphinx: 10 Minute Talk at Austin on Rails

I did a quick talk at Austin on Rails last night about my experience using Sphinx and Thinking Sphinx for doing full-text search on CheapTweet. We did a series of short talks last night so this is only a 10 minute overview. Hope some of you will find it helpful.

Mar 09

Austin on Rails Talk: Building TweetReach with Sinatra, Tokyo Cabinet and Grackle

I gave a talk last night (3/24/2009) at Austin on Rails about building a new (yet-to-be-released) application called TweetReach using tools that are somewhat off the beaten path. These included Sinatra, Tokyo Cabinet and my new Twitter API library: Grackle (which I’ll talk about in more detail in a future blog post).

I referenced some code during the “Building TweetReach” part of the presentation. If you’re interested, you can download it.

By the way, I was the second speaker of the night. Mike Perham spoke before me about caching with Rails. Check his stuff out as well. Thanks to everyone who came out.

Update 4/2/2009: Launched TweetReach! Go check it out.

Sep 08

A recent Rubyist goes to the Lone Star Ruby Conference

I attended the Lone Star Ruby Conference (LSRC), held here in Austin over the past two days. As this was my first conference on the topic, I thought I would share some observations. First, a vigorous caveat emptor… I’m a former Java guy still relatively new to the Ruby world. At the very least I hope to provide the humble perspective of a transplant from the crowded streets of Javaopolis to the rolling hills of Ruby country. So here goes…

Rubyists aren’t afraid of change (and some churn)

I’ve long been an interested outsider to Ruby and Rails – following along on the fringes for a couple years now by reading articles, running irb occasionally and generating some scaffolds. At that distance, you basically see Ruby and Rails. What you don’t see is the constant churn of open source projects, plugins, development tools, GUI toolkits, ideologies and best practices. Gregg Pollack and Jason Seifer’s talk on innovations in the last year and Evan Phoenix‘s keynote about Ruby memes made the currents of change pretty clear. Java experienced a lot of change during my 8 years but it definitely seemed to be at a more staid pace. There were a few bigger, more corporate, entities driving it (Sun, Apache, IBM) and fewer mythologized personalities like why. In the Ruby community, a lot of energy seems to be expended reinventing various wheels (as Evan pointed out with his list of ARGV processors). There’s a fine distinction between a healthy competition of ideas and a naval-gazing, ego-stroking churn that wastes time. That said, I would never stand in the way of the free market of ideas. So, by all means, (with no apologies to Mao) let 1000 ARGV processors bloom and let’s see which stick.

There are some surprising technology holes – but they’re being filled

As a former Java programmer who worked deeply with concurrency, Ruby’s after-thought of a concurrency model frightened me a bit. Java, though described by Dave Thomas on the last night’s panel as “a blunt tool”, has a basement filled with extremely sharp swords known as threads and a well-defined concurrency model. But, you say, Ruby has Threads. Yes, it does, but it’s a shadow of what’s available in Java. There, you can wield the sharp tools of concurrency with great effect on difficult problems but you can certainly stab yourself without understanding memory consistency effects, atomicity concerns and everything else.

Until recently, it seems the Rubyist approach to most of these issues has been to ignore them by slapping in a Big Fat Mutex. As someone who’s dealt with connection thread pools for RDBMs access for a long time, I was surprised to learn how innovative the non-blocking MySQL connector being worked on by espace as part of NeverBlock was considered. Fibers, mentioned by both Matz and Bruce Williams in discussing Ruby 1.9, offer a light-weight cooperative approach to something resembling concurrency. It was also heartening to hear Matz talk about how scalability is a big concern for the future of Ruby. Of course, no mention of these issues in Ruby is complete without noting that Ruby (and Rails) has been able to achieve a great deal in this area by using processes. This is usually a pretty simple and straightforward approach but it’s fairly large-grained and could certainly stand to be augmented with more sophisticated fine-grained capabilities that live inside the Interpreter.

Joy matters

Despite the above grumbling about concurrency, Ruby as a language is a beautiful thing. It really is damn pleasant to write Ruby code. I think this is a direct extension of Matz’s personality and concerns. In his keynote, he spoke powerfully on the need to see a prgramming language as part of the human interface to computers and the need to make that interface as joyful as possible to use. I don’t think I’ve once read “Joy” as a chief requirement in a JSR submitted to the Java Community Process for inclusion into Sun’s Java spec. That alone is probably enough reason to switch to Ruby. I do wonder how Matz’s vision of a humane language will hold up under the predicted onslaught of mindless Java-drones such as me that will create the 3 million new Ruby programmers expected over the next few years. As for that…

Don’t fear the hordes

We’re not all mindless and we’re not all drones. The unspoken sentiment when talking about the future growth of Ruby was “things are ok now because we’re all smart and dedicated craftsmen, unlike people who are using Java but will eventually invade beautiful ruby-land and make it ‘enterprisey'”. Growth is stressful for any group, especially one that gleefully defines itself as a minority in opposition to the mainstream as I believe Rubyists do. However, I think it benefits no one to assume that new Rubyists who may come to it later than you did are any less smart, dedicated or concerned about their work than you are. A community that doesn’t grow is bound to become introverted and ultimately stagnant. It would be a great loss if the Ruby community did that. The good news is I don’t think it will. Personally, I’ve felt welcomed by the Austin Rails community and I’ve also found plenty of resources elsewhere to help me learn. The best thing thing for Ruby’s future is to continue to embrace the hordes and show them what they’ve been missing.

And, In conclusion…

There are an incredible number of smart people that truly seem to love Ruby and want to see it succeed. Though there are ideologies, squabbles and disagreements, the part of the community assembled at LSRC was as close to a meritocracy as I’ve personally seen. The power in Ruby seems to lie with those that have most demonstrated their technical abilities and not with those that have the right corporate affiliations. This focus on merit leads to the sometimes duplicative and anarchic approach to developing new technology as people compete with their ideas. Ultimately, though, this is probably Ruby’s greatest strength and will serve it well in the future. I’m looking forward to being a part of it.

By the way, thanks to the Lone Star Ruby Foundation for putting on the conference. Great job. See you all at LSRC 2009.