02
Aug 10

Easy Twitter Streaming API Testing with Mockingbird

My company makes two applications that focus on Twitter: CheapTweet and TweetReach. Lately, we’ve started transitioning parts of these apps to the Twitter Streaming API. To do that, we’ve been developing the flamingo project to encapsulate all the best practices of working with Twitter streams so that others won’t have to solve all the same problems. I’m not going to talk about flamingo today, though (I’ll do that soon). I’m going to talk about testing.

Testing code that talks to the Twitter Streaming API is hard. The are 8 documented failure HTTP response codes, random disconnections, and second-to-second rate changes. And, since this is Twitter we’re talking about, basically anything can happen at any time – including complete downtime.

So, let me introduce you to mockingbird. Mockingbird is a simple streaming HTTP server written in Ruby that you can use in place of the server at stream.twitter.com when you’re testing your Streaming API code. With mockingbird, all it takes is a few lines of code and you can simulate all the issues I described above and more. Here’s an example config:

Mockingbird.setup(:port=>8080) do
  wait 1
  5.times do
    send '{"foo":"bar"}'
    wait { rand }
  end
  close
end

So what does this do?

  1. Starts a mockingbird server on port 8080 on your localhost
  2. When there’s a connection, it starts by waiting for 1 second before sending any data
  3. Then, it sends ‘{“foo”:”bar”}’ 5 times with each time followed by randoom wait of between 0 and 1 seconds
  4. After that, it closes the connection and waits for more

You can do lots of other things with this simple configuration style. See the README and code examples at github.

Mockingbird provides a simple setup/teardown mechanism so it’s easy to start a server and shut it down during your unit tests, specs, or whatever. Here’s an example Test::Unit test:

class TestConnection < Test::Unit::TestCase
  def setup
    # Starts the server running a separate process
    Mockingbird.setup(:port=>8080) do
      status 500, "Error"
      close
    end
  end

  def test_handle_500_errors
    # Your test code here for connecting to localhost:8080
    # and handling a 500 error here
  end

  def teardown
    # Stops the server process
    Mockingbird.teardown
  end
end

We’ve already found mockingbird to be extremely useful for testing flamingo. If you’re using the Streaming API and need to be confident your code can withstand the crazy stuff Twitter might throw at it, I think you’ll probably find it helpful, too. Of course if you have questions, comments, ideas or (even better) pull requests, hit me up.


26
Jul 10

ActiveSupport Considered Harmful

This an open letter to my fellow ruby gem authors and maintainers.

Hi Everybody,

It’s been a while. You’re doing awesome stuff; all this great code out there making Ruby do all sorts of nifty things. It’s a big reason – maybe the reason – why being a Rubyist is so great. But yeah, there’s just one thing that’s been bugging me… we need to talk about ActiveSupport.

ActiveSupport is great, don’t get me wrong. I love try and tap and sum as much as the next guy. It sure saves me some lines of code. But the problem is it’s, well, a little invasive. Many of us love Ruby precisely because it’s more flexible than a Russian gymnast. But just because you can reopen Object and add

class Object
  def your_great_method
    puts "HEY I'M CHANGING OBJECT"
  end
end

doesn’t mean you always should. As we all know, ActiveSupport is one giant ball of I’M CHANGING OBJECT (and Enumerable and Array and String and, well, you get the picture). This is a pretty awesome idea in a controlled environment like Rails. Rails is all-encompassing. When you write Rails, you’re not really in Ruby anymore, you’re 3 layers deep, Inception style, inside DHH’s dream of web programming. In that world, Object can do magic things and Strings can inflect right back on themselves. And so, of course, ActiveSupport is an integral part of Rails (more on that in a moment).

The problem comes when you decide you just can’t  live without tap or a few of these other modern conveniences and so you throw in the ol’ ActiveSupport dependency in the gemspec. The first effect of this is that whatever project your users are working on now much bring in this giant ball o’ Ruby changin’ with its various invasions, idiosyncrasies and occasional incompatibilities.

But, you say, most people are using Ruby for Rails and Rails has ActiveSupport baked right in. We’re good right? Well, no, actually. This is where it gets really painful. When you’re gemspecin’ and you decide to throw in that latest version of ActiveSupport, well, you’re basically telling every Rails user out there that’s not on the latest version that they need to upgrade their entire environment just to use your gem. Now, the Ruby community is made up of people who are pretty early adopters so many people are on the latest and won’t even notice. However, as Ruby gets more mature and lots of people have lots of versions of things out there, not everyone can just upgrade at the drop of a hat. You’re cutting off those people from your gem awesomeness unless they either a) upgrade (which isn’t always immediately possible) or b) resort to horrifying monkey patches. And all this because you just love sum so much.

So, my fellow gem authors, I ask you to think twice next time you really want to use that convenience method that’s only in ActiveSupport N.N.N. Perhaps a little helper method would do? Maybe one that’s in your gem’s own namespace perhaps? Your users will thank you.

Sincerely,

Hayes


19
Jul 10

Austin

At the beginning of 2005, I knew nothing about this city that I’ve come to love. That spring, Jenn was accepted into the Organizational Communication PhD program at UT and suddenly Austin was on the short list of places that we might make our home. We came to visit for a few days; it was warm (even in March), it was green, there was a river, there were hills, the people were friendly and there seemed to be a never-ending supply of bars and tex-mex. It was my first visit to Texas. We came back again for a weekend June. On that second visit to Texas, we bought a house. And so, five years ago this month, we moved to Austin from Raleigh, NC.

Jenn went to school and I worked remotely for a company in Raleigh. This was the first chapter. We made friends, we played kickball, we learned our way around. We got used to the heat (though the humidity wasn’t as bad as North Carolina despite what anyone said). This part was fun, but I never felt terribly connected to Austin. Most of our friends were here for grad school as well. They were from all over and knew they would be leaving in a few years. These are some of my dearest friends and will be so for the rest of my life, but we could all have met up anywhere. So that was not when I came to love Austin itself.

In 2008, I was feeling the itch to start a new company. This is the second chapter. That spring, we went to SXSW. In May, I quit my remote job in Raleigh and Jenn and I founded Appozite. I was a Ruby n00b but had decided that I would write our software in Ruby. I’ve never been much of a “user group” kind of person but I wanted to learn, so I went to an Austin on Rails meeting. I was in over my head during the presentations and I didn’t know anyone. Afterward though, Damon Clinkscales took the time to sit down with me over beers. He was genuinely interested in who I was and what I was doing. At that point, I realized there was something special here in the tech community Austin.

What followed has been the most fun, stressful and interesting two years of my life so far. And I owe most of that to the people here in Austin. Jenn and I went to all sorts of things: Refresh, tweetups, Bedouins, more SXSWs and on and on. I got to know other people struggling with the same challenges I was. I learned from those who had already been through it a few times, with their experience of both successes and failures. I felt what it was like to be part of a real community of people doing diverse things, learning from each other, sometimes competing with each other, and always supporting each other. I’ve made incredible friends in Austin over these last two years and met some incredible people who have advised us as we continue to build Appozite. This is when I learned to love the unique people and character of Austin.

The people are the reason it’s now so difficult to say goodbye. In a few weeks, Jenn and I will be moving to San Francisco to pursue our goals both personally and professionally. We have big ambitions for Appozite and our products and we believe that the Bay is the right place to realize those. More than that, we’ve always had a deep love of San Francisco; we spent our honeymoon there. We know that, for us, now is the time.

So, to all our friends in Austin and to so many of you who have supported us, thank you. You probably don’t realize the impact you’ve had on making us who we are today. We love this city and the people.

Goodbye, Austin.


16
Jun 10

Grackwalla: Talking to the Gowalla API using grackle

A while back I wrote a Twitter API ruby library called grackle. It’s seen quite a bit of use and takes a very lightweight approach to talking to the Twitter API. In fact, the thing I really like about grackle is that it really doesn’t make many assumptions about the API it’s talking to – it just issues RESTful requests using a straightforward syntax. This works great for talking to Twitter and it means grackle can easily be setup to talk to other APIs that use standard RESTful approaches. I thought I’d demonstrate this by setting grackle up to talk to Gowalla’s API. So, here goes:

client = Grackle::Client.new(
  :headers=>{"Content-Type"=>"application/json",
    "Accept"=>"application/json"},
  :auth=>{:type=>:basic,:username=>'USERNAME',:password=>'PASSWORD'})
client.api_hosts[:gowalla] = 'api.gowalla.com'
client.api = :gowalla

So what did I do there? Well here are the key elements:

  • First off, I set the headers as required by Gowalla’s API. Basically I’m just telling Gowalla that I want JSON back (which grackle understands) and can convert into a usable object.
  • Second, I tell grackle to use HTTP basic authentication as required by Gowalla
  • The client.api_hosts line is a little bit more esoteric. Grackle keeps a list of named “hosts”. That’s a little bit of a misnomer, it’s really a prefix for each request grackle makes. In this case, I’m telling grackle about the api.gowalla.com host and then telling it to use the new named host called :gowalla for subsequent requests.

If you’re unfamiliar with how grackle works, take a look at the grackle README and all will be explained. (Now might be a good time to do that… go ahead, I’ll wait…) With that bit of setup above, it’s now trivial to do things like:

#GET /users/hayesdavis
user = client.users.hayesdavis?
puts "#{user.first_name} #{user.last_name} has #{user.stamps_count} stamps"
# Prints "Hayes Davis has 184 stamps"

#GET /users/hayesdavis/stamps?limit=20
client.users.hayesdavis.stamps? :limit=>20 

#GET /users/categories
client.categories? 

#GET /spots?lat=30.2697&lng=-97.7494&radius=50
client.spots? :lat=>30.2697, :lng=>-97.7494, :radius=>50

The return values of the above methods will contain things called “TwitterStructs” but, never fear, those actually work like Ruby OpenStructs and match the JSON object returned by Gowalla’s API.

So go out and Gowalla with grackle. If you’re interested in connecting to other RESTful APIs, you can follow a similar pattern to get going.

P.S. My sincerest apologies for introducing the word “grackwalla” into the world.


28
Mar 10

Using MongoDB and a Relational Database Together

Below is the presentation I gave to the Austin MongoDB Day on 3/27/2010. I talked about the design decisions one needs to make when determining whether to introduce a document database like MongoDB into an existing application built on a relational database. I give a few examples of how we’ve made MongoDB and our MySQL database live together in harmony inside CheapTweet.

Thanks to Lynn Bender of GeekAustin, CoSpace, and the folks at 10Gen for putting on a great event.


29
Jul 09

Introduction to Sphinx and Thinking Sphinx: 10 Minute Talk at Austin on Rails

I did a quick talk at Austin on Rails last night about my experience using Sphinx and Thinking Sphinx for doing full-text search on CheapTweet. We did a series of short talks last night so this is only a 10 minute overview. Hope some of you will find it helpful.


25
Mar 09

Austin on Rails Talk: Building TweetReach with Sinatra, Tokyo Cabinet and Grackle

I gave a talk last night (3/24/2009) at Austin on Rails about building a new (yet-to-be-released) application called TweetReach using tools that are somewhat off the beaten path. These included Sinatra, Tokyo Cabinet and my new Twitter API library: Grackle (which I’ll talk about in more detail in a future blog post).

I referenced some code during the “Building TweetReach” part of the presentation. If you’re interested, you can download it.

By the way, I was the second speaker of the night. Mike Perham spoke before me about caching with Rails. Check his stuff out as well. Thanks to everyone who came out.

Update 4/2/2009: Launched TweetReach! Go check it out.


08
Jan 09

Bringing RSS to e-Commerce: Announcing ZiteFeed

ZiteFeed - Feeds for e-commerceRSS and related syndication technologies (like Atom) have become just about ubiquitous online these days. We expect every self respecting blog and news site to show us that cute little orange broadcast logo when we visit that lets us know they’re interested in keeping us up to date with their latest information. Pretty much every social site offers the option to receive updates via RSS. In short, it’s a critical and widely-supported technology all over the web. In fact, RSS is becoming so well known that people aren’t even searching for it to find out what it means anymore.

There’s one place online though where the syndication revolution just hasn’t taken hold. Yet… That’s on e-commerce sites. Many times I’ve been on a site that had some products I wanted to keep an eye on but found there was no way to be notified when prices change, the store has a sale or new items come in. Instead I was stuck with two options:

  1. Sign up for an email newsletter that would land in my inbox who knows how many times and probably not contain anything I was interested in.
  2. Try to remember the site and visit it later.

Guess which one I usually chose? You guessed it, neither one! That’s not good for me and it’s not good for the retailer. I lose out on finding just what I’m looking for and the retailer loses an eventual sale. If I had the option of receiving unobtrusive relevant updates in my feed reader, I would happily have subscribed and probably returned to the store to buy something. Without that, I just forget it.

Well, at Appozite, we’re fixing this problem with today’s launch of ZiteFeed. ZiteFeed provides a simple way for any e-commerce site to turn routine product updates and promotions into RSS feeds. ZiteFeed hosts the feeds, provides easy-to-use embeddable widgets for feed subscriptions and gives site owners in-depth reporting so they can measure how their feeds are used. Just like any RSS feeds these can be added to feed readers, placed on personalized home pages and generally subscribed to in any software that understands RSS.

If you run an e-commerce site and you’re interested in adding ZiteFeed functionality to your site, let us know. And if you’re buying online from a store that doesn’t offer feeds, tell them they’re missing out and send them to ZiteFeed!


07
Sep 08

A recent Rubyist goes to the Lone Star Ruby Conference

I attended the Lone Star Ruby Conference (LSRC), held here in Austin over the past two days. As this was my first conference on the topic, I thought I would share some observations. First, a vigorous caveat emptor… I’m a former Java guy still relatively new to the Ruby world. At the very least I hope to provide the humble perspective of a transplant from the crowded streets of Javaopolis to the rolling hills of Ruby country. So here goes…

Rubyists aren’t afraid of change (and some churn)

I’ve long been an interested outsider to Ruby and Rails – following along on the fringes for a couple years now by reading articles, running irb occasionally and generating some scaffolds. At that distance, you basically see Ruby and Rails. What you don’t see is the constant churn of open source projects, plugins, development tools, GUI toolkits, ideologies and best practices. Gregg Pollack and Jason Seifer’s talk on innovations in the last year and Evan Phoenix‘s keynote about Ruby memes made the currents of change pretty clear. Java experienced a lot of change during my 8 years but it definitely seemed to be at a more staid pace. There were a few bigger, more corporate, entities driving it (Sun, Apache, IBM) and fewer mythologized personalities like why. In the Ruby community, a lot of energy seems to be expended reinventing various wheels (as Evan pointed out with his list of ARGV processors). There’s a fine distinction between a healthy competition of ideas and a naval-gazing, ego-stroking churn that wastes time. That said, I would never stand in the way of the free market of ideas. So, by all means, (with no apologies to Mao) let 1000 ARGV processors bloom and let’s see which stick.

There are some surprising technology holes – but they’re being filled

As a former Java programmer who worked deeply with concurrency, Ruby’s after-thought of a concurrency model frightened me a bit. Java, though described by Dave Thomas on the last night’s panel as “a blunt tool”, has a basement filled with extremely sharp swords known as threads and a well-defined concurrency model. But, you say, Ruby has Threads. Yes, it does, but it’s a shadow of what’s available in Java. There, you can wield the sharp tools of concurrency with great effect on difficult problems but you can certainly stab yourself without understanding memory consistency effects, atomicity concerns and everything else.

Until recently, it seems the Rubyist approach to most of these issues has been to ignore them by slapping in a Big Fat Mutex. As someone who’s dealt with connection thread pools for RDBMs access for a long time, I was surprised to learn how innovative the non-blocking MySQL connector being worked on by espace as part of NeverBlock was considered. Fibers, mentioned by both Matz and Bruce Williams in discussing Ruby 1.9, offer a light-weight cooperative approach to something resembling concurrency. It was also heartening to hear Matz talk about how scalability is a big concern for the future of Ruby. Of course, no mention of these issues in Ruby is complete without noting that Ruby (and Rails) has been able to achieve a great deal in this area by using processes. This is usually a pretty simple and straightforward approach but it’s fairly large-grained and could certainly stand to be augmented with more sophisticated fine-grained capabilities that live inside the Interpreter.

Joy matters

Despite the above grumbling about concurrency, Ruby as a language is a beautiful thing. It really is damn pleasant to write Ruby code. I think this is a direct extension of Matz’s personality and concerns. In his keynote, he spoke powerfully on the need to see a prgramming language as part of the human interface to computers and the need to make that interface as joyful as possible to use. I don’t think I’ve once read “Joy” as a chief requirement in a JSR submitted to the Java Community Process for inclusion into Sun’s Java spec. That alone is probably enough reason to switch to Ruby. I do wonder how Matz’s vision of a humane language will hold up under the predicted onslaught of mindless Java-drones such as me that will create the 3 million new Ruby programmers expected over the next few years. As for that…

Don’t fear the hordes

We’re not all mindless and we’re not all drones. The unspoken sentiment when talking about the future growth of Ruby was “things are ok now because we’re all smart and dedicated craftsmen, unlike people who are using Java but will eventually invade beautiful ruby-land and make it ‘enterprisey’”. Growth is stressful for any group, especially one that gleefully defines itself as a minority in opposition to the mainstream as I believe Rubyists do. However, I think it benefits no one to assume that new Rubyists who may come to it later than you did are any less smart, dedicated or concerned about their work than you are. A community that doesn’t grow is bound to become introverted and ultimately stagnant. It would be a great loss if the Ruby community did that. The good news is I don’t think it will. Personally, I’ve felt welcomed by the Austin Rails community and I’ve also found plenty of resources elsewhere to help me learn. The best thing thing for Ruby’s future is to continue to embrace the hordes and show them what they’ve been missing.

And, In conclusion…

There are an incredible number of smart people that truly seem to love Ruby and want to see it succeed. Though there are ideologies, squabbles and disagreements, the part of the community assembled at LSRC was as close to a meritocracy as I’ve personally seen. The power in Ruby seems to lie with those that have most demonstrated their technical abilities and not with those that have the right corporate affiliations. This focus on merit leads to the sometimes duplicative and anarchic approach to developing new technology as people compete with their ideas. Ultimately, though, this is probably Ruby’s greatest strength and will serve it well in the future. I’m looking forward to being a part of it.

By the way, thanks to the Lone Star Ruby Foundation for putting on the conference. Great job. See you all at LSRC 2009.


30
Jul 08

1, 2, 3… ZiteFight!

It’s been really quiet around here because I’ve been hard at work for Appozite these past few weeks. We’re very excited to announce the launch of our first app, ZiteFight. So what’s the deal with ZiteFight? Well, it’s the World Championship of Style. ZiteFight pits user-submitted photos against each other and lets the world pick which one has the best style. Now, I know this blog skews developer (what with the Hadoop posts and all) and developers aren’t exactly known for their style but I’m sure you know someone who thinks they’ve got good taste. Tell them to go prove it to the world by joining the fight at ZiteFight.