Thursday, February 28, 2013

Drinking from the Streaming API

Today we’re open-sourcing the Streaming API. The client is full featured: it offers support for GZip, OAuth and partitioning; automatic reconnections with appropriate backfill counts; access to raw bytes payload; proper retry schemes, and relevant statistics. Even better, it’s been battle-tested in production by our internal teams. We highly recommend you take advantage of the Hosebird Client if you plan on working with the Streaming API.

Using Hosebird

The Hosebird Client is broken into two main modules: hbc-core and hbc-twitter4j. The hbc-core module uses a simple message queue that a consumer can poll for messages. The hbc-twitter4j module lets you use the superb Twitter4J project and its data model on top of the message queue to provide a parsing layer.

The first step to use Hosebird is to setup the client using the ClientBuilder API:

/ Create an appropriately sized blocking queue
BlockingQueue<String> queue = new LinkedBlockingQueue<String>(10000);
/ Authenticate via OAuth
Authentication auth = new OAuth1(consumerKey, consumerSecret, token, secret);
/ Build a hosebird client
ClientBuilder builder = new ClientBuilder()
   
.hosts(Constants.STREAM_HOST)
   
.authentication(auth)
   
.endpoint(new StatusesSampleEndpoint())
   
.processor(new StringDelimitedProcessor(queue))
   
.eventMessageQueue(queue);
Client hosebirdClient
= builder.build();


After we have created a Client, we can connect and process messages:

client.connect();
while (!client.isDone()) {
 String message = queue
.take();
 System.out.println(message); / print the message}


Hosebird Examples

We recommend you learn from the examples on GitHub or contribute your own.

If you want a quick example, set these properties in hbc-example/pom.xml:
<consumer.key>SECRET</consumer.key>
<consumer.secret>SECRET</consumer.secret>
<access.token>SECRET</access.token>
<acesss.token.secret>SECRET</acesss.token.secret>


Then you can run this command on the command line:
mvn exec:java -pl hbc-example

This will connect to the sample stream API and print 1000 JSON items from the API.

Acknowledgements
The Hosebird Client was primarily authored by Steven Liu (@TwitterAPI team for their thoughtful suggestions and help.

On behalf of the Hosebird team,
- Chris Aniszczyk, Manager of Open Source (@cra)

Follow Lee on X/Twitter - Father, Husband, Serial builder creating AI, crypto, games & web tools. We are friends :) AI Will Come To Life!

Check out: eBank.nz (Art Generator) | Netwrck.com (AI Tools) | Text-Generator.io (AI API) | BitBank.nz (Crypto AI) | ReadingTime (Kids Reading) | RewordGame | BigMultiplayerChess | WebFiddle | How.nz | Helix AI Assistant