Earlier today, Sam shared his executive dashboard that keeps him on top of our market’s information flow. I thought I would share some of the more technical details about integrating Netvibes, Pipes, AideRSS, and Dapper to create his dashboard.
Defining our objectives
We started this project with me interviewing Sam about the feeds in his reader. I also asked him about his behaviors online and existing relationships. I then pooled my notes [pic of my notes below] and began sketching up organizational concepts. Once Sam and I felt pretty good about the data we wanted and how to organize it, I began working on the dashboard. I found it easiest to do my work in this order:
- Create Dapper feeds
- Aggregate and manipulate feeds with Pipes
- Run outputs through AideRSS
- Import feeds and configure modules in Netvibes
Netvibes brings it all together
If you want some more help with using Netvibes, just search Google for “intro to Netvibes“. If you want to know more about how to use Netvibes, check out Dawn’s intro to Netvibes (it also talks about Pipes and other services).
Powering modules with Pipes
This is a big part of the secret sauce behind Sam’s dashboard. He was already using a feed reader to aggregate a slew of sites, but as his list grew well past a hundred feeds. The big list of feeds was becoming unwieldy. Using Pipes, I created a variety of higher quality feeds from his list. One thing I did was aggregated similar sources into a single feed, such as all of the 2.0 sites (like Read/WriteWeb, Techcrunch, Mashable, GigaOm, Web Worker Daily, and more). I then took that super 2.0 feed and ran it through AideRSS. So, Sam has a column with the 2.0 news and it’s topped with an AideRSS module that tells him what is most popular in that river of news, which helps him join important conversations via commenting and writing trackback posts.
In addition to aggregating feeds, we use Pipes to manipulate feeds. So, for example, we filter an aggregated list of feeds to sift for any posts that have trigger keywords, such as our names, competition names, bursty keyphrases, etc. We also use Pipes to do things like add the initials of the author to a story’s title, which helps Sam know who is writing what story in a river of news. If you want to learn more about leveraging Pipes, I have a five part series that offers a brief introduction to Yahoo! Pipes.
Leveraging popularity to drive focus
One of the common criteria that we used to steer Sam’s focus was popularity. AideRSS offers a really cool service that lets you analyze the items in a feed. It ranks the items based on a few factors such as number of Diggs, number of times it was saved in delicious, number of comments on the post, and more. While I love this service, there are three big problems with it.
- It doesn’t resort items based on the rankings. Hellooooo! It doesn’t take a genius to figure out that resorting the items based on the ranking is really useful. In fact, I think the ranking is useless without resorting the items. But, worry not. Dawn has made a sweet Pipe that resorts a feed based on the AideRSS ranking. She also did a write up on it on her blog.
- It doesn’t like Pipes. I have been able to use a Pipes output as an AideRSS input, but most of the time it chokes. I’ve tried things like truncating the Pipes output to a reasonable number of items, like 20, but that doesn’t help. Not sure what the problem is, but it royally sucks. My only advice is to keep trying to submit your Pipes, however, you may have to wait a full 24 hours before trying again, otherwise AideRSS gives you an error message about too many attempts from your IP address.
- AideRSS has some caching problems. For example, when I was first setting up Sam’s blog, I messed up his feed and it was pointing to my blog’s feed. I corrected almost immediatly, which was over a month ago. But, AideRSS still pulls my blog’s feed if you enter Sam’s address. Stupid and frustrating.
Making feeds where none previously existed
Some of the data we needed to monitor didn’t come as RSS feeds. No bother, Marshall introduced us to Dapper, which is a
screen scraper data extraction tool. I won’t go into a ton of techy detail here because Dapper isn’t very complicated. It does one thing and it does it well. Plus, Alex Iskold already did a great job detailing how Dapper works.
Less is more
While I have ideas that involve aggregating and filtering Dapper created feeds in Pipes, sending them out to AideRSS, then resorting them in Pipes based on the AideRSS ranking, and delivering them in a Sprout widget embedded in a Netvibes dashboard; it turns out that is a bad idea. Most of these services are in beta, which means they are unstable—I mean that both in terms of service and financially. Programming best practices recommends eliminating as many points of failure as possible. So, sometimes I’ll sacrifice a small advantage in favor of eliminating a service in the data pipeline. I ruled out using Sprouts as a whole because I found that I was basically displaying feeds with them anyway (with some extra buttons I could live without). Sprout makes much better sense as a syndication tool for promoting content, at least at this stage, but that is a subject for a deeper analysis of Sprout, which I’m releasing later this week. :)
One big annoyance from integrating multiple feed services is lag. Timely delivery of data is a crucial part of a dashboard’s success. The more systems you route your data through, the more impedance your pipeline has, which means data gets to you slower. I’m in the midst of charting how often each systems updates their feed and what factors influence caching.
Join the Portland Data Plumbers Google Group
Mashing up so many different services, most of which are in beta, around a loosely standardized language is a difficult and dynamic task. In order to stay on top of everything, a few of us have been informally getting together on a kinda monthly basis to problem solve together. If you are interested in participating with us, I started a Google Group called the Portland Data Plumbers. Check it out and join up if it piques your interest. We’re looking for people that can contribute, not just lurk.
I’m super interested in hearing other peoples’ experiences with feed service integration including quality feed sources, cool Pipes, integration tips, and dashboard use cases. I’m also down for helping people out, so if you have a question, drop me a comment. Helping people takes my mind places it wouldn’t go otherwise. :)