Integrating Netvibes, Pipes, AideRSS, Dapper for an intelligence dashboard

Sam’s dashboard Earlier today, Sam shared his executive dashboard that keeps him on top of our market’s information flow. I thought I would share some of the more technical details about integrating Netvibes, Pipes, AideRSS, and Dapper to create his dashboard.

Defining our objectives

We started this project with me interviewing Sam about the feeds in his reader. I also asked him about his behaviors online and existing relationships. I then pooled my notes [pic of my notes below] and began sketching up organizational concepts. Once Sam and I felt pretty good about the data we wanted and how to organize it, I began working on the dashboard. I found it easiest to do my work in this order:

  1. Create Dapper feeds
  2. Aggregate and manipulate feeds with Pipes
  3. Run outputs through AideRSS
  4. Import feeds and configure modules in Netvibes

Jive Talks feed stats show Netvibes is used by 50% of our readers

Netvibes brings it all together

dashboard notes I have to say that I haven’t done a comparative analysis between PageFlakes, NetVibes, iGoogle, MyYahoo, and the other players in the dashboard space. I noticed that 50% of the subscribers to Jive Talks used NetVibes, so I went with it [see image at right]. Netvibes offered a great selection of pre-built modules, and they have a large community of 3rd party modules, which has some really cool shit in there. Netvibes offers some great drag and drop functionality, as well as cool Javascript editing, like clicking on a tab’s name coverts it into a text field with save and cancel buttons, which really speeds up development. Plus, once I’ve got Sam dialed in, it’s easy for me to share the entire thing with the other Jive execs, which saves me a ton of time. If you want some more help with using Netvibes, just search Google for “intro to Netvibes“. If you want to know more about how to use Netvibes, check out Dawn’s intro to Netvibes (it also talks about Pipes and other services).

Powering modules with Pipes

This is a big part of the secret sauce behind Sam’s dashboard. He was already using a feed reader to aggregate a slew of sites, but as his list grew well past a hundred feeds. The big list of feeds was becoming unwieldy. Using Pipes, I created a variety of higher quality feeds from his list. One thing I did was aggregated similar sources into a single feed, such as all of the 2.0 sites (like Read/WriteWeb, Techcrunch, Mashable, GigaOm, Web Worker Daily, and more). I then took that super 2.0 feed and ran it through AideRSS. So, Sam has a column with the 2.0 news and it’s topped with an AideRSS module that tells him what is most popular in that river of news, which helps him join important conversations via commenting and writing trackback posts.

In addition to aggregating feeds, we use Pipes to manipulate feeds. So, for example, we filter an aggregated list of feeds to sift for any posts that have trigger keywords, such as our names, competition names, bursty keyphrases, etc. We also use Pipes to do things like add the initials of the author to a story’s title, which helps Sam know who is writing what story in a river of news. If you want to learn more about leveraging Pipes, I have a five part series that offers a brief introduction to Yahoo! Pipes.

Leveraging popularity to drive focus

One of the common criteria that we used to steer Sam’s focus was popularity. AideRSS offers a really cool service that lets you analyze the items in a feed. It ranks the items based on a few factors such as number of Diggs, number of times it was saved in delicious, number of comments on the post, and more. While I love this service, there are three big problems with it.

  1. It doesn’t resort items based on the rankings. Hellooooo! It doesn’t take a genius to figure out that resorting the items based on the ranking is really useful. In fact, I think the ranking is useless without resorting the items. But, worry not. Dawn has made a sweet Pipe that resorts a feed based on the AideRSS ranking. She also did a write up on it on her blog.
  2. It doesn’t like Pipes. I have been able to use a Pipes output as an AideRSS input, but most of the time it chokes. I’ve tried things like truncating the Pipes output to a reasonable number of items, like 20, but that doesn’t help. Not sure what the problem is, but it royally sucks. My only advice is to keep trying to submit your Pipes, however, you may have to wait a full 24 hours before trying again, otherwise AideRSS gives you an error message about too many attempts from your IP address.
  3. AideRSS has some caching problems. For example, when I was first setting up Sam’s blog, I messed up his feed and it was pointing to my blog’s feed. I corrected almost immediatly, which was over a month ago. But, AideRSS still pulls my blog’s feed if you enter Sam’s address. Stupid and frustrating.

Making feeds where none previously existed

Some of the data we needed to monitor didn’t come as RSS feeds. No bother, Marshall introduced us to Dapper, which is a screen scraper data extraction tool. I won’t go into a ton of techy detail here because Dapper isn’t very complicated. It does one thing and it does it well. Plus, Alex Iskold already did a great job detailing how Dapper works.

Less is more

While I have ideas that involve aggregating and filtering Dapper created feeds in Pipes, sending them out to AideRSS, then resorting them in Pipes based on the AideRSS ranking, and delivering them in a Sprout widget embedded in a Netvibes dashboard; it turns out that is a bad idea. Most of these services are in beta, which means they are unstable—I mean that both in terms of service and financially. Programming best practices recommends eliminating as many points of failure as possible. So, sometimes I’ll sacrifice a small advantage in favor of eliminating a service in the data pipeline. I ruled out using Sprouts as a whole because I found that I was basically displaying feeds with them anyway (with some extra buttons I could live without). Sprout makes much better sense as a syndication tool for promoting content, at least at this stage, but that is a subject for a deeper analysis of Sprout, which I’m releasing later this week. :)

One big annoyance from integrating multiple feed services is lag. Timely delivery of data is a crucial part of a dashboard’s success. The more systems you route your data through, the more impedance your pipeline has, which means data gets to you slower. I’m in the midst of charting how often each systems updates their feed and what factors influence caching.

Join the Portland Data Plumbers Google Group

Mashing up so many different services, most of which are in beta, around a loosely standardized language is a difficult and dynamic task. In order to stay on top of everything, a few of us have been informally getting together on a kinda monthly basis to problem solve together. If you are interested in participating with us, I started a Google Group called the Portland Data Plumbers. Check it out and join up if it piques your interest. We’re looking for people that can contribute, not just lurk.

I’m super interested in hearing other peoples’ experiences with feed service integration including quality feed sources, cool Pipes, integration tips, and dashboard use cases. I’m also down for helping people out, so if you have a question, drop me a comment. Helping people takes my mind places it wouldn’t go otherwise. :)

[...] August 11, 2006), so I hadn’t really thought much about how I read my feeds until recently. Justin was in the process of putting together some intelligence dashboards for the execs at Jive to keep [...]

From Tips for RSS Feeds, Yahoo Pipes, Attention, and Netvibes at Fast Wonder Blog on February 6th, 2008 at 5:41 pm

Hi, perfect step by step approach! Thanks.I will get myself into Yahoo pipes…;)

From netviber on February 7th, 2008 at 4:44 am

[...] been meaning to write about my personal technology toolkit for a few weeks. Reading about executive dashboards at Jive reminded me [...]

From Different Tools For Different Folks « Dyepot, Teapot on February 7th, 2008 at 12:42 pm

[...] Metafluence - Integrating Netvibes, Pipes, AideRSS, Dapper for an intelligence dashboard Explanation of how Justin pulls together a dashboard for the execs at Jive (tags: dashboard rss feeds netvibes) [...]

From User First Web » links for 2008-02-08 on February 8th, 2008 at 1:22 am

Tremendous post. I use NetVibes and AideRSS quite a bit already; Pipes and Dapper have always intimidated me a bit. But I’m in the business of teaching/coaching social-web literacy, and I’ve long thought that the really good dashboard is about as important as anything you can provide, for example, to a new blogger.

Frankly, it’s going to be a challenge for me to duplicate what you’ve done — I appreciate the detail here (and links off to additional detail), and imagine I’ll be digesting it in small bites over the next little while. There’s a lot here for me.

From Ian Wilker on February 8th, 2008 at 7:50 pm

Ian, I’m stoked to hear that there is enough here in this post to keep you going a while! :)

From Justin on February 9th, 2008 at 1:02 pm

[...] from Metafluence started a project which takes things ten steps further in a quest to create a more intelligent dashboard. With mixed [...]

From Advanced Streamline Techniques » Darren Hoyt Dot Com on February 18th, 2008 at 11:18 pm

[...] einem Kommentar zu diesem Post, verweist Fabian Herbel auf ein Tutorial von Metafluence, in dem beschrieben wird, wie ein MashUp aus Netvibes, Yahoo Pipes, AidRSS und Dapper genutzt [...]

From Yahoo Pipes beim Monitoring - on February 25th, 2008 at 5:49 am

[...] Metafluence - Integrating Netvibes, Pipes, AideRSS, Dapper for an intelligence dashboard An awesome post — create a NetVibes dashboard that separates the wheat from the chaff, no matter how unmanageable your feed collection has become. (tags: RSS feeds startpages intelligence) [...]

From links for 2008-02-09 : roots.lab on February 27th, 2008 at 4:00 pm

[...] Metafluence - Integrating Netvibes, Pipes, AideRSS, Dapper for an intelligence dashboard Klingt nach sehr professionellem Informationsmanagement (via ckreutz) (tags: howto mashup feed) [...]

From links for 2008-03-04 « Das Textdepot on March 3rd, 2008 at 11:20 pm

[...] mining and tracking what’s happening on the social web that do not require the user to master hellaciously complicated routines? I kind of like hellaciously complicated, but I’m a freak that [...]

From roots.lab on April 11th, 2008 at 11:43 am

What say you about all of this?

Trackback URL Comment feed

You must be logged in to post a comment.

Clicky Web Analytics