A brief reintroduction to Yahoo! Pipes - Part 1 of 5

picture-4.pngRecently, I was reintroduced to Yahoo! Pipes by my buddy Paul. It wasn’t that I didn’t know about the site. In fact, I remember thinking it had great potential and signed up for the service when I first read about it on Techcrunch almost a year ago today. At the time, like most of us, I felt overwhelmed. I’m glad Paul made me delve back into because it has since changed my world. (If you’re not sure what Pipes is, I put a brief description at the end of this post)

Now, here’s where you come in. I don’t need this sweet service to disappear because a bunch of you don’t use it due to misunderstanding it. It is in your own best interest to read this damn post and apply what you read here. Not that I’ll send ninjas to kill your family if you don’t read it, but because it will positively change your world as well. I’m going to start this five part series by focusing on the types of pipes because that should send your imaginative minds reeling and make you come back for the details.


Probably the first thing a person thinks about doing with Pipes is to aggregate multiple sources into a single feed. There are some advantages to creating a feed from aggregate sources when compared to pulling individual feeds in a feed reader. If you give someone a single feed (instead of an OPML), then you can update it without requiring your subscribers to change their source. Also, Yahoo! Pipes offers some powerful filtering that can make a feed less noisy, such as eliminating duplicates or dropping negative matches. Here are a couple of types of aggregation:

Super feeds
Super feeds splice multiple sources into a single feed. You might use it to combine all of your favorite sports sites. Perhaps you could pull all of your favorite blogs. Maybe you’d use it to create a personal super feed of all of your data, such as your blog feed, Flickr feed, del.icio.us feed, etc.
Persistent searches
Many search sites now offer search results as a feed. So, if you wanted to stay on top of “web 2.0″, for example, then you could combine searches on Google News, Technorati, del.icio.us, and more into a single feed.


Notifications are a great way to stay on top of information without having to spend time doing it. Pipes can be used to create a variety of alerts, such as:

Watches are very specific searches that produce few results. There are many things a person might want to watch for, such as Xboxes for less than $500 on eBay, or an apartment on Craigslist that is near a park, or job postings that meet a certain criteria on Monster.com.
Response alert
A response alert is a notification that someone has responded to something you said. It’s great for watching comment threads, forum threads, and other places where you want to know if someone replied.
Let’s say you are waiting for something to come out, like a movie, CD, book, or video game. A release notification can be a great way to know when something becomes available for sale.
High priority information
This is just like a persistent search except it is a high priority to you, so you’d like it to be delivered via a text message or email. These differ from watches because they provide frequent results. If you are a real estate agent, you might want an email to your iPhone anytime someone posts a house for sale on Craigslist in your territory.
Some people in your network are very important to you, so you might want to know anytime they post a blog, Twitter, leave a comment, and more. Well, Pipes allows you to stay on top of any activity a person does that produces a feed, which is a lot of stuff. When used for ill will, this is also known as cyber stalking.


Pipes is great for bringing information to you, but it is also great for pumping it out to others too. Here are a few ways you can syndicate content from the web:

News feed
A news feed takes any feed and publishes it on your site.
Widgets take any feed and publishes it on someone else’s site.
Channels are podcasts, video feeds, or another encapsulated feed that a person can subscribe to for consumption.
A portal combines a variety of syndication feeds into a single location. The concept is similar to Net Vibes, PageFlakes, iGoogle, and the other homepage services.
This is where you take one source as a search criteria for another source. For example, one could create a YouTube channel powered by searching for the most popular tracks on Last.fm.


Reputation tracking
Reputation tracking is essentially persistent searches for yourself. It’s a great way to find out what’s being said about you online.
Buzz tracking
Yahoo! Pipes can be used to track the ripples a message of yours generates. You can track how many people talked about your post in the blogosphere, if it was bookmarked, if it was linked to, if it was Twittered, etc.
Meme tracking
Stay informed of popular topics using Yahoo! Pipes. Like a customized version of PopURLs.

This Friday I meeting with a small group of smart folks to further discuss Pipes. They may identify additional types of Pipes, which I’ll add to this post.

Coming up next in this series is:

  • Breakdown of Pipe elements
  • Tips for improving results
  • Pipes for profit
  • The future of pipes
About Yahoo! Pipes

Yahoo! Pipes is a web-based service that allows you to remix feed-based content on the web. It offers a graphic user interface for easy drag-and-drop style “programming”. Here’s a screenshot of the Pipe editor:

Justin - great post. I can’t wait to read the whole series. Pipes is an amazing and powerful tool. Just be careful not to give away too much of the secret sauce!

From Paul Biggs on January 9th, 2008 at 11:06 am

Bob Ross taught the world how to paint, but the world isn’t overrun with artists. ;)

From Justin on January 9th, 2008 at 1:34 pm

[...] Metafluence - A brief reintroduction to Yahoo! Pipes - Part 1 of 5 Probably the first thing a person thinks about doing with Pipes is to aggregate multiple sources into a single feed. There are some advantages to creating a feed from aggregate sources when compared to pulling individual feeds in a feed reader. (tags: yahoo pipes feed aggregator) [...]

From links for 2008-01-10 on January 9th, 2008 at 11:21 pm

I want an API that lets me aggregate and search on-the-fly, as my application needs it, though so far nothing I have found shows I can get that end result in any way with pipes.. Very sad.

From Josh on January 13th, 2008 at 12:05 am

I’ve been tinkering with a few pipes myself - like this one to geocode Obama’s tweets so I can display them on a map: http://blogs.open.ac.uk/Maths/ajh59/012712.html

Search http://ouseful.info for ‘yahoo pipes’ for plenty more examples

From Tony Hirst on February 1st, 2008 at 3:59 pm

Right on, Tony. I’ve been wanting to do more with mapping feed items using geocodes.

From Justin on February 2nd, 2008 at 8:48 pm

[...] A brief reintroduction to Yahoo! Pipes - Part 1 of 5 [...]

From Pipes Blog » Blog Archive » Our one year anniversary on February 7th, 2008 at 4:02 pm

At risk of blowing my own trumpet, anyone interested in Yahoo! Pipes might be interested in my book “Working with Yahoo! Pipes, No Programming Required” at http://www.lulu.com/content/1974260

From Tony Loton on February 14th, 2008 at 11:09 am

Hi Tony, I saw your book on the Yahoo! Pipes blog. I’d like to check it out.

From Justin on February 14th, 2008 at 8:35 pm

Tony, your website needs some help ;)

From Justin on February 14th, 2008 at 8:39 pm

[...] If you’d like a brief introduction to Pipes, I wrote a five part series on Pipes. [...]

From Minutes from “Dashboards” at Beer and Blog on February 18th, 2008 at 1:21 pm

[...] pipes, dapper, aiderss) out there to make the search for information so much easier, as well as great people writing free tutorials about it. The open and jointly collection of feeds could be the first step to build a ICT4D news ticker. The [...]

From Contribute to the combined ICT4D news feed : crisscrossed blog on March 5th, 2008 at 10:07 am

[...] by like-minded people in StumbleUpon and Justin Kirstner dropped the idea of implementing and using Yahoo Pipes with the RSS Feed. Great ideas both, but they are time-consuming efforts; that was also one of the problems of [...]

From Remove the RSS Curse; share your feed | social media and green horses on March 28th, 2008 at 5:00 am

What say you about all of this?

Trackback URL Comment feed

You must be logged in to post a comment.

Clicky Web Analytics