A brief reintroduction to Yahoo! Pipes - Part 2 of 5

picture-4.pngIn Part 1 of this series, I provided some examples of things you could do with Yahoo! Pipes, but I didn’t offer any explanation about how they work. That’s where this Part 2 comes in. I am not a developer by trade. I am a designer turned strategist. The following is an explanation of how I understand Pipes to function.

Pipe elements

Yahoo! offers a good explanation for the service:

Yahoo! Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.

The front end of a pipe looks like this:


This is an overlay describing what’s on the page:

If you have are logged into your Pipes account you can create new pipes from scratch or copy other peoples’ pipes, which you can use as is or modify. You can edit pipes by hand coding or use their drag-and-drop interface. I use their graphic user interface (GUI) to create pipes, so this information is directed toward Pipes’ GUI, which looks like this:

This is an overlay describing what’s on the page:

Pipes are comprised of several elements:

These are the feeds you are bringing into the pipe. It might be a Google search, a blog’s RSS feed, or some other feed. Sources are the data you are aggregating, manipulating, or mashing up.
User Inputs
User Inputs are the form elements that appear on the front end of your pipe to make them user friendly, like the search box for an aggregated search pipe. Inputs are things like search terms, the date, a location, or another datum a user would enter into a pipe.
Operators are the action-oriented modules in Pipes. They perform tasks like sorting, filtering, merging, etc.
URL work with sources that require a URL input. Yahoo! created some sources like a Yahoo! Local search or Flickr, but they didn’t make a module for every possible source. For example, they don’t have a Technorati source module. URL modules make it possible to create your own sources.
String modules are for the more advanced plumbers. Since this is a brief introduction to Yahoo! Pipes, we’re not going to cover it here. If you know what a string is, you’ll know what to do with these modules.
Pipes are built to enable a flow of information, so time is a critical piece to pipes. You might use the Date modules as an input, such as a picture of the day pipe that only pulls Flickr images posted the same day.
One of the sweet things you can do with the data a pipe spits out is render it on a map. Location modules help you manipulate geo data.
Like time and location, numbers come up a lot, so they have modules to deal with them.

You can also use Pipes as modules in a new pipeline (if you have a ton, Favorites can be a good way to quickly access your block and tackle pipes)

Pipe patterns

Generally, pipes have three main sections:

Inputs are the form elements that appear on the front end of your pipe to make them user friendly, like the search box for an aggregated search pipe. Inputs are things like search terms, the date, a location, or another datum a user would enter into a pipe.
A pipe could have one or many sources. Sources include RSS feeds, CSV files, JSON, KML, and more. These are the modules that define what data you are going to put through your pipeline.
The tail is a big part of the magic of a pipe. This is where you do stuff like filtering, sorting, and other kinds of data manipulation. When there are multiple feeds, the tail looks like the bottom of a funnel, which is appropriate because that’s kinda what it does. The tail brings the data together and cleans it up.

Here is an image of the basic pipe structure outlined above:

Here is an overlay labeling the three main sections:

What makes a pipe valuable is its output. I don’t know which is more valuable–knowing the sources to put into a pipe or how to process them. For inspiration for the kinds of things you can do with pipes, check out Part 1 of this series.

Output options

As I covered earlier in this post, Yahoo! offers a front end to their pipes that allow a person to consume the output on the site. You can also get the content in the following ways:

Great for consuming in a feed reader or for inputting into another system, such as a widget for syndicating the feed’s content.
Great for those who still see email as a important channel for information.
Great for high priority information.
Great for pumping information into an application or script.


Again, since this is a brief introduction to Yahoo! Pipes, I won’t go into detail about what each module does. However, know that you have a basic understanding of pipes, you should be able to view the source of other people’s pipes. Tweaking other people’s pipes is a great way to familiarize yourself with the modules and jump start your imagination. Here are some links to pipes that got my brain moving:

Stay tuned for the rest of this series:

  • Tips for improving results
  • Pipes for profit
  • The future of pipes

Very good tutorial, keep working!
Did you see my site MacrosReader? You can run a lot of Yahoo Pipes there:

From Dan N. Moldovan on January 9th, 2008 at 8:07 am

Thanks, Dan. I hadn’t seen MacrosReader before. I’m checking it out now.

From Justin on January 9th, 2008 at 9:48 am

I’m starting think of all kinds of things I could do with these :-)

From Dawn Foster on January 9th, 2008 at 5:08 pm

That’s what I’m talking about. :)

From Justin on January 9th, 2008 at 6:49 pm

You missed out Ical and CSV output.

From hapdaniel on January 17th, 2008 at 4:25 am

Thanks for the catch, hapdaniel.

From Justin on January 17th, 2008 at 10:20 am

I’ve already produced some Pipes … re-reading your series to heh see what I missed.

Do me a favor and indulge this silly question: what did you use to produce the pinked-out screenshot markups? I’m looking to do that, a lot of that, but haven’t found a way that’s dead easy.
Would you email me with info?


From Ben Tremblay on January 31st, 2008 at 3:34 pm

Ben, I used Photoshop. I took a screenshot and in Photoshop I added another layer above the screenshot and used the marquee tool to define a selection that was the same size as the module below it. I filled it with red, and then made that layer 30% transparent, which made it pink and allowed the module to show through.

From Justin on February 2nd, 2008 at 8:35 pm

Thanks Justin. (HTML allowed?) I appreciate the mini-tutorial.

Allow me to seque here; when I want pains-taking, I do cloisonee. If there’s one reason why I’ve been alientated from the dev_community as a whole it’s because, well, when I needed to do MIL-SPEC docs I needed to do the stuff by the pound. Weekly. Month-in, month-out. So, force majeure I became a toolsmythe. When I see the effort you put in here (and thank you for it) I see just how/why I’m so completely side-lined.

That should be almost a drag/drop process. I can’t believe nobody has produced such a widget. Or, better, I can: we don’t have micro-payment, so those making tools do so on their free-time, coasting on whatever good jobs they have for the moment.

Such a nice thing to do, that sort of “markup”. Such a shame that it has be done as though making jewelry.

HeyHo … good stuff.

stay well

p.s. I wonder who in M$ wrote a long memo assessing Pipes’ value.

From Ben Tremblay on February 2nd, 2008 at 9:31 pm

There is a tool that makes this kind of graphic creation easier. For Windows there’s a tool called SnagIt, which allows you to capture a screen shot and then add elements like arrows and text. It is very convenient. I just started my career as a designer and prefer the full flexibility of Photoshop. :)

And, I worry that M$ isn’t going to understand Pipes and will destroy the project.

From Justin on February 3rd, 2008 at 1:22 pm

SnagIt, right, that rings a bell. *peeks TechSmith.com* Oh I was gonna say, That sort of little app would really fill things in” but it’s also a screen recorder; they’re the folk who make Camtasia!

I’ve mocked up screens using PowerPoint; SnagIt looks like the way to go.

I wonder if the kidz at google.code have any interest in acquiring Pipes.

From Ben Tremblay on February 3rd, 2008 at 1:48 pm

hi! I have a lot of feed that have just the title. then you need go to the site for read the news/feeds. But I have feeds with the whole news. So how can I do a new feed, using yahoo pipes, to do this,ie the whole news instead just the title

From Alexandre Monassa Moreira on July 31st, 2008 at 10:33 am

What say you about all of this?

Trackback URL Comment feed

You must be logged in to post a comment.

Clicky Web Analytics