Yahoo's new Pipes service may be for you.
Still in the early stages of beta, the "interactive feed aggregator and manipulator" lets ordinary people -- even Computerworld editors -- customize RSS feeds and build mashups from different sources.
It was so popular upon first being introduced Feb. 7 that it Yahoo's servers the next day. Visitors were greeted with the message: "Our Pipes are clogged! We've called the plumbers!"
After a good flushing, the service has thrived, with thousands of new Pipes being built, discussed, shared and improved among an active user community.
Just drag and drop
Using a simple drag-and-drop system, users choose modules and drag them to an editor where they are configured and connected through flexible, winding ... well, Pipes.
So, for instance, you can drag out a "Fetch" module that simply includes a text input box where you specify the URL of an RSS feed that you want to use. Doing this automatically creates a "Pipe Output" module. You can click on a button on the bottom of the Fetch module that generates a flexible Pipe that follows your cursor. Connect the Pipe to the Output module and you have an instant application in just a few minutes. You can then "publish" the Pipe so it is accessible by others or keep it for your own use at a specific URL or as an RSS feed, for instance.
While that is the simplest example and not very useful, it's easy to add more feeds, customize their presentation and combine information from disparate sources into one application. Some of the more interesting Pipes have been built using modules for user input, operator functions such as sort and count, date formatting, and built-in sources such as Flickr and Google Base and, of course, Yahoo search.
For example, one Pipe called "Apartment Near Something" combines Craigslist and Yahoo Local to generate "searches for apartments near things you care about!"
Another one called "New York Times thru Flickr" takes The New York Times home page feed, passes it through a Content Analysis module that extracts keywords to find related photos at the Flickr site.
One called "eBay Price Watch" is described as "designed to use eBay's RSS API to find items within a certain price range."
Yahoo says it's seeking user input to improve the service, and that plans already in place for future versions include:
-- Programmatic access to the Pipes engine
-- Support for additional data sources (such as Keyhole Markup Language)
-- More built-in processing modules
-- The ability to extend Pipes with external, user-contributed modules
-- More ways to render output (Badges, Maps)
Building a Pipe
To find out exactly how it works and if it was really as easy as advertised, I decided to build my own Pipe. Since I'm the networking editor at Computerworld.com, I built a Pipe that collects networking news from various sources and combines them into one feed. Here's how I went about it.
Clicking on "Create A New Pipe" brings up the editor, with a list of available modules on the left and the editing area on the right, sporting the helpful directions to "drag modules here."
The modules are arranged in these categories:
-- Sources
-- User inputs
-- Operators
-- URL
-- String
-- Date
-- My Pipes
I choose the Fetch module from Sources and drag it into the editing area. There I enter Computerworld's Networking RSS feed URL (http://feeds.computerworld.com/Computerworld/
Networking/News) into the text input field and click a + button to open up two more fields. I enter the URLs for feeds from Network World and InfoWorld into those fields.
To test if everything is working so far, I simply click on the Fetch module and its input is displayed in a Debugger field at the bottom of the editor. The Debugger shows that headlines are indeed being generated by the Fetch module.
I click on a round button on the bottom of the Fetch module and drag the flexible Pipe that is instantly generated down to the Pipe Output module and I have a working Pipe, combining RSS feeds from three publications.
To add more news, I decide to add search results from Yahoo and Google.
From the URL category, I drag in a URLBuilder module. Hovering the mouse over the ? symbol in this module pops up a message that informs me "This module builds URLs in either traditional or Web 2.0 style query-string format from a series of input fields."
One of the cool features of this module is that it lets you generate a customized search in Yahoo News, for example, and take the RSS feed from the results of the search and paste it into the module's Base text input field. The module then parses out all the key/value pairs from the query string of the URL generated by the search.
So a search of Yahoo News articles generated in the last hour in the Technology section with the word "network" in the headline -- in RSS format -- results in a convoluted URL like this:
http://news.search.yahoo.com/news/rss?ei=
UTF-8&_bcrumb=4716ffcd3093870630ea9271
301b4281%2C1171571208&va=&va_vt=any
&vp=&vp_vt=any&vo=network&vo_vt=title&ve=
&ve_vt=any&datesort=&pub=&timeago=hour
&smonth=1&sday=16&emonth=2&eday=15
&source=&location=&catfilt=1&cat=technology
&fl=0&n=10&eo=UTF-8
I paste that URL into the Base field of the URLBuilder module and it parses out the key/value pairs like this:
I do a similar search on Google News and paste its URL into a new URLBuilder.
From these two modules, I drag Pipes to the Fetch module, after clicking on its + button to open up two more input fields.
Clicking on the Pipe Output module (or hovering over the Debugger tab at the bottom and clicking on Pipe Output from the resulting pop-up list) shows me in the Debugger field that the appropriate headlines are being generated.
My Pipe now combines RSS feeds from Computerworld, Network World and InfoWorld, along with customized news searches from Yahoo and Google.
To add a little further customization, I drag out a TextInput module from the User inputs category. As the name suggests, it collects text string input from users. Other available modules allow the input of URLs, locations or dates.
The Yahoo and Google searches looked for articles with the word "network" in the headline. To customize the Pipe and let me search for other terms instead, I configure the TextInput module so it provides a prompt to the user telling them they can change the default search term.
I then connect a Pipe from this module to input fields in the URLBuilder modules where they have parsed out the search term used in the Yahoo and Google searches.
The final product is done, in less than an hour (well, it would have been if I wasn't so obtuse and curious). You can look at it and run it here. (And please ignore the "old man" avatar -- it's my 10-year-old son's doing and not what I really look like. I just haven't figured out how to change it to something else.)
Endless possibilities
Theoretically, I could add other input fields to change the time searched from one hour to a day, for example, or change the Yahoo section searched from Technology to something else. Or look for the search term in the body of the Google articles instead of the headline. Using all the other modules and ideas from people smarter than me, the possibilities are endless.
The beta version does exhibit some strange behavior sometimes, popping up error messages upon actions that were completed successfully in the same exact manner earlier (and later), for example.
But it's easy to work through the glitches and play around with your project. It's also easy to find another Pipe that interests you and tinker with its components to customize it for your own uses or use it to build a brand-new Pipe.
The site lists the Pipes that have been run the most times or cloned the most times. It also features an active Message Boards area.
Some people have called Pipes the future of programming. I don't know about that, but it's certainly easy and fun. Go on, give it a try.