Big IDEA

Shhh, I’m hunting wabbits

22 June 2005

Or, how to track topics across the Web

If you are studying a topic, you can save yourself from having to perform repeated Web searches by using a few freely-available tools. My basic recipe is:

1. search the news
2. search the blogosphere
3. aggregate RSS feeds from searches
4. make it public (optional)

For an example, I’m going to use the topic of government corruption since I know it will be difficult to find anything on that ;).

search the news

For news searches I prefer Yahoo! for the simple fact that it generates an RSS feed for my results. Visiting Yahoo! News, I search for government corruption. In the right column of my results page I see the orange XML button, so I know I can add this to my aggregator.

Yahoo! now offers Subscriptions Search (beta) for various publications like The Wall Street Journal, Consumer Reports, and Forrester Research. I’ll go ahead and search for government corruption here, too. The drawback here is that you need to pay for a subscription to read these articles, but at least I can find out if there’s anything I want to read.

After I search subscriptions I don’t see an orange button, but Firefox shows an orange icon in the lower-right corner so I know there is a feed for this site. I decide to view the page source and hit Control+f on the keyboard to find “link”. After a few stops I find the rss/xml link, which I can copy and add to my aggregator.

As you may know, Google News does not offer RSS feeds of search results. You can get around this, however, by signing up for a news alert. After searching for government corruption, I find the option to create an email alert for this search. Instead of entering my own email address, however, I’ll create one at MailBucket, which will let me access those alerts via RSS. I used the address govtcorrupt@mailbucket.org, but you can choose any name you want. I’ll have to check in at MailBucket to confirm the Google News alert but after that, it’s maintenance free.

update 6/27/05MailBucket seems to be down; however, dodgeit offers a similar service (also free).

OK, maybe Google News alert to MailBucket is a little too much work for you; a decent alternative is Topix.net, which will create an RSS feed for your search results.

search the blogosphere

In searching the blogosphere there are only a couple of sites I’ve found useful: Technorati and Feedster.

At Technorati, the first thing I do is search for government corruption. On the results page I have an option to add the search to a Watchlist, which is available via RSS (you’ll need to register with Technorati to do this). I add it to my Watchlist and I see a blue (blue?!) icon for the RSS feed.

Next I’ll search the Technorati tags–words that people use to classify their blog posts. I click the Search link at the top of the screen and then the options link. I want to search for posts tagged with corruption, and the results page again gives me the blue icon for the RSS feed.

Feedster is another tool for searching the blogosphere; simply execute the search and grab the RSS feed–hey, it’s orange!

N.B. As I scan the results for these searches, if I see the same blog popping up, chances are that’s a blog I’ll want to add to my aggregator.

searching other people’s bookmarks

Since I’m really interested in government corruption, I might want to see what sites other people are bookmarking. del.icio.us is a social bookmarking tool that allows people to tag their bookmarks (like Technorati). So, when I visit del.icio.us, I see a list of popular tags in the right column. I click the “See more tags” link and I can search for corruption. Lo and behold, at the bottom of the results page there’s an orange (!) icon for the RSS feed.

You can also try Furl for searching bookmarks. While I don’t find an easy way to browse topics, I noticed that you can use the URL http://www.furl.net/furled.jsp?topic=whatever and replace whatever with corruption, for example. The resulting page provides an RSS feed.

aggregate RSS feeds

This is really the easiest part; I just take the URL’s of the RSS feeds from my searches and add them to my aggregator.

make it public

One of the trickier things to do is share your feeds. Below are a few strategies for doing just that.

You can point people to your Bloglines (an Web-based aggregator) page, but people will see your other feeds, too.

If your aggregator supports it, you can export a folder of feeds to an OPML file, which allows people to import your collection of feeds to their own aggregator.

You can create a group page on Blogdigger, which would also allow other people to add feeds to your collection. See my government corruption group. Notice the OPML link on the right on the Blogdigger group page?

Create a “mix” using RSS Mix–similar to Blogdigger, but the number of items returned is really limited. See my mix.

If you have your own resources and know-how, you can aggregate several feeds together using PlanetPlanet. We recently started using PlanetPlanet to host some internal planets. See planet corruption.

As you can see, it doesn’t take much technical skill to track a topic on the Web. The important thing is to get a variety of sources so that you and your students can (hopefully) read diverse opinions to stimulate thought and discussion.

wabbit photo cwedit: http://flickr.com/photos/matana/20428527/