{ "attachments": [], "comments_archived": true, "date": "2004-06-14T22:54:55.000Z", "excerpt": "Where's the state-of-the-art for feed aggregators, and what's next? I'm tired of reverse-chronological versus three-pane; I'm tired of copying Usenet and email. What needs to happen next to expand our Info Freako capacity by an order of magnitude or two? The invention of aggregators has opened the door to the first few orders, but I need more.", "layout": "post", "tags": [ "syndication" ], "title": "Info Freako, or who's already past arguing about syndication formats?", "wordpress_id": 528, "wordpress_slug": "info-freako-or-whos-already-past-arguing-about-syndication-formats", "wordpress_url": "http://www.decafbad.com/blog/?p=528", "year": "2004", "month": "06", "day": "14", "isDir": true, "slug": "info-freako-or-whos-already-past-arguing-about-syndication-formats", "type": "entry", "postName": "2004-06-14-info-freako-or-whos-already-past-arguing-about-syndication-formats", "html": "
\n

You know that sleep is getting hard to get

\n
\n
\n

'Cause you never know what you'll forget

\n
\n
\n

And I've got to know of all the news

\n
\n
\n

'Cause one day there'll be news for me

\n
\n
\n
\n
\n

I never let a headline by

\n
\n
\n

'Cause every one will catch my eye

\n
\n
\n

And though it's tough to keep alert

\n
\n
\n

You never know what could hurt me

\n
\n
Source: Jesus Jones, \"Info Freako\"\n\n
\n\n\n\n

The Syndications Wars are over--at least, as far as I'm concerned.

\n

It's hard to resist jumping in where I think someone's got it wrong or when my fingers compel me to feed trolls, but resisting that impulse is what needs to be done. At this point, all that seems to happen is that the same old threads get recycled. No one's got anything new to bring, except maybe ad hominem grousing or possibly a Yo Mama joke (though I've yet to see that particular innovation). Anyone who cares can do a bit of Googling to catch up on the story.

\n

The fact is, I no longer care about RSS versus Atom versus $foo. Mark Pilgrim enables me to do so with his feed parser, and most other aggregators I might care about have also implemented support for both Atom and RSS. And if they don't, I can route around the damage just like I do when I scrape sites devoid of any feeds.

\n

While I do prefer Atom over RSS, almost a year later I still say the magic is in syndication, not the format. I'll let the tag-level grumblers foam on without comment and just thank them for their work when a good new spec bubbles up or when something fun and useful comes out. I'm circling that whole area of concern and sticking a post-it on it that reads: RSS and Atom both useful, neither perfect, neither going away.

\n

Whew. That's a weight off my brain. Now what is there left to talk about?

\n

Well, how about let's talk some more about what to do with the items we get, once we do manage to parse a feed? (Sheesh, you mean we're not already fighting over that topic?)

\n

Rogers Cadenhead wants gluttonous RSS feeders:

\n
\n

With thousands of information sources producing RSS and Atom feeds, we need people like Thauvin [whose linkblog is here] who have integrated weblogging into their daily news-gathering routine. Weblog links are like ant trails -- a lot of people have to link to something good in order to get noticed.

\n
\n

I self-identify as such, since my feed list has topped 550 in count. But I'm happily surprised to find that I'm not even in the top 10 of prolific subscribers--at least I'm not the biggest Info Freako. (Yet.)

\n

I'm adding between 2-3 feeds to my list daily, so I can see myself approaching 1000 eventually. But I'm starting to hesitate at adding one more feed now. Even with my current streamlined multi-pass skimming process, I'm starting to see diminishing returns. I breeze past screen loads of chaff that I'll never view, but it still bogs me down. I can only think that people with twice the subscriptions as I either have more free time, or have a better mousetrap.

\n

The usual response I get toward my subscriptions is, \"Why don't you cut that list down to about 100 essentials?\" And even that's said with a smirk, usually by someone with under 50 subscriptions and usually by someone who's not as obsessive an Info Freako as I am. Thing is, though, good stuff has at one point or another shown up on each and every feed I monitor. I want to figure out how to scale up from 1 to 10 to 100 to 1000 to 10000 sources and beyond.

\n

(Singing interlude: \"Info Freako / There is no end to what I want to know\")

\n

Besides, this is an area where I can tinker with and learn about another area I'm interested in: machine intelligence and intelligence amplification. Rogers says, \"I want a Bayesian filter that can guess which new headlines I'm most likely to read\" Though someone else might apply Bayes in a way that works for them, I didn't find my experiments with SpamBayes very satisfying. I suspect it has something to do with the fact that SpamBayes is geared toward sorting out a quasi-binary world of spam-versus-ham, while I'm interested in a spectrum between must-read and shrugs.

\n

But, the idea of introducing another pass through items at the head of the process, this one partially or completely automated, has great appeal. Done right, this could be the bit that adds an order of magnitude to my capacity to monitor feeds. I need to investigate other machine learning approaches.

\n

The idea is that, while I freakishly want to catch as much info as possible, I can only handle so much in a day. For certain, I can't handle everything that might be interesting to me, so I need some prioritization and some pre-filtering before my attention gets applied to the flow.

\n

The way I picture this is trying to apply a sort of inverted pyramid approach to the incoming flow of items.

\n

I started with a few primitive tools in AmphetaOutlines' adaptivity to reading patterns, limited mostly to just sorting channels by a count of items read historically. I also introduced some information hiding and exploration aspects: I tried to hide or de-emphasize older items by use of font size and weight; and I put items into a JavaScript-driven outline where item descriptions and more ancient items could be hidden or revealed via disclosure triangle.

\n

In my latest attempt, I've not yet implemented any adaptive sorting, but I've kept and improved the outline display (see: screenshot #1, screenshot #2). Also, I can now mark items as seen and/or flag them to be viewed in a queue for later. I've got some lame SpamBayes integration in there, but I've let it atrophy in daily use due to a complete lack of usefulness.

\n

I'm starting to think about next steps now toward a more advanced aggregator. I've still got my wishlist for AmphetaOutlines, and I've actually covered quite a few of the items with this new aggregator. But, I'm thinking things like the following would be useful to pursue:

\n\n

Now, I haven't done any sort of comprehensive survey of the aggregator landscape in a long time, so I'd be very intrigued if any existing software implements these sorts of things. I've seen some progress toward monitoring feed health, but I've seen next to nothing toward automatic filtering of items and recommendations based on past behavior. I have seen manually constructed filters, but I'm too lazy to try to figure out how to tell the computer what I want. I want the machine to ride shotgun, watch and learn.

\n

\"Need Where's the state-of-the-art for feed aggregators, and what's next? I'm tired of reverse-chronological versus three-pane; I'm tired of copying Usenet and email. What needs to happen next to expand our Info Freako capacity by an order of magnitude or two? The invention of aggregators has opened the door to the first few orders, but I need more.

\n

Exit singing (while twitching for more info):

\n
\n

Info Freako,

\n
\n
\n

There is no end to what I want to know

\n
\n
\n
\n
\n

But it means I'll have the edge over you

\n
\n
\n

And it means I'll always have the edge over you

\n
\n
\n

And you know there's nothing that you can do

\n
\n

Archived Comments

\n\n\n\n
\n\n\n", "body": "> You know that sleep is getting hard to get
\r\n> 'Cause you never know what you'll forget
\r\n> And I've got to know of all the news
\r\n> 'Cause one day there'll be news for me
\r\n>
\r\n> I never let a headline by
\r\n> 'Cause every one will catch my eye
\r\n> And though it's tough to keep alert
\r\n> You never know what could hurt me
\r\n\r\n
Source: Jesus Jones, \"Info Freako\"
\r\n\r\nThe Syndications Wars are over--at least, as far as I'm concerned. \r\n\r\nIt's hard to resist [jumping in][jumping in] where I think someone's got it wrong or when my fingers compel me to feed trolls, but resisting that impulse is what needs to be done. At this point, all that seems to happen is that the same old threads get recycled. No one's got anything new to bring, except maybe ad hominem grousing or possibly a Yo Mama joke (though I've yet to see that particular innovation). Anyone who cares can do a bit of Googling to catch up on the story.\r\n\r\nThe fact is, I no longer care about RSS versus Atom versus $foo. Mark Pilgrim enables me to do so with his [feed parser][feedparser], and most other aggregators I might care about have also implemented support for both Atom and RSS. And if they don't, I can [route around the damage][rss colored glasses] just like I do when I [scrape sites][scrape sites] devoid of any feeds.\r\n\r\nWhile I do prefer Atom over RSS, almost a year later I still say [the magic is in syndication, not the format][syndication magic]. I'll let the tag-level grumblers foam on without comment and just thank them for their work when a good new spec bubbles up or when something fun and useful comes out. I'm circling that whole area of concern and sticking a post-it on it that reads: *RSS and Atom both useful, neither perfect, neither going away.*\r\n\r\nWhew. That's a weight off my brain. Now what is there left to talk about?\r\n\r\nWell, how about let's talk some more about what to do with the items we get, once we *do* manage to parse a feed? (Sheesh, you mean we're not already fighting over that topic?) \r\n\r\n[Rogers Cadenhead wants gluttonous RSS feeders][cadenhead]:\r\n\r\n> With thousands of information sources producing RSS and Atom feeds, we need people like Thauvin [whose linkblog is [here][thauvin]] who have integrated weblogging into their daily news-gathering routine. Weblog links are like ant trails -- a lot of people have to link to something good in order to get noticed.\r\n\r\nI self-identify as such, since my feed list has topped 550 in count. But I'm happily surprised to find that I'm not even in the [top 10 of prolific subscribers][prolific subs]--at least I'm not the biggest Info Freako. (Yet.) \r\n\r\nI'm adding between 2-3 feeds to my list daily, so I can see myself approaching 1000 eventually. But I'm starting to hesitate at adding one more feed now. Even with my current streamlined multi-pass skimming process, I'm starting to see diminishing returns. I breeze past screen loads of chaff that I'll never view, but it still bogs me down. I can only think that people with twice the subscriptions as I either have more free time, or have a better mousetrap.\r\n\r\nThe usual response I get toward my subscriptions is, \"Why don't you cut that list down to about 100 essentials?\" And even that's said with a smirk, usually by someone with under 50 subscriptions and usually by someone who's not as obsessive an [Info Freako][info freako] as I am. Thing is, though, good stuff has at one point or another shown up on each and every feed I monitor. I want to figure out how to scale *up* from 1 to 10 to 100 to 1000 to 10000 sources and beyond.\r\n\r\n(Singing interlude: \"Info Freako / There is no end to what I want to know\")\r\n\r\nBesides, this is an area where I can tinker with and learn about another area I'm interested in: machine intelligence and intelligence amplification. Rogers says, \"I want a Bayesian filter that can guess which new headlines I'm most likely to read\" Though someone else might apply Bayes in a way that works for them, I didn't find [my experiments with SpamBayes][bayes] very satisfying. I suspect it has something to do with the fact that SpamBayes is geared toward sorting out a quasi-binary world of spam-versus-ham, while I'm interested in a spectrum between must-read and shrugs.\r\n\r\nBut, the idea of introducing another pass through items at the head of the process, this one partially or completely automated, has great appeal. Done right, this could be the bit that adds an order of magnitude to my capacity to monitor feeds. I need to investigate other machine learning approaches.\r\n\r\nThe idea is that, while I freakishly want to catch as much info as possible, I can only handle so much in a day. For certain, I can't handle everything that might be interesting to me, so I need some prioritization and some pre-filtering before my attention gets applied to the flow.\r\n\r\nThe way I picture this is trying to apply a sort of [inverted pyramid][inverted pyramid] approach to the incoming flow of items.\r\n\r\nI started with a few primitive tools in [AmphetaOutlines' adaptivity to reading patterns][amphetaoutlines], limited mostly to just sorting channels by a count of items read historically. I also introduced some information hiding and exploration aspects: I tried to hide or de-emphasize older items by use of font size and weight; and I put items into a JavaScript-driven outline where item descriptions and more ancient items could be hidden or revealed via disclosure triangle.\r\n\r\nIn my [latest attempt][dbagg2], I've not yet implemented any adaptive sorting, but I've kept and improved the outline display (see: [screenshot #1][dbagg2 screen1], [screenshot #2][dbagg2 screen2]). Also, I can now mark items as seen and/or flag them to be viewed in a queue for later. I've got some lame SpamBayes integration in there, but I've let it atrophy in daily use due to a complete lack of usefulness.\r\n\r\nI'm starting to think about next steps now toward a more advanced aggregator. I've still got my [wishlist][wishlist] for AmphetaOutlines, and I've actually covered quite a few of the items with this new aggregator. But, I'm thinking things like the following would be useful to pursue:\r\n\r\n* **What do you think is more important?** Do you value one group of feeds over another? Personally, I want to see every single web comic that appears in my queue, most items from [Engadget][engadget] and [Boing Boing][boing boing], and maybe only a few from some of the firehoses I've hooked myself up to. Also, there are [some][blogger1] [bloggers][blogger2] who post somewhat infrequently, but I don't want to miss a thing when they *do* post. I need to be able to group and prioritize manually.\r\n\r\n* **What do you *demonstrate* as important?** Which feeds' items receive more of your attention, and within those feeds, what topics and phrases appear most frequently? The machine should be able to make some observations about your history of behavior and give some input into the organization of items presented. Also, it should give me some way to give feedback to its recommendations with a simple and lazy thumbs up and thumbs down.\r\n\r\n* **Republishing of interesting items to a linkblog is a must.** On the flip-side, it would be nice to somehow pull in others' linkblogs in a more meaningful way than simply watching their feeds. I should be able to triangulate some things and get some recommendations based on mutual links predicting future interest in items. We need to start chasing ant trails unconsciously and automatically.\r\n\r\n* **Time-limited subscriptions which expire after a set time, or request renewal from the user.** Use these to track comment threads which offer RSS feeds. (Like [this one][comment feed].)\r\n\r\n* **More statistics and health monitoring of subscriptions.** How active are your feeds? Which are dead & gone, or merely just in hiatus? Have any moved?\r\n\r\nNow, I haven't done any sort of comprehensive survey of the aggregator landscape in a long time, so I'd be very intrigued if any existing software implements these sorts of things. I've seen some progress toward monitoring feed health, but I've seen next to nothing toward automatic filtering of items and recommendations based on past behavior. I have seen manually constructed filters, but I'm too lazy to try to figure out how to tell the computer what I want. I want the machine to ride shotgun, watch and learn.\r\n\r\n\"Need Where's the state-of-the-art for feed aggregators, and what's next? I'm tired of reverse-chronological versus three-pane; I'm tired of copying Usenet and email. What needs to happen next to expand our Info Freako capacity by an order of magnitude or two? The invention of aggregators has opened the door to the first few orders, but I need more. \r\n\r\nExit singing (while twitching for more info):\r\n\r\n> Info Freako,
\r\n> There is no end to what I want to know
\r\n>
\r\n> But it means I'll have the edge over you
\r\n> And it means I'll always have the edge over you
\r\n> And you know there's nothing that you can do
\r\n\r\n[jumping in]: http://www.reactuate.com/index.php?itemid=682\r\n[blogger1]: http://interconnected.org/home/\r\n[blogger2]: http://www.ecyrd.com/ButtUgly/Wiki.jsp?page=Main_blogentry_130604_1\r\n[engadget]: http://www.engadget.com/\r\n[boing boing]: http://boingboing.net/\r\n[inverted pyramid]: http://mtsu32.mtsu.edu:11178/171/pyramid.htm\r\n[comment feed]: http://www.pycs.net/system/comments.py?u=0000001&p=1802&format=rss\r\n[wishlist]: http://www.decafbad.com/twiki/bin/view/Main/AmphetaOutlinesWishList\r\n[dbagg2 screen1]: http://www.decafbad.com/2004/06/dbagg2a.jpg\r\n[dbagg2 screen2]: http://www.decafbad.com/2004/06/dbagg2b.jpg\r\n[dbagg2]: http://www.decafbad.com/cvs/dbagg2\r\n[amphetaoutlines]: http://www.decafbad.com/blog/2002/08/05/ooobbf\r\n[bayes]: http://www.decafbad.com/blog/2003/08/16/bayes_agg_one\r\n[info freako]: http://www.jesusjonesarchive.com/lyrics.htm#Info%20Freako \"Jesus Jones lyrics\"\r\n[thauvin]: http://www.thauvin.net/linkblog/\r\n[prolific subs]: http://feeds.scripting.com/prolificSubscribers\r\n[cadenhead]: http://www.cadenhead.org/workbench/2004/06/10.html#a1802\r\n[syndication magic]: http://www.decafbad.com/blog/2003/07/07/syndications_formats\r\n[scrape sites]: http://www.decafbad.com/twiki/bin/view/Main/XslScraper\r\n[rss colored glasses]: http://www.decafbad.com/blog/2004/05/03/put_on_your_rsscolored_glasses_and_forget_about_atom\r\n[feedparser]: http://diveintomark.org/projects/feed_parser/\r\n\r\n
\r\n

Archived Comments

\r\n \r\n \r\n \r\n
\r\n ", "parentPath": "./content/posts/archives/2004/2004-06-14-info-freako-or-whos-already-past-arguing-about-syndication-formats", "path": "2004/06/14/info-freako-or-whos-already-past-arguing-about-syndication-formats", "thumbnail": "http://www.johnny-five.com/simplenet/Shortcircuit/Pics/Pictures/Misc/jfive.gif", "needsBuild": true, "prevPostPath": "2004/06/13/i-will-do-the-fandango", "prevPostTitle": "I will do the Fandango.", "nextPostPath": "2004/06/16/wishofthemonthclub1", "nextPostTitle": "Wish-of-the-Month Club, Part 1 of 3" }