---
title: "On the destructive/constructive nature of news aggregation"
date: 2002-04-24T18:33:43-04:00
url: https://blog.lmorchard.com/2002/04/24/oooaai/
author: Les Orchard
---

# On the destructive/constructive nature of news aggregation

<p>Playing with a range of news aggregators once more, since <a href="http://radio.userland.com">Radio <a href="http://www.decafbad.com/twiki/bin/view/Main/UserLand">UserLand</a></a> is making me itchy again.  Pretty much the only ones I really like are Radio, <a href="http://www.disobey.com/amphetadesk">AmphetaDesk</a>, and sometimes <a href="http://www.oreillynet.com/~rael/lang/python/peerkat/">Peerkat</a>.</p>
<p>For the last week or so I've been mostly alternating between Radio and <a href="http://www.decafbad.com/twiki/bin/view/Main/AmphetaDesk">AmphetaDesk</a>.  The difference in aggregation styles is interesting:  While Radio slices and dices and aggregates feeds by items and serves them up to me in an interwoven chronological order, <a href="http://www.decafbad.com/twiki/bin/view/Main/AmphetaDesk">AmphetaDesk</a> serves my feeds up to me whole and in order of feed modification.  I'm not sure which I like more now.</p>
<p>I like Radio's style, because I see what's new and only what's new.  Usually.  But, I like <a href="http://www.decafbad.com/twiki/bin/view/Main/AmphetaDesk">AmphetaDesk</a>'s style because I see everything, and have realized that I miss things with the rush of new items from all my feeds.  For instance, if someone posts something once per day, I'll likely miss it with Radio unless I check every hour.  But, with <a href="http://www.decafbad.com/twiki/bin/view/Main/AmphetaDesk">AmphetaDesk</a>, I get to see what's new with every <i>person</i> or <i>source</i> whenever I check, and I only miss items if that feed has scrolled them off.</p>
<p><a href="http://www.burningbird.net/weblog/2002_03_01_burningbird_archive.php#75003900">Shelley Powers of burningbird.net</a> <a href="http://p2psmoke.org/distributed/rss.htm">wrote a bit about</a> the context destroying nature that <a href="http://www.decafbad.com/twiki/bin/view/Main/RSS">RSS</a> and aggregators have on weblogs.  I agree with her somewhat, in that pulling the words out of the surrounding context of the blog and its presentation and community has an altering effect, I wouldn't say that it destroys the weblog.</p>
<p>For me, when I see something on my news aggregator, it's as if I'm overhearing it from another room.  I don't get the whole context or see who else is listening or responding, but I hear the gist of something.  And, when subscribed to 100+ <a href="http://www.decafbad.com/twiki/bin/view/Main/RSS">RSS</a> feeds, it's like I'm floating in this Nth dimensional space where I can overhear voices from hundreds of rooms without being overwhelmed.  When something triggers some of my mental filters and watchwords, I click the link and delve deeper.</p>
<p>There's no way, without an aggregator, that I'd be able to track 100+ people and news sources in a day.  But, because I can, I've been able to learn and discover things and hear voices I never would have before.</p>
<p>But, I think the way <a href="http://www.decafbad.com/twiki/bin/view/Main/AmphetaDesk">AmphetaDesk</a> merges these sources might be a nicer alternative.  By not chopping up the feeds, some intra-item context is maintained at least, so I can see developing trains of thought.</p>
<p>Okay, must go back to work now.</p>
<!--more-->
shortname=oooaai

<div id="comments" class="comments archived-comments">
            <h3>Archived Comments</h3>
            
        <ul class="comments">
            
        <li class="comment" id="comment-221089999">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">Rael</a>
                </div>
                <a href="#comment-221089999" class="permalink"><time datetime="2002-04-24T19:21:30">2002-04-24T19:21:30</time></a>
            </div>
            <div class="content">Howdy,

If you like (in any way ;-) Peerkat, you might want to take a gander at the dynamic duo of Blosxom (Weblog application) and Blagg (RSS Aggregator).

Rael</div>
            
        </li>
    
        <li class="comment" id="comment-221090001">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.decafbad.com"><img src="http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.decafbad.com">l.m.orchard</a>
                </div>
                <a href="#comment-221090001" class="permalink"><time datetime="2002-04-24T19:44:13">2002-04-24T19:44:13</time></a>
            </div>
            <div class="content">Wow.  I've been playing a bit with blosxom, resisting the urge to do any tinkering that would complicate it.  It's like a weblog haiku on my iBook .  And Blagg looks pretty dang nifty too.  Must pla y with these more.  It even does the sort of thing I was thinking of with GroupWeblogWithRadioUserLand, only in a terribly less complicated way. :)

So the other thing I have to ask:  How does one become a researcher for a major publisher and get to play with things like this all day? :)</div>
            
        </li>
    
        <li class="comment" id="comment-221090003">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://kavery.ecs.fullerton.edu/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=132888da5dedf3b4903bddd7ad721e63&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://kavery.ecs.fullerton.edu/">Prof Avery</a>
                </div>
                <a href="#comment-221090003" class="permalink"><time datetime="2002-04-24T21:39:26">2002-04-24T21:39:26</time></a>
            </div>
            <div class="content">I've been putting off installing AmphetaDesk on my OS X machine at home, and now I suspect I won't bother -- Rael hit another one out of the park!

Me too, how do I get a job like this?

P.S.  You might find my XML-RPC interface for Blosxom entertaining.</div>
            
        </li>
    
        <li class="comment" id="comment-221090005">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.decafbad.com"><img src="http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.decafbad.com">l.m.orchard</a>
                </div>
                <a href="#comment-221090005" class="permalink"><time datetime="2002-04-24T21:52:43">2002-04-24T21:52:43</time></a>
            </div>
            <div class="content">I just went nuts and via a lil regex mangling, threw my whole Radio UserLand subscriptions list at Blagg.  Let's see how much I can torture it :)</div>
            
        </li>
    
        <li class="comment" id="comment-221090007">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">Rael</a>
                </div>
                <a href="#comment-221090007" class="permalink"><time datetime="2002-04-25T06:14:34">2002-04-25T06:14:34</time></a>
            </div>
            <div class="content">> Let's see how much I can torture it :)

So nu, l.m., how did it go?

> and get to play with things like this all day? :)

Make that all evening and you've your answer ;-)</div>
            
        </li>
    
        <li class="comment" id="comment-221090010">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">Rael</a>
                </div>
                <a href="#comment-221090010" class="permalink"><time datetime="2002-04-25T06:21:04">2002-04-25T06:21:04</time></a>
            </div>
            <div class="content">Fresh from the oven: you might enjoy Blaggplugs, plug-ins for Blagg hooking it into the Blogger API, Movable Type, et al.</div>
            
        </li>
    
        <li class="comment" id="comment-221090012">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">Rael</a>
                </div>
                <a href="#comment-221090012" class="permalink"><time datetime="2002-04-25T06:40:08">2002-04-25T06:40:08</time></a>
            </div>
            <div class="content">Prof: your BlosXMLRPC is a hoot!</div>
            
        </li>
    
        <li class="comment" id="comment-221090014">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.decafbad.com"><img src="http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.decafbad.com">l.m.orchard</a>
                </div>
                <a href="#comment-221090014" class="permalink"><time datetime="2002-04-25T12:39:40">2002-04-25T12:39:40</time></a>
            </div>
            <div class="content">Ahh, blagg.  Another haiku of code :)

Torturing blagg with my pile of subscriptions seems to have gone alright.  I need to tweak around with the templates for blosxom (Blog Some?) to make it more pleasant to visually scan.  Also need to tweak the max # of items on the page.  Maybe some sort of "Next/Prev 20 items" pagination would be nice.

Also, templatizing how blagg constructs posts from RSS items would be hot, too.

First snag I see though, which you probably already know about, is the way blagg determines whether or not an item is new.  Blagg, of course, uses the title.  However, some of my feeds' items never have titles (ie. Radio UserLand peeps who haven't turned on titles), and others often have the same titles (ie. RSS feeds from wikis).

I've been trying to figure out the best way to do this in general, and all my solutions keep ending up at a bruteforce MD5 of the whole item.  Of course, the problem in doing that in blagg is that MD5 doesn't come out of the box with perl in OS X, which kills the whole drag-n-drop installation theme.  (Nevermind that MD5ing everything might be a tad slow.)

Hrm.  Still playing.</div>
            
        </li>
    
        <li class="comment" id="comment-221090016">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.decafbad.com"><img src="http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.decafbad.com">l.m.orchard</a>
                </div>
                <a href="#comment-221090016" class="permalink"><time datetime="2002-04-25T12:45:12">2002-04-25T12:45:12</time></a>
            </div>
            <div class="content">Oh yeah, and one other blagg thing:  I see the Blaggplugs.  Keen!  Something that would be nice, though, is if the RSS feed source could be separated from the post title and passed as a separate parameter to the post() sub of the plugin.  Then I could even do something like transform the source attribute into a category or something, should I make a metaWeblog API plugin.  Just supposing I did, that is.</div>
            
        </li>
    
        <li class="comment" id="comment-221090018">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.disobey.com/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=7c86a3bdffadc965eb444f588c24a3f3&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.disobey.com/">Morbus Iff</a>
                </div>
                <a href="#comment-221090018" class="permalink"><time datetime="2002-04-25T20:45:18">2002-04-25T20:45:18</time></a>
            </div>
            <div class="content">> blagg determines whether or not an item is new

Hmm. Yeah, MD5 would be a little slow. Perhaps the filename could include a character length count, something like "raelity.617.title". That should cover Wiki changes (as a same count edit would be rare, I'd hope). That'd cover RU a bit too.

Speaking of which, I just sent Rael a Blaggplug that pings weblogs.com with an update of your blog. I've got another one planned for either today or tomorrow as well.</div>
            
        </li>
    
        <li class="comment" id="comment-221090020">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">Rael</a>
                </div>
                <a href="#comment-221090020" class="permalink"><time datetime="2002-04-26T17:18:24">2002-04-26T17:18:24</time></a>
            </div>
            <div class="content">Another about-unique index I was toying with was to select the xth, yth, and zth character of the description and include that in the filename.

What are the chances you'd get the same title from the same feed with the same 3 arbitrary characters in the same 3 arbitrary spots?

What do you think?</div>
            
        </li>
    
        <li class="comment" id="comment-221090022">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.disobey.com/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=7c86a3bdffadc965eb444f588c24a3f3&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.disobey.com/">Morbus Iff</a>
                </div>
                <a href="#comment-221090022" class="permalink"><time datetime="2002-04-26T18:26:50">2002-04-26T18:26:50</time></a>
            </div>
            <div class="content">Seems suitably random to me.</div>
            
        </li>
    
        <li class="comment" id="comment-221090024">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.decafbad.com"><img src="http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.decafbad.com">l.m.orchard</a>
                </div>
                <a href="#comment-221090024" class="permalink"><time datetime="2002-04-26T18:54:29">2002-04-26T18:54:29</time></a>
            </div>
            <div class="content">Hmm...  You might still be in trouble with a wiki RSS feed like the Meatball wiki RSS feed or the MoinMoin RSS feed where edits are minor and the page is left largely unchanged.  But, it sounds like a better step.</div>
            
        </li>
    
        <li class="comment" id="comment-221090026">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">rael</a>
                </div>
                <a href="#comment-221090026" class="permalink"><time datetime="2002-04-27T05:05:12">2002-04-27T05:05:12</time></a>
            </div>
            <div class="content">New email and IM Blaggplugs -- get 'em while they're hot!</div>
            
        </li>
    
        <li class="comment" id="comment-221090028">
            <div class="meta">
                <div class="author">
                    <a class="avatar image" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/"><img src="http://www.gravatar.com/avatar.php?gravatar_id=e5f046c5b46cfb3ef1089da58b2a48bb&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png"/></a>
                    <a class="avatar name" rel="nofollow" 
                       href="http://www.oreillynet.com/~rael/">rael</a>
                </div>
                <a href="#comment-221090028" class="permalink"><time datetime="2002-04-29T21:14:21">2002-04-29T21:14:21</time></a>
            </div>
            <div class="content">Woohoo!  I'm now entirely self-hosted (read: Blosxom and Blagg)!</div>
            
        </li>
    
        </ul>
    
        </div>
