{
"attachments": [],
"comments_archived": true,
"date": "2002-10-11T18:16:20.000Z",
"layout": "post",
"title": "It's all about the conversation, not the referrer logs.",
"wordpress_id": 281,
"wordpress_slug": "ooocof",
"wordpress_url": "http://www.decafbad.com/blog/?p=281",
"year": "2002",
"month": "10",
"day": "11",
"isDir": false,
"slug": "ooocof",
"type": "entry",
"postName": "2002-10-11-ooocof",
"html": "Mark Pilgrim implements something I've been thinking about for awhile: His \"Further Reading Upgrades\" now harvest what appears to be the paragraph surrounding a link on a referring page. Along with the RSS feed of \"Further Reading\" items he's made available, he's got a nice game of follow-the-leader set up for the rest of us referrer log watchers. Nice & elegant & makes me ashamed. :)
\nUpdate: Mark pulls back the curtain and reveals the secret. Yay!
\nYet another demonstration to me that ideas are f'ing worthless and the making is what matters. Lots of things I've been thinking about doing, but never get done. (See also: Blog:000305) Just a few weeks ago, I managed to upgrade my referrer tracking to dig out the titles of referring pages. And I've had an RSS feed of referrers myself for a few months now, but mine's been ugly as sin and so I've kept it to myself. So now I'll need to think in earnest about how to do some extraction of the link-surrounding excerpt in referring pages. Mark's referrer handling really is elegant - it even seems to know how to collapse multiple views on the same referring entry (ie. front page, archive page, individual entry page).
\nMore things to play with :)
\n\n\n\nshortname=ooocof
\n\n\n\n",
"body": "Mark Pilgrim implements something I've been thinking about for awhile: His \"Further Reading Upgrades\" now harvest what appears to be the paragraph surrounding a link on a referring page. Along with the RSS feed of \"Further Reading\" items he's made available, he's got a nice game of follow-the-leader set up for the rest of us referrer log watchers. Nice & elegant & makes me ashamed. :)\r\n
\r\nUpdate: Mark pulls back the curtain and reveals the secret. Yay!\r\n
\r\nYet another demonstration to me that ideas are f'ing worthless and the making is what matters. Lots of things I've been thinking about doing, but never get done. (See also: Blog:000305) Just a few weeks ago, I managed to upgrade my referrer tracking to dig out the titles of referring pages. And I've had an RSS feed of referrers myself for a few months now, but mine's been ugly as sin and so I've kept it to myself. So now I'll need to think in earnest about how to do some extraction of the link-surrounding excerpt in referring pages. Mark's referrer handling really is elegant - it even seems to know how to collapse multiple views on the same referring entry (ie. front page, archive page, individual entry page).\r\n
\r\nMore things to play with :)\r\n\r\nshortname=ooocof\r\n\r\n\r\n ",
"parentPath": "./content/posts/archives/2002",
"path": "2002/10/11/ooocof",
"summary": "Mark Pilgrim implements something I've been thinking about for awhile: His \"Further Reading Upgrades\" now harvest what appears to be the paragraph surrounding a link on a referring page. Along with the RSS feed of \"Further Reading\" items he's made available, he's got a nice game of follow-the-leader set up for the rest of us referrer log watchers. Nice & elegant & makes me ashamed. :)\nUpdate: Mark pulls back the curtain and reveals the secret. Yay!\nYet another demonstration to me that ideas are f'ing worthless and the making is what matters. Lots of things I've been thinking about doing, but never get done. (See also: Blog:000305) Just a few weeks ago, I managed to upgrade my referrer tracking to dig out the titles of referring pages. And I've had an RSS feed of referrers myself for a few months now, but mine's been ugly as sin and so I've kept it to myself. So now I'll need to think in earnest about how to do some extraction of the link-surrounding excerpt in referring pages. Mark's referrer handling really is elegant - it even seems to know how to collapse multiple views on the same referring entry (ie. front page, archive page, individual entry page).\nMore things to play with :)",
"needsBuild": true,
"prevPostPath": "2002/10/10/ooocoe",
"prevPostTitle": "A breakthrough in outline transclusion!",
"nextPostPath": "2002/10/15/ooocog",
"nextPostTitle": "Microsoft and Clip Art Follies"
} ![](\"http://www.gravatar.com/avatar.php?gravatar_id=2aac8a68f46151f080ddb17ee0d2454e&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\"/)
\r\n Mark Pilgrim\r\n Actually, it's even better than that. I have a list of HTML tags that act as dividers: P, BR, HR, PRE, DIV, SPAN, TD, and a few others. I grab the text surrounding the link, demarcated by any combination of these dividers.\r\n\r\nThere's some other wicked cool stuff in there too, like trying to guess permalinks (and getting it right about 80% of the time, and avoiding false positives the other 20%). First it looks for trackback data; if found, that's golden. Otherwise it looks for the last anchor (name or id attribute in any tag, not just A) before the start of the excerpt, then looks for the first link after the excerpt that contains that anchor. This picks up the permalink on most blogs, since they include anchor tags before the entry named the same as the permalink (either the filename or the fragment).\r\n\r\nA concrete example (from your own home page): this post is preceded by an anchor \"\"\"[a name=\"000306\"][/a]\"\"\", and followed by a permalink \"\"\"[a href=\"http://www.decafbad.com/news_archives/000306.phtml\"]11:16 AM[/a]\"\"\". See how they match?\r\n\r\nIt's not just A tags, though; some anchors are just an id attribute stuck on a div or h3 tag. There are lots of variations. Life is messy.\r\n \r\n \r\n \r\n ![](\"http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\"/)
\r\n l.m.orchard\r\n I just noticed the tag detection as I looked at the sample code you posted. Much more sophisticated than I'd initially thought. :) I forgot that Python has a useful SGML parser - I need to play with that more. Usually I assume that I'm stuck in trial-and-error regex-land when I want to dig out some content from a given HTML page.\r\n\r\nAs for the permalink detection, yes I'd found some false positives from the start, but I was amazed at how many non-obvious ones it found. Very cool!\r\n \r\n \r\n \r\n ![](\"http://www.gravatar.com/avatar.php?gravatar_id=def808e6072af95b27c6d298fbb05a2b&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\"/)
\r\n Dorothea Salo\r\n Oo. I think that was a challenge I just heard.\r\n\r\nOff to read the chapter now.\r\n \r\n \r\n \r\n ![](\"http://www.gravatar.com/avatar.php?gravatar_id=2fc977e82d9f8395a0f49e6f4958bc84&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\"/)
\r\n Mark\r\n He he. I'd forgotten about that one. As I recall, I wrote it at like 4 AM and considered taking it out upon reading it the next morning, but left it in and forgot about it.\r\n \r\n \r\n \r\n
Archived Comments
\n\n- \n
\n\n![](\"http://www.gravatar.com/avatar.php?gravatar_id=2aac8a68f46151f080ddb17ee0d2454e&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\")
\nMark Pilgrim\n \n\n\n \n\n\nActually, it's even better than that. I have a list of HTML tags that act as dividers: P, BR, HR, PRE, DIV, SPAN, TD, and a few others. I grab the text surrounding the link, demarcated by any combination of these dividers.\nThere's some other wicked cool stuff in there too, like trying to guess permalinks (and getting it right about 80% of the time, and avoiding false positives the other 20%). First it looks for trackback data; if found, that's golden. Otherwise it looks for the last anchor (name or id attribute in any tag, not just A) before the start of the excerpt, then looks for the first link after the excerpt that contains that anchor. This picks up the permalink on most blogs, since they include anchor tags before the entry named the same as the permalink (either the filename or the fragment).\nA concrete example (from your own home page): this post is preceded by an anchor \"\"\"[a name=\"000306\"][/a]\"\"\", and followed by a permalink \"\"\"[a href=\"http://www.decafbad.com/news_archives/000306.phtml\"]11:16 AM[/a]\"\"\". See how they match?\nIt's not just A tags, though; some anchors are just an id attribute stuck on a div or h3 tag. There are lots of variations. Life is messy. \n\n\n \n- \n
\n\n![](\"http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\")
\nl.m.orchard\n \n\n\n \n\n\nI just noticed the tag detection as I looked at the sample code you posted. Much more sophisticated than I'd initially thought. :) I forgot that Python has a useful SGML parser - I need to play with that more. Usually I assume that I'm stuck in trial-and-error regex-land when I want to dig out some content from a given HTML page.\nAs for the permalink detection, yes I'd found some false positives from the start, but I was amazed at how many non-obvious ones it found. Very cool! \n\n\n \n- \n
\n\n![](\"http://www.gravatar.com/avatar.php?gravatar_id=2aac8a68f46151f080ddb17ee0d2454e&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\")
\nMark Pilgrim\n \n\n\n \n\n\nPython's SGML parser totally rocks. It took me ages to grok it, but when I did, it changed my world.\nMy script to pull titles, language codes, and RSS auto-discovery URLs from pages? sgmllib.\nMy script to extract excerpts around links? sgmllib.\nMy script to find RSS feeds even without auto-discovery links? sgmllib.\nMy ultra-liberal RSS parser? sgmllib.\nI wrote up a whole chapter on sgmllib ( http://diveintopython.org/dialect_divein.html ), which no one ever understands. \n\n\n \n- \n
\n\n![](\"http://www.gravatar.com/avatar.php?gravatar_id=def808e6072af95b27c6d298fbb05a2b&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\")
\nDorothea Salo\n \n\n\n \n\n\nOo. I think that was a challenge I just heard.\nOff to read the chapter now. \n\n\n \n- \n
\n\n![](\"http://www.gravatar.com/avatar.php?gravatar_id=def808e6072af95b27c6d298fbb05a2b&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\")
\nDorothea Salo\n \n\n\n \n\n\nMark, dude, you write the best damn footnotes I think I have ever seen.\nhttp://diveintopython.org/dialect_extract.html#ftn.d0e14872\nOh, and this is supposed to be hard or something? I get it, and I'm not that bright. \n\n\n \n- \n
\n\n![](\"http://www.gravatar.com/avatar.php?gravatar_id=2fc977e82d9f8395a0f49e6f4958bc84&size=32&default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\")
\nMark\n \n\n\n \n\n\nHe he. I'd forgotten about that one. As I recall, I wrote it at like 4 AM and considered taking it out upon reading it the next morning, but left it in and forgot about it. \n\n\n \n
\n\n\n\n
\n
\n
\n
\n
\n