{ "attachments": [], "comments_archived": true, "date": "2003-08-16T04:28:41.000Z", "layout": "post", "title": "Issues in using SpamBayes to filter news items", "wordpress_id": 462, "wordpress_slug": "bayes-agg-one", "wordpress_url": "http://www.decafbad.com/blog/?p=462", "year": "2003", "month": "08", "day": "16", "isDir": false, "slug": "bayes-agg-one", "type": "entry", "postName": "2003-08-16-bayes-agg-one", "html": "

Despite a reading an entry by Srijith\ndiscussing Bayes-based classification as unsuitable\nfor use in news aggregators, I tied SpamBayes\ninto my homebrew news aggregator\nand have been trying it out this week. I know I’ve been talking about it\nfor awhile, but procrastination and being busy all round kept me from getting\nto it. Funny thing is, when I finally got a chance to really check things out,\nthe integration was a snap. I’d anticipated a bit of work, but was pleasantly\nsurprised. I doubt that any other aggregator written in\nPython would have a hard time with it.

\n\n
<p>If, that is, anyone else wants to do it.  I already knew it wasn&#8217;t\n
\n

magic pixy dust\nbut I figured it might be worth a try. I will be eating my dogfood\nfor awhile with this, but I’m thinking already that what’s good for spam\nmight not be so good for news aggregators.

\n
<p>Srijith&#8217;s <a href=\"http://www.srijith.net/trinetre/archives/2003/08/11/index.shtml#000373\">post</a>\n
\n

mentions some snags in ignoring some of the semantics of a news item,\nsuch as whether a word appears in the item’s title or information about\nthe item’s source. I don’t think that this completely\napplies to how I’m doing classification, since SpamBayes appears to\ndifferentiate between words found in email headers and the body itself.\nWhen I feed an item to SpamBayes for training and scoring, I represent\nit as something like an email message, with headers like date, subject,\nfrom, and an “X-Link” header for the link. However, even with this,\nI think Srijith’s got a point when he writes that this method will miss\na lot of available clues for classification.

\n
<p>Unlike Srijith&#8217;s examples, though, I&#8217;m not trying to train my\n
\n

aggregator to sift entries into any specific categories. So far, I’ve\nbeen trying to get it to discriminate between what I really want to\nread, and what I’m not so interested in. So, I figured that something\nwhich can learn the difference between spam and normal email could help.\nBut, although it’s early, I’m noticing a few things about the results and\nI’ve had a few things occur to me.

\n
<p>See, in the case of ham vs spam, I really want all the ham and none of\n
\n

the spam. A method to differentiate between these two should be\noptimized toward one answer or the other. SpamBayes offers “I don’t\nknow” as a third answer, but it’s not geared toward anything else\nin-between. However, in measuring something like “interest“,\ninbetween answers are useful. I want all of the interesting stuff,\nsome of the sort-of interesting stuff, and a little of the rest.

\n
<p>This is also a problem for me in deciding to what I\n
\n

should give a thumbs up and what gets the thumbs down. Even though\nI’ve subscribed to a little over 300 feeds, every item from each of\nthem is somewhat interesting to me. I wouldn’t have subscribed to the\nfeed if there wasn’t anything of interest there, so I’ve already\nbiased the content of what I receive. Some items are more interesting\nthan others, but the difference between them is nowhere near the\ndifference of wanted ham vs unsolicited spam. So, I find myself\ngiving the nod to lots of items, but only turning down a few.\nSpamBayes would like equal examples of both, if possible.

\n
<p>I&#8217;ll still be playing with this for awhile, but I need to look\n
\n

around at other machine learning tech. I’m just hacking around,\nbut the important thing is to try to understand the algorithms\nbetter and know how they work and why. Bayes is in vogue right now,\nbut as Mark Pilgrim intimated, it’s not magic. It’s just “advanced” :)

\n
<p>In the immortal words of <a href=\"http://www.spidereyeballs.com/os6/set3/small_os6_d3_3596_sm.html\">Mark Jason Dominus</a>: &#8220;You can&#8217;t just make shit\n
\n

up and expect the computer to know what you mean, retardo!”

\n\n\n\n

shortname=bayes_agg_one

\n

Archived Comments

\n\n\n\n
\n\n\n", "body": "

Despite a reading an entry by Srijith\r\ndiscussing Bayes-based classification as unsuitable\r\nfor use in news aggregators, I tied SpamBayes\r\ninto my homebrew news aggregator\r\nand have been trying it out this week. I know I’ve been talking about it\r\nfor awhile, but procrastination and being busy all round kept me from getting\r\nto it. Funny thing is, when I finally got a chance to really check things out,\r\nthe integration was a snap. I’d anticipated a bit of work, but was pleasantly\r\nsurprised. I doubt that any other aggregator written in\r\nPython would have a hard time with it.

\r\n\r\n\t

If, that is, anyone else wants to do it. I already knew it wasn’t\r\nmagic pixy dust\r\nbut I figured it might be worth a try. I will be eating my dogfood\r\nfor awhile with this, but I’m thinking already that what’s good for spam\r\nmight not be so good for news aggregators.

\r\n\r\n\t

Srijith’s post\r\nmentions some snags in ignoring some of the semantics of a news item,\r\nsuch as whether a word appears in the item’s title or information about\r\nthe item’s source. I don’t think that this completely\r\napplies to how I’m doing classification, since SpamBayes appears to\r\ndifferentiate between words found in email headers and the body itself.\r\nWhen I feed an item to SpamBayes for training and scoring, I represent\r\nit as something like an email message, with headers like date, subject,\r\nfrom, and an “X-Link” header for the link. However, even with this,\r\nI think Srijith’s got a point when he writes that this method will miss\r\na lot of available clues for classification.

\r\n\r\n\t

Unlike Srijith’s examples, though, I’m not trying to train my\r\naggregator to sift entries into any specific categories. So far, I’ve\r\nbeen trying to get it to discriminate between what I really want to\r\nread, and what I’m not so interested in. So, I figured that something\r\nwhich can learn the difference between spam and normal email could help.\r\nBut, although it’s early, I’m noticing a few things about the results and\r\nI’ve had a few things occur to me.

\r\n\r\n\t

See, in the case of ham vs spam, I really want all the ham and none of\r\nthe spam. A method to differentiate between these two should be\r\noptimized toward one answer or the other. SpamBayes offers “I don’t\r\nknow” as a third answer, but it’s not geared toward anything else\r\nin-between. However, in measuring something like “interest“,\r\ninbetween answers are useful. I want all of the interesting stuff,\r\nsome of the sort-of interesting stuff, and a little of the rest.

\r\n\r\n\t

This is also a problem for me in deciding to what I\r\nshould give a thumbs up and what gets the thumbs down. Even though\r\nI’ve subscribed to a little over 300 feeds, every item from each of\r\nthem is somewhat interesting to me. I wouldn’t have subscribed to the\r\nfeed if there wasn’t anything of interest there, so I’ve already\r\nbiased the content of what I receive. Some items are more interesting\r\nthan others, but the difference between them is nowhere near the\r\ndifference of wanted ham vs unsolicited spam. So, I find myself\r\ngiving the nod to lots of items, but only turning down a few.\r\nSpamBayes would like equal examples of both, if possible.

\r\n\r\n\t

I’ll still be playing with this for awhile, but I need to look\r\naround at other machine learning tech. I’m just hacking around,\r\nbut the important thing is to try to understand the algorithms\r\nbetter and know how they work and why. Bayes is in vogue right now,\r\nbut as Mark Pilgrim intimated, it’s not magic. It’s just “advanced” :)

\r\n\r\n\t

In the immortal words of Mark Jason Dominus: “You can’t just make shit\r\nup and expect the computer to know what you mean, retardo!”

\r\n\r\nshortname=bayes_agg_one\r\n\r\n
\r\n

Archived Comments

\r\n \r\n \r\n \r\n
\r\n ", "parentPath": "./content/posts/archives/2003", "path": "2003/08/16/bayes-agg-one", "summary": "Despite a reading an entry by Srijith\ndiscussing Bayes-based classification as unsuitable\nfor use in news aggregators, I tied SpamBayes\ninto my homebrew news aggregator\nand have been trying it out this week. I know I’ve been talking about it\nfor awhile, but procrastination and being busy all round kept me from getting\nto it. Funny thing is, when I finally got a chance to really check things out,\nthe integration was a snap. I’d anticipated a bit of work, but was pleasantly\nsurprised. I doubt that any other aggregator written in\nPython would have a hard time with it.\n\n

If, that is, anyone else wants to do it. I already knew it wasn’t\n\nmagic pixy dust\nbut I figured it might be worth a try. I will be eating my dogfood\nfor awhile with this, but I’m thinking already that what’s good for spam\nmight not be so good for news aggregators.\n

Srijith’s post\n\nmentions some snags in ignoring some of the semantics of a news item,\nsuch as whether a word appears in the item’s title or information about\nthe item’s source. I don’t think that this completely\napplies to how I’m doing classification, since SpamBayes appears to\ndifferentiate between words found in email headers and the body itself.\nWhen I feed an item to SpamBayes for training and scoring, I represent\nit as something like an email message, with headers like date, subject,\nfrom, and an “X-Link” header for the link. However, even with this,\nI think Srijith’s got a point when he writes that this method will miss\na lot of available clues for classification.\n

Unlike Srijith’s examples, though, I’m not trying to train my\n\naggregator to sift entries into any specific categories. So far, I’ve\nbeen trying to get it to discriminate between what I really want to\nread, and what I’m not so interested in. So, I figured that something\nwhich can learn the difference between spam and normal email could help.\nBut, although it’s early, I’m noticing a few things about the results and\nI’ve had a few things occur to me.\n

See, in the case of ham vs spam, I really want all the ham and none of\n\nthe spam. A method to differentiate between these two should be\noptimized toward one answer or the other. SpamBayes offers “I don’t\nknow” as a third answer, but it’s not geared toward anything else\nin-between. However, in measuring something like “interest“,\ninbetween answers are useful. I want all of the interesting stuff,\nsome of the sort-of interesting stuff, and a little of the rest.\n

This is also a problem for me in deciding to what I\n\nshould give a thumbs up and what gets the thumbs down. Even though\nI’ve subscribed to a little over 300 feeds, every item from each of\nthem is somewhat interesting to me. I wouldn’t have subscribed to the\nfeed if there wasn’t anything of interest there, so I’ve already\nbiased the content of what I receive. Some items are more interesting\nthan others, but the difference between them is nowhere near the\ndifference of wanted ham vs unsolicited spam. So, I find myself\ngiving the nod to lots of items, but only turning down a few.\nSpamBayes would like equal examples of both, if possible.\n

I’ll still be playing with this for awhile, but I need to look\n\naround at other machine learning tech. I’m just hacking around,\nbut the important thing is to try to understand the algorithms\nbetter and know how they work and why. Bayes is in vogue right now,\nbut as Mark Pilgrim intimated, it’s not magic. It’s just “advanced” :)\n

In the immortal words of Mark Jason Dominus: “You can’t just make shit\n\nup and expect the computer to know what you mean, retardo!”", "needsBuild": true, "prevPostPath": "2003/08/14/tree-files-too", "prevPostTitle": "Tree files too (the prequel)", "nextPostPath": "2003/08/19/cookies-are-yummy", "nextPostTitle": "Cookies are yummy" }