{ "attachments": [], "comments_archived": true, "date": "2003-09-29T17:48:28.000Z", "layout": "post", "title": "Dynamic polling times for news aggregators, II", "wordpress_id": 484, "wordpress_slug": "dynamic-polling-freq-too", "wordpress_url": "http://www.decafbad.com/blog/?p=484", "year": "2003", "month": "09", "day": "29", "isDir": false, "slug": "dynamic-polling-freq-too", "type": "entry", "postName": "2003-09-29-dynamic-polling-freq-too", "html": "

\nOkay, so that thing with the SQL I did Friday?\nI'm not exactly sure what I was thinking with it. I was doing something\nthat seems really odd now, trying to collect counts of new items together\nby hour, then averaging those hourly counts across a week. Instead, I'm\ntrying this now:\n

\n\n
SELECT\n  source,\n  'update_period' AS name,\n  round(min(24,max(1,(max(1,(iso8601_to_epoch(max(created)) -\n    max(now() - (7*24*60*60), iso8601_to_epoch(min(created)))) /\n   (60*60))) / count(id))),2) AS value\nFROM\n  items\nWHERE\n  created >= epoch_to_iso8601(now() - (7*24*60*60)) \nGROUP BY\n  source
\n\n

\nThis bit of SQL, though still ugly, is much simpler. This leaves out\nthe subselect, which I think I might have been playing with in order\nto build a little graph display of new items over time by source. What\nthe above does now is to get an average time between new items for the\npast week, with a minimum of an hour, and a maximum of a day. This\nseems to be working much better.\n

\n\n

\nAn alternate algorithm I've been playing with was suggested in\na comment\nby Gnomon,\ninspired by TCP/IP's Additive Increase / Multiplicative Decrease.\nWith this, I subtract an hour from the time between polls when a\npoll finds new items, and then multiply by 2 every time a poll\ncomes up with nothing new.\n

\n\n

\nUsing the average of new items over time lessens my pummeling\nof servers per hour, but the second approach is even lighter\non polling since it's biased toward large leaps backing off\nfrom polling when new items are not found. I'll likely be trading\noff between the two to see which one seems to work best.\n

\n\n

\nHoping that, after playing a bit, I'll settle on one and my\naggregator will play much nicer with feeds, especially once\nI get the HTTP client usage to correctly use things like\nlast-modified headers and ETags. There's absolutely no reason\nfor a news aggregator to poll a feed every single hour of a day,\nunless you're monitoring a feed that's mostly quiet, except\nfor emergencies. In that case, well, a different polling\nalgorithm is needed, or maybe an instant messaging or pub/sub\narchitecture is required.\n

\n\n

\nUpdate: As Gnomon\nhas corrected me in comments, I've got the AIMD algorithm mixed up.\nWhat I really should be doing is making quick jumps up in polling\nfrequency in response to new items (multiplicative decrease of\npolling period) and creeping away in response to no new items\n(additive increase of polling period). As he notes, this approach\nshould make an aggregator jump to attention when clumps of new\nposts come in, and gradually get bored over periods of silence.\nI've adjusted my code and will be tinkering with it.\n

\n\n

\nAlso, although Gnomon makes\na good point that bloggers and their posting habits are not easily\nsubject to statistical analysis,\nI've further refined my little SQL query to catch sources\nwhich haven't seen any updates during the week (or ever):\n

\n\n
SELECT \n  id as source,\n  'update_period' AS name,\n  round(min(24,max(1,coalesce(update_period,24)))) AS value\nFROM sources\nLEFT JOIN (\n     SELECT\n      source AS source_id,\n            (iso8601_to_epoch(max(created)) -\n              max(\n                now()-(7*24*60*60),\n                iso8601_to_epoch(min(created))\n              )\n            ) / (60*60) / count(id)\n        AS update_period\n    FROM items\n    WHERE created >= epoch_to_iso8601(now() - (7*24*60*60)) \n    GROUP BY source\n) ON sources.id=source_id
\n\n

\nAlso, in case anyone's interested, I've checked all the above\ninto CVS. This beastie's far from ready for prime time, but it\nmight be interesting to someone.\n

\n\n\n\n\n\n

shortname=dynamic_polling_freq_too

\n

Archived Comments

\n\n\n\n
\n\n\n", "body": "

\r\nOkay, so that thing with the SQL I did Friday?\r\nI'm not exactly sure what I was thinking with it. I was doing something\r\nthat seems really odd now, trying to collect counts of new items together\r\nby hour, then averaging those hourly counts across a week. Instead, I'm\r\ntrying this now:\r\n

\r\n\r\n
SELECT\r\n  source,\r\n  'update_period' AS name,\r\n  round(min(24,max(1,(max(1,(iso8601_to_epoch(max(created)) -\r\n    max(now() - (7*24*60*60), iso8601_to_epoch(min(created)))) /\r\n   (60*60))) / count(id))),2) AS value\r\nFROM\r\n  items\r\nWHERE\r\n  created >= epoch_to_iso8601(now() - (7*24*60*60)) \r\nGROUP BY\r\n  source
\r\n\r\n

\r\nThis bit of SQL, though still ugly, is much simpler. This leaves out\r\nthe subselect, which I think I might have been playing with in order\r\nto build a little graph display of new items over time by source. What\r\nthe above does now is to get an average time between new items for the\r\npast week, with a minimum of an hour, and a maximum of a day. This\r\nseems to be working much better.\r\n

\r\n\r\n

\r\nAn alternate algorithm I've been playing with was suggested in\r\na comment\r\nby Gnomon,\r\ninspired by TCP/IP's Additive Increase / Multiplicative Decrease.\r\nWith this, I subtract an hour from the time between polls when a\r\npoll finds new items, and then multiply by 2 every time a poll\r\ncomes up with nothing new.\r\n

\r\n\r\n

\r\nUsing the average of new items over time lessens my pummeling\r\nof servers per hour, but the second approach is even lighter\r\non polling since it's biased toward large leaps backing off\r\nfrom polling when new items are not found. I'll likely be trading\r\noff between the two to see which one seems to work best.\r\n

\r\n\r\n

\r\nHoping that, after playing a bit, I'll settle on one and my\r\naggregator will play much nicer with feeds, especially once\r\nI get the HTTP client usage to correctly use things like\r\nlast-modified headers and ETags. There's absolutely no reason\r\nfor a news aggregator to poll a feed every single hour of a day,\r\nunless you're monitoring a feed that's mostly quiet, except\r\nfor emergencies. In that case, well, a different polling\r\nalgorithm is needed, or maybe an instant messaging or pub/sub\r\narchitecture is required.\r\n

\r\n\r\n

\r\nUpdate: As Gnomon\r\nhas corrected me in comments, I've got the AIMD algorithm mixed up.\r\nWhat I really should be doing is making quick jumps up in polling\r\nfrequency in response to new items (multiplicative decrease of\r\npolling period) and creeping away in response to no new items\r\n(additive increase of polling period). As he notes, this approach\r\nshould make an aggregator jump to attention when clumps of new\r\nposts come in, and gradually get bored over periods of silence.\r\nI've adjusted my code and will be tinkering with it.\r\n

\r\n\r\n

\r\nAlso, although Gnomon makes\r\na good point that bloggers and their posting habits are not easily\r\nsubject to statistical analysis,\r\nI've further refined my little SQL query to catch sources\r\nwhich haven't seen any updates during the week (or ever):\r\n

\r\n\r\n
SELECT \r\n  id as source,\r\n  'update_period' AS name,\r\n  round(min(24,max(1,coalesce(update_period,24)))) AS value\r\nFROM sources\r\nLEFT JOIN (\r\n     SELECT\r\n      source AS source_id,\r\n            (iso8601_to_epoch(max(created)) -\r\n              max(\r\n                now()-(7*24*60*60),\r\n                iso8601_to_epoch(min(created))\r\n              )\r\n            ) / (60*60) / count(id)\r\n        AS update_period\r\n    FROM items\r\n    WHERE created >= epoch_to_iso8601(now() - (7*24*60*60)) \r\n    GROUP BY source\r\n) ON sources.id=source_id
\r\n\r\n

\r\nAlso, in case anyone's interested, I've checked all the above\r\ninto CVS. This beastie's far from ready for prime time, but it\r\nmight be interesting to someone.\r\n

\r\n\r\nshortname=dynamic_polling_freq_too\r\n\r\n
\r\n

Archived Comments

\r\n \r\n \r\n \r\n
\r\n ", "parentPath": "./content/posts/archives/2003", "path": "2003/09/29/dynamic-polling-freq-too", "summary": "Okay, so that thing with the SQL I did Friday?\nI'm not exactly sure what I was thinking with it. I was doing something\nthat seems really odd now, trying to collect counts of new items together\nby hour, then averaging those hourly counts across a week. Instead, I'm\ntrying this now:\n\n\nSELECT\n source,\n 'update_period' AS name,\n round(min(24,max(1,(max(1,(iso8601_to_epoch(max(created)) -\n max(now() - (7*24*60*60), iso8601_to_epoch(min(created)))) /\n (60*60))) / count(id))),2) AS value\nFROM\n items\nWHERE\n created >= epoch_to_iso8601(now() - (7*24*60*60)) \nGROUP BY\n source\n\n\nThis bit of SQL, though still ugly, is much simpler. This leaves out\nthe subselect, which I think I might have been playing with in order\nto build a little graph display of new items over time by source. What\nthe above does now is to get an average time between new items for the\npast week, with a minimum of an hour, and a maximum of a day. This\nseems to be working much better.\n\n\n\nAn alternate algorithm I've been playing with was suggested in\na comment\nby Gnomon,\ninspired by TCP/IP's Additive Increase / Multiplicative Decrease.\nWith this, I subtract an hour from the time between polls when a\npoll finds new items, and then multiply by 2 every time a poll\ncomes up with nothing new.\n\n\n\nUsing the average of new items over time lessens my pummeling\nof servers per hour, but the second approach is even lighter\non polling since it's biased toward large leaps backing off\nfrom polling when new items are not found. I'll likely be trading\noff between the two to see which one seems to work best.\n\n\n\nHoping that, after playing a bit, I'll settle on one and my\naggregator will play much nicer with feeds, especially once\nI get the HTTP client usage to correctly use things like\nlast-modified headers and ETags. There's absolutely no reason\nfor a news aggregator to poll a feed every single hour of a day,\nunless you're monitoring a feed that's mostly quiet, except\nfor emergencies. In that case, well, a different polling\nalgorithm is needed, or maybe an instant messaging or pub/sub\narchitecture is required.\n\n\n\nUpdate: As Gnomon\nhas corrected me in comments, I've got the AIMD algorithm mixed up.\nWhat I really should be doing is making quick jumps up in polling\nfrequency in response to new items (multiplicative decrease of\npolling period) and creeping away in response to no new items\n(additive increase of polling period). As he notes, this approach\nshould make an aggregator jump to attention when clumps of new\nposts come in, and gradually get bored over periods of silence.\nI've adjusted my code and will be tinkering with it.\n\n\n\nAlso, although Gnomon makes\na good point that bloggers and their posting habits are not easily\nsubject to statistical analysis,\nI've further refined my little SQL query to catch sources\nwhich haven't seen any updates during the week (or ever):\n\n\nSELECT \n id as source,\n 'update_period' AS name,\n round(min(24,max(1,coalesce(update_period,24)))) AS value\nFROM sources\nLEFT JOIN (\n SELECT\n source AS source_id,\n (iso8601_to_epoch(max(created)) -\n max(\n now()-(7*24*60*60),\n iso8601_to_epoch(min(created))\n )\n ) / (60*60) / count(id)\n AS update_period\n FROM items\n WHERE created >= epoch_to_iso8601(now() - (7*24*60*60)) \n GROUP BY source\n) ON sources.id=source_id\n\n\nAlso, in case anyone's interested, I've checked all the above\ninto CVS. This beastie's far from ready for prime time, but it\nmight be interesting to someone.", "needsBuild": true, "prevPostPath": "2003/09/25/dynamic-feed-scan-times", "prevPostTitle": "Dynamic feed polling times for news aggregators", "nextPostPath": "2003/10/01/mailbucket-feeds", "nextPostTitle": "Mailbucket syndication feeds" }