{ "attachments": [], "comments_archived": true, "date": "2003-09-26T02:45:29.000Z", "layout": "post", "title": "Dynamic feed polling times for news aggregators", "wordpress_id": 483, "wordpress_slug": "dynamic-feed-scan-times", "wordpress_url": "http://www.decafbad.com/blog/?p=483", "year": "2003", "month": "09", "day": "25", "isDir": false, "slug": "dynamic-feed-scan-times", "type": "entry", "postName": "2003-09-25-dynamic-feed-scan-times", "html": "

Today, my aggregator got\nthe following SQL worked into its feed poll scheduling machinery:

\n\n
SELECT id as source,\n       'update_period' as name,\n       max(1, 1/max((1.0/24.0),\n                    sum(update_count)/(7*24))) AS value \nFROM sources \nLEFT JOIN (\n    SELECT source AS count_id,\n                round(iso8601_to_epoch(created)/(60*60)) AS hour, \n                count(id) AS update_count \n    FROM items \n    WHERE created>epoch_to_iso8601(now()-(7*(24*60*60))) \n    GROUP BY hour\n) ON id=count_id\nGROUP BY source\nORDER BY value
\n\n

\nIt's likely that this is really nasty, but I have only a street-level\nworking knowledge of SQL. Also, a few of the date functions are\nspecific to how I've extended sqlite in Python. It works though, and\nwhat it does is this:\n

\n\n

\nFor each feed to which I'm subscribed, work out\nan average time between updates for the past week, with a maximum\nperiod of 24 hours and a minimum of 1 hour.\n

\n\n

\nMy aggregator does this daily, and uses the results to determine how\nfrequently to schedule scans. In this way, it automatically backs off\non checking feeds which update infrequently, and ramps up its polling\nof more active feeds. This shortens my feed downloading and scanning\ntime, and is kinder in general to everyone on my subscription list.\n

\n\n

\nNext, among other things, I have to look into making sure that the\nHTTP client parts of this beast pass all the\naggregator client\nHTTP tests that Mark\nPilgrim put together.\n

\n\n

\nUpdate: Well, it seemed like a good idea, anyway. But, on\nfurther examination, it has flaws. The most notable is that it\nassumes a polling frequency of once per hour. This works right up\nuntil I start changing the polling frequency with the results of the\ncalculation. I haven't poked at it yet, but maybe if I take this\ninto account, it'll be more accurate.\n

\n\n

\nOn the other hand, I've also been thinking about a much simpler\napproach to ramping polling frequency up and down: Start out at\na poll every hour. If, after a poll, no new items are found,\ndouble the time until the next poll. If new items were found,\nhalve the time until the next poll.

\n\n

\nProvide lower and upper limits to this, say between 1 hour and 1\nweek. Also, consider the ramp up and ramp down factor as a variable\nsetting too. Instead of a factor of 2, maybe try 1.5 or even 1.25 for\na more gradual change. To go even further, I wonder if it would be\nvaluable to dynamically alter this factor itself, to try to get the\npolling time zeroed in on a realistic polling time.\n

\n\n

\nOkay. There the simpler approach leaves simplicity. I'm sure there's\nsome decently elegant math that could be pulled in here. :)\n

\n\n\n\n\n\n

shortname=dynamic_feed_scan_times

\n

Archived Comments

\n\n\n\n
\n\n\n", "body": "

Today, my aggregator got\r\nthe following SQL worked into its feed poll scheduling machinery:

\r\n\r\n
SELECT id as source,\r\n       'update_period' as name,\r\n       max(1, 1/max((1.0/24.0),\r\n                    sum(update_count)/(7*24))) AS value \r\nFROM sources \r\nLEFT JOIN (\r\n    SELECT source AS count_id,\r\n                round(iso8601_to_epoch(created)/(60*60)) AS hour, \r\n                count(id) AS update_count \r\n    FROM items \r\n    WHERE created>epoch_to_iso8601(now()-(7*(24*60*60))) \r\n    GROUP BY hour\r\n) ON id=count_id\r\nGROUP BY source\r\nORDER BY value
\r\n\r\n

\r\nIt's likely that this is really nasty, but I have only a street-level\r\nworking knowledge of SQL. Also, a few of the date functions are\r\nspecific to how I've extended sqlite in Python. It works though, and\r\nwhat it does is this:\r\n

\r\n\r\n

\r\nFor each feed to which I'm subscribed, work out\r\nan average time between updates for the past week, with a maximum\r\nperiod of 24 hours and a minimum of 1 hour.\r\n

\r\n\r\n

\r\nMy aggregator does this daily, and uses the results to determine how\r\nfrequently to schedule scans. In this way, it automatically backs off\r\non checking feeds which update infrequently, and ramps up its polling\r\nof more active feeds. This shortens my feed downloading and scanning\r\ntime, and is kinder in general to everyone on my subscription list.\r\n

\r\n\r\n

\r\nNext, among other things, I have to look into making sure that the\r\nHTTP client parts of this beast pass all the\r\naggregator client\r\nHTTP tests that Mark\r\nPilgrim put together.\r\n

\r\n\r\n

\r\nUpdate: Well, it seemed like a good idea, anyway. But, on\r\nfurther examination, it has flaws. The most notable is that it\r\nassumes a polling frequency of once per hour. This works right up\r\nuntil I start changing the polling frequency with the results of the\r\ncalculation. I haven't poked at it yet, but maybe if I take this\r\ninto account, it'll be more accurate.\r\n

\r\n\r\n

\r\nOn the other hand, I've also been thinking about a much simpler\r\napproach to ramping polling frequency up and down: Start out at\r\na poll every hour. If, after a poll, no new items are found,\r\ndouble the time until the next poll. If new items were found,\r\nhalve the time until the next poll.

\r\n\r\n

\r\nProvide lower and upper limits to this, say between 1 hour and 1\r\nweek. Also, consider the ramp up and ramp down factor as a variable\r\nsetting too. Instead of a factor of 2, maybe try 1.5 or even 1.25 for\r\na more gradual change. To go even further, I wonder if it would be\r\nvaluable to dynamically alter this factor itself, to try to get the\r\npolling time zeroed in on a realistic polling time.\r\n

\r\n\r\n

\r\nOkay. There the simpler approach leaves simplicity. I'm sure there's\r\nsome decently elegant math that could be pulled in here. :)\r\n

\r\n\r\nshortname=dynamic_feed_scan_times\r\n\r\n
\r\n

Archived Comments

\r\n \r\n \r\n \r\n
\r\n ", "parentPath": "./content/posts/archives/2003", "path": "2003/09/25/dynamic-feed-scan-times", "summary": "Today, my aggregator got\nthe following SQL worked into its feed poll scheduling machinery:\n\nSELECT id as source,\n 'update_period' as name,\n max(1, 1/max((1.0/24.0),\n sum(update_count)/(7*24))) AS value \nFROM sources \nLEFT JOIN (\n SELECT source AS count_id,\n round(iso8601_to_epoch(created)/(60*60)) AS hour, \n count(id) AS update_count \n FROM items \n WHERE created>epoch_to_iso8601(now()-(7*(24*60*60))) \n GROUP BY hour\n) ON id=count_id\nGROUP BY source\nORDER BY value\n\n\nIt's likely that this is really nasty, but I have only a street-level\nworking knowledge of SQL. Also, a few of the date functions are\nspecific to how I've extended sqlite in Python. It works though, and\nwhat it does is this:\n\n\n\nFor each feed to which I'm subscribed, work out\nan average time between updates for the past week, with a maximum\nperiod of 24 hours and a minimum of 1 hour.\n\n\n\nMy aggregator does this daily, and uses the results to determine how\nfrequently to schedule scans. In this way, it automatically backs off\non checking feeds which update infrequently, and ramps up its polling\nof more active feeds. This shortens my feed downloading and scanning\ntime, and is kinder in general to everyone on my subscription list.\n\n\n\nNext, among other things, I have to look into making sure that the\nHTTP client parts of this beast pass all the\naggregator client\nHTTP tests that Mark\nPilgrim put together.\n\n\n\nUpdate: Well, it seemed like a good idea, anyway. But, on\nfurther examination, it has flaws. The most notable is that it\nassumes a polling frequency of once per hour. This works right up\nuntil I start changing the polling frequency with the results of the\ncalculation. I haven't poked at it yet, but maybe if I take this\ninto account, it'll be more accurate.\n\n\n\nOn the other hand, I've also been thinking about a much simpler\napproach to ramping polling frequency up and down: Start out at\na poll every hour. If, after a poll, no new items are found,\ndouble the time until the next poll. If new items were found,\nhalve the time until the next poll.\n\n\nProvide lower and upper limits to this, say between 1 hour and 1\nweek. Also, consider the ramp up and ramp down factor as a variable\nsetting too. Instead of a factor of 2, maybe try 1.5 or even 1.25 for\na more gradual change. To go even further, I wonder if it would be\nvaluable to dynamically alter this factor itself, to try to get the\npolling time zeroed in on a realistic polling time.\n\n\n\nOkay. There the simpler approach leaves simplicity. I'm sure there's\nsome decently elegant math that could be pulled in here. :)", "needsBuild": true, "prevPostPath": "2003/09/25/atom-is-its-name-o", "prevPostTitle": "Atom is its Name-O?", "nextPostPath": "2003/09/29/dynamic-polling-freq-too", "nextPostTitle": "Dynamic polling times for news aggregators, II" }