{
  "attachments": [],
  "comments_archived": true,
  "date": "2003-08-22T17:32:31.000Z",
  "layout": "post",
  "title": "Scraping HTML with curl, tidy, and XSL",
  "wordpress_id": 465,
  "wordpress_slug": "rss-scrape-xsl",
  "wordpress_url": "http://www.decafbad.com/blog/?p=465",
  "year": "2003",
  "month": "08",
  "day": "22",
  "isDir": false,
  "slug": "rss-scrape-xsl",
  "type": "entry",
  "postName": "2003-08-22-rss-scrape-xsl",
  "html": "<blockquote cite=\"http://www.whump.com/moreLikeThis/date/21/08/2003\"><i>Continuing with making it easier for \"Big Pubs\" to create RSS feeds. I'm assuming that they have a publishing system, but it wasn't built with RSS in mind, but they want on the bandwagon.</i></blockquote>\n\n<div class=\"credit\" align=\"right\"><small>Source: <cite><a href=\"http://www.whump.com/moreLikeThis/date/21/08/2003\">More Like This WebLog: Thursday, 21 August 2003</a></cite></small>\n\n</div>\n\n<p>Using curl, tidy, and <span class=\"caps\">XSL</span> to scrape content from <span class=\"caps\">HTML</span> pages into an <span class=\"caps\">RSS</span> feed.  This is basically what I do now with a half-baked Java app using JTidy, XPath, and BeanShell.  I keep meaning to release it, but it’s too embarassing to share so far.  Yet, it’s been working well enough to scrape what sites I’m interested in such that I haven’t been too motivated to tidy it up and tarball it.  One thing I like better about Bill Humphries’ approach, though, is that it doesn’t use Java :)</p>\n\n\n<!--more-->\n\n\n<p>shortname=rss_scrape_xsl</p>\n<div id=\"comments\" class=\"comments archived-comments\"><h3>Archived Comments</h3>\n<ul class=\"comments\">\n<li class=\"comment\" id=\"comment-221090870\">\n<div class=\"meta\">\n<div class=\"author\">\n<a class=\"avatar image\" rel=\"nofollow\" href=\"http://www.whump.com/moreLikeThis/\"><img src=\"http://www.gravatar.com/avatar.php?gravatar_id=622548e3f303e03297375ab20ddcb696&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\" width=\"\" height=\"\"></a>\n<a class=\"avatar name\" rel=\"nofollow\" href=\"http://www.whump.com/moreLikeThis/\">Bill Humphries</a>\n</div>\n\n\n<p><a href=\"#comment-221090870\" class=\"permalink\"><time datetime=\"2003-08-22T20:23:59\">2003-08-22T20:23:59</time></a></p>\n</div>\n\n\n<div class=\"content\">Well, it could use Java, if you really, really, want to since Xalan and Saxon have command line variants. I'm using LibXSLT in the demo.</div>\n\n\n</li>\n<li class=\"comment\" id=\"comment-221090871\">\n<div class=\"meta\">\n<div class=\"author\">\n<a class=\"avatar image\" rel=\"nofollow\" href=\"http://www.decafbad.com/blog\"><img src=\"http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\" width=\"\" height=\"\"></a>\n<a class=\"avatar name\" rel=\"nofollow\" href=\"http://www.decafbad.com/blog\">l.m.orchard</a>\n</div>\n\n\n<p><a href=\"#comment-221090871\" class=\"permalink\"><time datetime=\"2003-08-23T13:21:08\">2003-08-23T13:21:08</time></a></p>\n</div>\n\n\n<div class=\"content\">Well, I actually like the idea of chaining a few shell programs together much better than the all-in-one Java scraper I was tinkering with.  Seems so much easier all around.</div>\n\n\n</li>\n</ul>\n\n\n</div>\n\n\n",
  "body": "<blockquote cite=\"http://www.whump.com/moreLikeThis/date/21/08/2003\"><i>Continuing with making it easier for \"Big Pubs\" to create RSS feeds. I'm assuming that they have a publishing system, but it wasn't built with RSS in mind, but they want on the bandwagon.</i></blockquote><div class=\"credit\" align=\"right\"><small>Source: <cite><a href=\"http://www.whump.com/moreLikeThis/date/21/08/2003\">More Like This WebLog: Thursday, 21 August 2003</a></cite></small></div>\t<p>Using curl, tidy, and <span class=\"caps\">XSL</span> to scrape content from <span class=\"caps\">HTML</span> pages into an <span class=\"caps\">RSS</span> feed.  This is basically what I do now with a half-baked Java app using JTidy, XPath, and BeanShell.  I keep meaning to release it, but it&#8217;s too embarassing to share so far.  Yet, it&#8217;s been working well enough to scrape what sites I&#8217;m interested in such that I haven&#8217;t been too motivated to tidy it up and tarball it.  One thing I like better about Bill Humphries&#8217; approach, though, is that it doesn&#8217;t use Java :)</p>\r\n<!--more-->\r\nshortname=rss_scrape_xsl\r\n\r\n<div id=\"comments\" class=\"comments archived-comments\">\r\n            <h3>Archived Comments</h3>\r\n            \r\n        <ul class=\"comments\">\r\n            \r\n        <li class=\"comment\" id=\"comment-221090870\">\r\n            <div class=\"meta\">\r\n                <div class=\"author\">\r\n                    <a class=\"avatar image\" rel=\"nofollow\" \r\n                       href=\"http://www.whump.com/moreLikeThis/\"><img src=\"http://www.gravatar.com/avatar.php?gravatar_id=622548e3f303e03297375ab20ddcb696&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\"/></a>\r\n                    <a class=\"avatar name\" rel=\"nofollow\" \r\n                       href=\"http://www.whump.com/moreLikeThis/\">Bill Humphries</a>\r\n                </div>\r\n                <a href=\"#comment-221090870\" class=\"permalink\"><time datetime=\"2003-08-22T20:23:59\">2003-08-22T20:23:59</time></a>\r\n            </div>\r\n            <div class=\"content\">Well, it could use Java, if you really, really, want to since Xalan and Saxon have command line variants. I'm using LibXSLT in the demo.</div>\r\n            \r\n        </li>\r\n    \r\n        <li class=\"comment\" id=\"comment-221090871\">\r\n            <div class=\"meta\">\r\n                <div class=\"author\">\r\n                    <a class=\"avatar image\" rel=\"nofollow\" \r\n                       href=\"http://www.decafbad.com/blog\"><img src=\"http://www.gravatar.com/avatar.php?gravatar_id=2ac2cffd36ada8c734b90e02a1e5c1ac&amp;size=32&amp;default=http://mediacdn.disqus.com/1320279820/images/noavatar32.png\"/></a>\r\n                    <a class=\"avatar name\" rel=\"nofollow\" \r\n                       href=\"http://www.decafbad.com/blog\">l.m.orchard</a>\r\n                </div>\r\n                <a href=\"#comment-221090871\" class=\"permalink\"><time datetime=\"2003-08-23T13:21:08\">2003-08-23T13:21:08</time></a>\r\n            </div>\r\n            <div class=\"content\">Well, I actually like the idea of chaining a few shell programs together much better than the all-in-one Java scraper I was tinkering with.  Seems so much easier all around.</div>\r\n            \r\n        </li>\r\n    \r\n        </ul>\r\n    \r\n        </div>\r\n    ",
  "parentPath": "./content/posts/archives/2003",
  "path": "2003/08/22/rss-scrape-xsl",
  "summary": "Continuing with making it easier for \"Big Pubs\" to create RSS feeds. I'm assuming that they have a publishing system, but it wasn't built with RSS in mind, but they want on the bandwagon.\n\nSource: More Like This WebLog: Thursday, 21 August 2003\n\n\n\nUsing curl, tidy, and XSL to scrape content from HTML pages into an RSS feed.  This is basically what I do now with a half-baked Java app using JTidy, XPath, and BeanShell.  I keep meaning to release it, but it’s too embarassing to share so far.  Yet, it’s been working well enough to scrape what sites I’m interested in such that I haven’t been too motivated to tidy it up and tarball it.  One thing I like better about Bill Humphries’ approach, though, is that it doesn’t use Java :)",
  "needsBuild": true,
  "prevPostPath": "2003/08/22/email-feeds",
  "prevPostTitle": "Syndication feeds to replace email?",
  "nextPostPath": "2003/08/23/rss-scrape-urls",
  "nextPostTitle": "Scraping HTML with web services"
}