{ "attachments": [], "comments_archived": true, "date": "2003-08-23T17:57:06.000Z", "layout": "post", "title": "Scraping HTML with web services", "wordpress_id": 466, "wordpress_slug": "rss-scrape-urls", "wordpress_url": "http://www.decafbad.com/blog/?p=466", "year": "2003", "month": "08", "day": "23", "isDir": false, "slug": "rss-scrape-urls", "type": "entry", "postName": "2003-08-23-rss-scrape-urls", "html": "

After checking out Bill Humphries’ approach to scraping yesterday, I recalled the various things Jon Udell has written about URL-as-command-line and the various places I’ve seen the W3C XSLT Servlet used in XSLT tinkering. I also remembered that there’s an HTML Tidy service offered by W3C as well.

\n\n
<p>So&#8230;  these are all URLs.  I figured I could pull together the site <span class=\"caps\">URL</span>, <a href=\"http://www.whump.com/dropbox/nationrss/nation.xsl\">Bill&#8217;s <span class=\"caps\">XSLT</span></a>, the tidy service, and the <span class=\"caps\">XSLT</span> service, and have a whole lot of scraping going on right in my browser or via wget or curl.  Here are the steps in how I composed the <span class=\"caps\">URL</span>:</p>\n\n<ol>\n<li><a href=\"http://www.thenation.com\">http://www.thenation.com</a></li>\n
\n
  • http://cgi.w3.org/cgi-bin/tidy?docAddr=http%3A%2F%2Fwww.thenation.com
    • \n
  • http://www.w3.org/2000/06/webdata/xslt?\n\n

    xslfile=http%3A%2F%2Fwww.whump.com%2Fdropbox%2Fnationrss%2Fnation.xsl&#38;

    \n

    xmlfile=http%3A%2F%2Fcgi.w3.org%2Fcgi-bin%2Ftidy%3F

    \n

    docAddr%3Dhttp%253A%252F%252Fwww.thenation.com&transform=Submit

    • \n\n\n
    <p>Unfortunately, this doesn&#8217;t work.  In particular, step [#2](/tag/2) fails, the Tidy service reporting a failure in processing the original <span class=\"caps\">HTML</span>.  I imagine, had that worked, the whole process at step [#3](/tag/3) would be producing <span class=\"caps\">RSS</span>.  On my command line, <span class=\"caps\">HTML </span>Tidy works fine, so I&#8217;ve been thinking of throwing together my own web interface to that program and seeing if that works.</p>\n\n<p>If it works, this with the addition of a cache at each stage could allow for what I think is a pretty nifty, all web-based means of scraping news items from web sites.  </p>\n\n    <p>What would really be nice for apps like this is a better way to express the URLs-within-URLs without escaping and escaping and escaping and...  Thinking some very lightweight scripting here, or some LISP-ish expressions would help.</p>\n
    \n\n\n\n

    shortname=rss_scrape_urls

    \n

    Archived Comments

    \n\n\n\n
    \n\n\n", "body": "

    After checking out Bill Humphries’ approach to scraping yesterday, I recalled the various things Jon Udell has written about URL-as-command-line and the various places I’ve seen the W3C XSLT Servlet used in XSLT tinkering. I also remembered that there’s an HTML Tidy service offered by W3C as well.

    \r\n\r\n\t

    So… these are all URLs. I figured I could pull together the site URL, Bill’s XSLT, the tidy service, and the XSLT service, and have a whole lot of scraping going on right in my browser or via wget or curl. Here are the steps in how I composed the URL:

    \r\n\r\n\t
      \r\n\t
    1. http://www.thenation.com
      2. \r\n
    2. http://cgi.w3.org/cgi-bin/tidy?docAddr=http%3A%2F%2Fwww.thenation.com
      3. \r\n
    3. http://www.w3.org/2000/06/webdata/xslt?
      xslfile=http%3A%2F%2Fwww.whump.com%2Fdropbox%2Fnationrss%2Fnation.xsl&
      xmlfile=http%3A%2F%2Fcgi.w3.org%2Fcgi-bin%2Ftidy%3F
      docAddr%3Dhttp%253A%252F%252Fwww.thenation.com&transform=Submit
      4. \r\n
    \r\n\r\n\t

    Unfortunately, this doesn’t work. In particular, step #2 fails, the Tidy service reporting a failure in processing the original HTML. I imagine, had that worked, the whole process at step #3 would be producing RSS. On my command line, HTML Tidy works fine, so I’ve been thinking of throwing together my own web interface to that program and seeing if that works.

    \r\n\r\n\t

    If it works, this with the addition of a cache at each stage could allow for what I think is a pretty nifty, all web-based means of scraping news items from web sites.

    \r\n \r\n

    What would really be nice for apps like this is a better way to express the URLs-within-URLs without escaping and escaping and escaping and... Thinking some very lightweight scripting here, or some LISP-ish expressions would help.

    \r\n\r\nshortname=rss_scrape_urls\r\n\r\n
    \r\n

    Archived Comments

    \r\n \r\n \r\n \r\n
    \r\n ", "parentPath": "./content/posts/archives/2003", "path": "2003/08/23/rss-scrape-urls", "summary": "After checking out Bill Humphries’ approach to scraping yesterday, I recalled the various things Jon Udell has written about URL-as-command-line and the various places I’ve seen the W3C XSLT Servlet used in XSLT tinkering. I also remembered that there’s an HTML Tidy service offered by W3C as well.\n\n

    So… these are all URLs. I figured I could pull together the site URL, Bill’s XSLT, the tidy service, and the XSLT service, and have a whole lot of scraping going on right in my browser or via wget or curl. Here are the steps in how I composed the URL:

    \n\n
      \n
    1. http://www.thenation.com
      2. \n\nhttp://cgi.w3.org/cgi-bin/tidy?docAddr=http%3A%2F%2Fwww.thenation.com\nhttp://www.w3.org/2000/06/webdata/xslt?\n\nxslfile=http%3A%2F%2Fwww.whump.com%2Fdropbox%2Fnationrss%2Fnation.xsl&\nxmlfile=http%3A%2F%2Fcgi.w3.org%2Fcgi-bin%2Ftidy%3F\ndocAddr%3Dhttp%253A%252F%252Fwww.thenation.com&transform=Submit\n\n\n

      Unfortunately, this doesn’t work. In particular, step [#2](/tag/2) fails, the Tidy service reporting a failure in processing the original HTML. I imagine, had that worked, the whole process at step [#3](/tag/3) would be producing RSS. On my command line, HTML Tidy works fine, so I’ve been thinking of throwing together my own web interface to that program and seeing if that works.

      \n\n

      If it works, this with the addition of a cache at each stage could allow for what I think is a pretty nifty, all web-based means of scraping news items from web sites.

      \n\n

      What would really be nice for apps like this is a better way to express the URLs-within-URLs without escaping and escaping and escaping and... Thinking some very lightweight scripting here, or some LISP-ish expressions would help.

      ", "needsBuild": true, "prevPostPath": "2003/08/22/rss-scrape-xsl", "prevPostTitle": "Scraping HTML with curl, tidy, and XSL", "nextPostPath": "2003/08/23/rss-scrape-urls2", "nextPostTitle": "Scraping with web services: Success" }