I suspect that the major crawlers have better error handling, but I have used Elmo from OpenRDF.org.<div><br></div><div>It does not have very robust error handling. It will try to pull in anything in it's whitelist that is linked via seeAlso, and fail if it is a PDF.</div>
<div><br></div><div>I have not tried Virtuoso for data crawling since I have worked out other ways to get RDF, but I suspect that it does a much better job.</div><div><br></div><div>Most groups now make their data available as an RDF dump which eliminated the need to crawl if you want to pull in a lot of data.</div>
<div><br></div><div>I guess the question is do you want to use a generic seeAlso which most crawlers follow, vs some more specific predicate that says "here is the PDF"</div><div><br></div><div>My reluctance was more about minting my own vs. finding some other vocabulary which has a similar predicate.</div>
<div><br></div><div>With the <b>hasPDF</b> predicate it would be pretty easy to query for all species concepts that have a linked original description PDF etc.</div><div><br></div><div>I suspect that some standard predicate will eventually become accepted since it is very useful to have something more specific than foaf:Document.</div>
<div><br></div><div>Respectively,</div><div><br></div><div>- Pete</div><div><br></div><div><br></div><div><br></div><div><br><br><div class="gmail_quote">On Wed, Jan 5, 2011 at 6:54 PM, Paul Murray <span dir="ltr"><<a href="mailto:pmurray@anbg.gov.au">pmurray@anbg.gov.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><br><div><div>On 06/01/2011, at 7:48 AM, Peter DeVries wrote:</div><br><blockquote type="cite">
<div>Also, although I like a lot of what Steve says, I think that most existing crawlers expect that a seeAlso link is to some html, xml, rdf type thing and will</div><div>not be able to handle a multi-megabyte PDF.</div>
<div><br></div><div>This is why I reluctantly minted the predicate "hasPDF"</div></blockquote><br></div><div>Hmm. This is an issue with linkeddata: when you fetch a URI while crawling the semantic web, if it redirects, then it's an "other resource" and you get RDF. If not, then you are potentially pulling a multimegabyte "information resource" across the wire.</div>
<div><br></div><div>A solution is to use an HTTP "HEAD" request when you do the initial URI fetch. If it's an "other resource", the HEAD return will be a 303 and contain redirect that you want in the "Location" header, and that's all you need. If not, the 200 result will contain the content type and possibly even the size, which is what you need to know before you GET it.</div>
<div><br></div>So .. the problem that "hasPDF" is meant to address might be addressable by the crawlers just being a bit smarter about how they browse the semweb.<div class="im"><div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div>
<div><font size="1"><span style="font-size:9px"><span style="font-size:medium">_______________________________________________</span></span></font></div></div></span>
</div>
<br></div><p>If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.
Please consider the environment before printing this email.</p>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>---------------------------------------------------------------<br>Pete DeVries<br>Department of Entomology<br>University of Wisconsin - Madison<br>445 Russell Laboratories<br>
1630 Linden Drive<br>Madison, WI 53706<br><a href="http://www.taxonconcept.org/" target="_blank">TaxonConcept Knowledge Base</a> / <a href="http://lod.geospecies.org/" target="_blank">GeoSpecies Knowledge Base</a><br><a href="http://about.geospecies.org/" target="_blank">About the GeoSpecies Knowledge Base</a><br>
------------------------------------------------------------<br>
</div>