<html dir="ltr"><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta content="MSHTML 6.00.6000.16830" name="GENERATOR">
<style title="owaParaStyle">P {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
</style>
</head>
<body ocsi="x">
<div dir="ltr"><font face="Courier New" color="#000000" size="2">Hi Rod,</font></div>
<div dir="ltr"><font face="courier new" size="2"></font> </div>
<div dir="ltr"><font face="courier new" size="2">I'm fairly sure a HTTP 30* redirect would help - if you think about what a web crawler is doing, its just processing the contents of a page and whilst doing that building a list of links for further processing.
If referencing one of those links returns a redirect response with another URL to try, the returned URL is pushed onto the queue of links to be processed.</font></div>
<div dir="ltr"><font face="courier new" size="2"></font> </div>
<div dir="ltr"><font face="courier new" size="2">The TDWG resolver could fairly easily return a 301 (or some other variant of 30* redirect if more appropriate) as its not embellishing the IPNI data at all, it is presented "as is" ie compare:</font></div>
<div dir="ltr"><font face="courier new" size="2">
<div dir="ltr"><font face="courier new" size="2"><a href="http://lsid.tdwg.org/urn:lsid:ipni.org:names:30000959-2">http://lsid.tdwg.org/urn:lsid:ipni.org:names:30000959-2</a></font></div>
<div dir="ltr"><font face="courier new">and</font></div>
<div dir="ltr"><font face="courier new"><a href="http://www.ipni.org/ipni/plantNameByVersion.do?id=30000959-2&version=1.1.2.1&output_format=lsid-metadata&show_history=true">http://www.ipni.org/ipni/plantNameByVersion.do?id=30000959-2&version=1.1.2.1&output_format=lsid-metadata&show_history=true</a></font></div>
<div dir="ltr"><font face="courier new">(The latter being the end address used to access LSID metadata at ipni.org).</font></div>
<div dir="ltr"><font face="courier new"></font> </div>
<div dir="ltr"><font face="courier new">Only the "summary" <font face="courier new">
page adds anything to the metadata - reformatted into a more user friendly layout:</font>
<div dir="ltr">
<div dir="ltr"><font face="courier new" size="2"><a href="http://lsid.tdwg.org/summary/urn:lsid:ipni.org:names:30000959-2">http://lsid.tdwg.org/summary/urn:lsid:ipni.org:names:30000959-2</a></font></div>
<div dir="ltr"><font face="courier new"></font> </div>
<div dir="ltr"></font><font face="courier new"></font> </div>
<div dir="ltr"><font face="courier new">As you point out, the TDWG LSID resolver is indeed a full blown LSID resolver, and hence also generates calls to access the WSDL(s) for the LSID authority, in addition to that required to get the metadata. The authority
WSDL is the same every time so could well be cached. According to the spec, the service WSDL must indicate if the requested LSID does in fact exist. But slowing down the traffic from the current 3 x calls per request with no crawl delay to:
</font></div>
<div dir="ltr"><font face="courier new">1 x potentially cached request for authority WSDL</font></div>
<div dir="ltr"><font face="courier new">1 x request service WSDL</font></div>
<div dir="ltr"><font face="courier new">1 x HTTP 30* redirected and hence crawl delayed (the actual metadata address)</font></div>
<div dir="ltr"><font face="courier new"><font face="courier new">...</font>should improve our situation a bit.
</font></div>
</div>
</div>
<div dir="ltr"><font face="courier new"></font> </div>
<div dir="ltr">
<div dir="ltr"><font face="courier new">I'm not sure how one would go about making the TDWG resolver implement the robots exclusion protocol, as the resolver is not itself a crawler.</font></div>
<div dir="ltr"><font face="courier new"></font> </div>
<div dir="ltr"><font face="courier new">cheers,</font></div>
<div dir="ltr"><font face="courier new">Nicky</font></div>
<div dir="ltr"></font> </div>
</div>
</div>
<div dir="ltr"><font face="Courier New" size="2">- Nicola Nicolson<br>
- Science Applications Development,<br>
- Royal Botanic Gardens, Kew,<br>
- Richmond, Surrey, TW9 3AB, UK<br>
- email: n.nicolson@rbgkew.org.uk<br>
- phone: 020-8332-5766</div>
</font>
<div id="divRpF860431" style="DIRECTION: ltr">
<hr tabindex="-1">
<font face="Tahoma" size="2"><b>From:</b> Roderic Page [r.page@bio.gla.ac.uk]<br>
<b>Sent:</b> 27 April 2009 14:30<br>
<b>To:</b> Nicola Nicolson<br>
<b>Cc:</b> tdwg-tag@lists.tdwg.org<br>
<b>Subject:</b> Re: [tdwg-tag] LSIDs: web based (HTTP) resolvers and web crawlers<br>
</font><br>
</div>
<div></div>
<div>Dear Nicky,
<div><br>
</div>
<div>Ouch!</div>
<div><br>
</div>
<div>I'm not sure I fully understand how 30* redirects work with respect to web crawlers, but I'm not sure they will help in this case. </div>
<div><br>
</div>
<div>If the TDWG LSID resolver is a full blown resolver, then for each request from the crawler it will be doing the full LSID resolution (three calls, one for authority WSDL, one for service WSDL, one for metadata). It may cache the WSDLs, but it will still
do at least one call to the service (unless it has cached the metadata as well).</div>
<div><br>
</div>
<div>Is the solution to add TDWG to your robots.txt file, and have the LSID resolver respect the settings in that file? TDWG could also implement metadata caching so it wouldn't need to hammer you so much (i.e., when a crawler hit TDWG, TDWG would reply with
the cached metadata).</div>
<div><br>
</div>
<div>Perhaps LSID services such as IPNI's could also implement etag headers, which would help avoid excessive traffic from TDWG when caching (TDWG could regularly cache metadata form IPNI, respecting the robots.txt files, and first checking whether the metadata
had changed using etag and/or last modified headers).</div>
<div><br>
</div>
<div>I assume the DOI resolve has similar issues. It's robots.txt file looks like this:</div>
<div><br>
</div>
<div>
<div>Crawl-delay: 5</div>
<div>Request-rate: 1/5</div>
<div><br>
</div>
</div>
<div><br>
</div>
<div>Hope this makes sense, my understanding of the HTTP headers/redirects/robots.txt is not particularly deep.</div>
<div><br>
</div>
<div>Regards</div>
<div><br>
</div>
<div>Rod</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On 27 Apr 2009, at 13:54, Nicola Nicolson wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite"><span class="Apple-style-span">
<div>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">Hi,</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">Further to my last design question re LSID HTTP proxies (thanks for the responses), I wanted to raise the issue of HTTP LSID proxies and crawlers, in particular the crawl delay
part of the robots exclusion protocol.</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">I'll outline a situation we had recently:</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">The GBIF portal and ZipCodeZoo site both inclde IPNI LSIDs in the pages. These are presented in their proxied form using the TDWG LSID resolver (eg<span class="Apple-converted-space"> </span></font><a href="http://lsid.tdwg.org/urn:lsid:ipni.org:names:783030-1" target="_blank"><font face="Courier New" size="2">http://lsid.tdwg.org/urn:lsid:ipni.org:names:783030-1</font></a><font face="Courier New" size="2">).
Using the TDWG resolver to access the data for an IPNI LSID does not issue any kind of HTTP redirect, instead the web resolver uses the LSID resolution steps to get the data and presents it in its own response (ie returning a HTTP 200 OK response).</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">The problem happens when one of these sites that includes proxied IPNI LSIDs is crawled by a search engine. The proxied links appear to belong to tdwg.org, so whatever crawl
delay is agreed between TDWG and the crawler in question is used. The crawler has no knowledge that behind the scenes the TDWG resolver is hitting ipni.org. We (ipni.org) have agreed our own crawl limits with Google and the other major search engines using
directives in robots.txt and directly agreed limits with Google (who don't use the robots.txt directly).</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">On a couple of occasions in the past we have had to deny access to the TDWG LSID resolver as it has been responsible for far more traffic than we can support (up to 10 times
the crawl limits we have agreed with search engine bots) - this due to the pages on the GBIF portal and / or zipcodezoo being crawled by a search engine, which in turn triggers a high volume of requests from TDWG to IPNI. The crawler itself has no knowledge
that it is in effect accessing data held at ipni.org rather than tdwg.org as the HTTP response is HTTP 200.</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">One of Rod's emails recently mentioned that we need a resolver to act like a tinyurl or bit.ly. I have pasted below the HTTP headers for an HTTP request to the TDWG LSID resolver,
and to tinyurl / bit.ly. To the end user it looks as though tdwg.org is the true location of the LSID resource, whereas with the tinyurl and bitly both just redirect traffic.</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">I'm just posting this for discussion really - if we are to mandate use of a web based HTTP resolver/proxies, it should really issue 30* redirects so that established crawl delays
between producer and consumer will be used. The alternative would be for the HTTP resolver to read and process the directives in robots.txt, but this would be difficult to implement as it is not in itself a crawler, just a gateway.</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">I'm sure that if proxied forms of LSIDs become more prevalent this problem will become more widespread, so now - with the on-going attempt to define what services a GUID resolver
should provide - might be a good time to plan how to fix this.</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">cheers,<br>
Nicky</font></div>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><br>
<font face="Courier New" size="2">[nn00kg@kvstage01 ~]$ curl -I<span class="Apple-converted-space"> </span></font><a href="http://lsid.tdwg.org/urn:lsid:ipni.org:names:783030-1" target="_blank"><font face="Courier New" size="2">http://lsid.tdwg.org/urn:lsid:ipni.org:names:783030-1</font></a><br>
<font face="Courier New" size="2">HTTP/1.1 200 OK<br>
Via: 1.1 KISA01<br>
Connection: close<br>
Proxy-Connection: close<br>
Date: Mon, 27 Apr 2009 11:41:55 GMT<br>
Content-Type: application/xml<br>
Server: Apache/2.2.3 (CentOS)</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">[nn00kg@kvstage01 ~]$ curl -I<span class="Apple-converted-space"> </span></font><a href="http://tinyurl.com/czkquy" target="_blank"><font face="Courier New" size="2">http://tinyurl.com/czkquy</font></a><br>
<font face="Courier New" size="2">HTTP/1.1 301 Moved Permanently<br>
Via: 1.1 KISA01<br>
Connection: close<br>
Proxy-Connection: close<br>
Date: Mon, 27 Apr 2009 12:16:38 GMT<br>
Location:<span class="Apple-converted-space"> </span></font><a href="http://www.ipni.org/ipni/plantNameByVersion.do?id=783030-1&version=1.4&output_format=lsid-metadata&show_history=true" target="_blank"><font face="Courier New" size="2">http://www.ipni.org/ipni/plantNameByVersion.do?id=783030-1&version=1.4&output_format=lsid-metadata&show_history=true</font></a><br>
<font face="Courier New" size="2">Content-type: text/html<br>
Server: TinyURL/1.6<br>
X-Powered-By: PHP/5.2.9</font></div>
<p style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2"></font> </p>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><font face="Courier New" size="2">[nn00kg@kvstage01 ~]$ curl -I<span class="Apple-converted-space"> </span></font><a href="http://bit.ly/KO1Ko" target="_blank"><font face="Courier New" size="2">http://bit.ly/KO1Ko</font></a><br>
<font face="Courier New" size="2">HTTP/1.1 301 Moved Permanently<br>
Via: 1.1 KISA01<br>
Connection: Keep-Alive<br>
Proxy-Connection: Keep-Alive<br>
Content-Length: 287<br>
Date: Mon, 27 Apr 2009 12:19:48 GMT<br>
Location:<span class="Apple-converted-space"> </span></font><a href="http://www.ipni.org/ipni/plantNameByVersion.do?id=783030-1&version=1.4&output_format=lsid-metadata&show_history=true" target="_blank"><font face="Courier New" size="2">http://www.ipni.org/ipni/plantNameByVersion.do?id=783030-1&version=1.4&output_format=lsid-metadata&show_history=true</font></a><br>
<font face="Courier New" size="2">Content-Type: text/html;charset=utf-8<br>
Server: nginx/0.7.42<br>
Allow: GET, HEAD, POST</font></div>
<div style="MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px"><br>
<br>
<br>
<font face="Courier New" size="2">- Nicola Nicolson<br>
- Science Applications Development,<br>
- Royal Botanic Gardens, Kew,<br>
- Richmond, Surrey, TW9 3AB, UK<br>
- email:<span class="Apple-converted-space"> </span><a href="mailto:n.nicolson@rbgkew.org.uk">n.nicolson@rbgkew.org.uk</a><br>
- phone: 020-8332-5766</font></div>
_______________________________________________<br>
tdwg-tag mailing list<br>
<a href="mailto:tdwg-tag@lists.tdwg.org">tdwg-tag@lists.tdwg.org</a><br>
<a href="http://lists.tdwg.org/mailman/listinfo/tdwg-tag" target="_blank">http://lists.tdwg.org/mailman/listinfo/tdwg-tag</a><br>
</div>
</span></blockquote>
</div>
<br>
<div><span class="Apple-style-span">
<div><span class="Apple-style-span">
<div><span class="Apple-style-span">
<div><span class="Apple-style-span">
<div><span class="Apple-style-span">
<div>
<div>---------------------------------------------------------</div>
<div>Roderic Page</div>
<div>Professor of Taxonomy</div>
<div>DEEB, FBLS</div>
<div>Graham Kerr Building</div>
<div>University of Glasgow</div>
<div>Glasgow G12 8QQ, UK</div>
<div><br>
</div>
<div>Email: <a href="mailto:r.page@bio.gla.ac.uk">r.page@bio.gla.ac.uk</a></div>
<div>Tel: +44 141 330 4778</div>
<div>Fax: +44 141 330 2792</div>
<div>AIM: <a href="mailto:rodpage1962@aim.com">rodpage1962@aim.com</a></div>
<div>Facebook: <a href="http://www.facebook.com/profile.php?id=1112517192" target="_blank">http://www.facebook.com/profile.php?id=1112517192</a></div>
<div>Twitter: <a href="http://twitter.com/rdmpage" target="_blank">http://twitter.com/rdmpage</a></div>
<div>Blog: <a href="http://iphylo.blogspot.com" target="_blank">http://iphylo.blogspot.com</a></div>
<div>Home page: <a href="http://taxonomy.zoology.gla.ac.uk/rod/rod.html" target="_blank">
http://taxonomy.zoology.gla.ac.uk/rod/rod.html</a></div>
<div><br>
</div>
<div><br>
</div>
</div>
</span></div>
</span></div>
</span><br class="Apple-interchange-newline">
</div>
</span><br class="Apple-interchange-newline">
</div>
</span><br class="Apple-interchange-newline">
</div>
<br>
</div>
</div>
</body>
</html>