May 2009 - tdwg-tag - lists.tdwg.org

Re: [tdwg-tag] reaching TDWG groups
by Lee Belbin 04 May '09

04 May '09

Hi Bob I'd suggest tdwg-tag(a)lists.tdwg.org, tdwg(a)lists.tdwg.org and maybe even a small news item on the home page. That way, the historical event gets a good spread and is logged. Cheers Lee Lee Belbin TDWG Secretariat From: Bob Morris [mailto:morris.bob@gmail.com] Sent: Tuesday, 5 May 2009 9:40 AM To: Lee Belbin Subject: reaching TDWG groups To announce call for comment on MRTG schema draft, what lists would you suggest I send to. Thanks Bob -- Robert A. Morris Professor of Computer Science UMASS-Boston ram(a)cs.umb.edu http://bdei.cs.umb.edu/ http://www.cs.umb.edu/~ram http://www.cs.umb.edu/~ram/calendar.html phone (+1)617 287 6466

1 0

TaxonOccurrence non-standard (Spinning off from something Peter DeVries mentioned)
by Roger Hyam 01 May '09

01 May '09

Hi Peter, The TaxonOccurrence standard is not actually a standard. It is part of the vocabulary that was put together two years ago and maps well to DwC. Meanwhile, as Markus points out, DwC has gone its own way. The whole TDWG ontology needs some clarification and sorting out and as Bob points out this is now done on a voluntary basis. I am working on the TaxonName and TaxonConcept vocabularies because I need to use them. I hope I will provide what Markus requests (good return type definitions for GUIDs) but I will only do it for taxon names and concepts because that is all I am working on. I hope to take this to a stage where it is solid, documented and standardizable. This will entail doing some stuff on the way the TDWG ontology/ vocabulary is presented but really it is not my department to do that alone. This is very important for TDWG (as Markus points out). What really needs to happen is for people who need things to happen in the ontology to role up their sleeves and make them happen. For occurrence stuff I recommend you join forces with John Wieczorek. All the best, Roger

13 24

LSIDs: web based (HTTP) resolvers and web crawlers
by Nicola Nicolson 01 May '09

01 May '09

Hi, Further to my last design question re LSID HTTP proxies (thanks for the responses), I wanted to raise the issue of HTTP LSID proxies and crawlers, in particular the crawl delay part of the robots exclusion protocol. I'll outline a situation we had recently: The GBIF portal and ZipCodeZoo site both inclde IPNI LSIDs in the pages. These are presented in their proxied form using the TDWG LSID resolver (eg http://lsid.tdwg.org/urn:lsid:ipni.org:names:783030-1). Using the TDWG resolver to access the data for an IPNI LSID does not issue any kind of HTTP redirect, instead the web resolver uses the LSID resolution steps to get the data and presents it in its own response (ie returning a HTTP 200 OK response). The problem happens when one of these sites that includes proxied IPNI LSIDs is crawled by a search engine. The proxied links appear to belong to tdwg.org, so whatever crawl delay is agreed between TDWG and the crawler in question is used. The crawler has no knowledge that behind the scenes the TDWG resolver is hitting ipni.org. We (ipni.org) have agreed our own crawl limits with Google and the other major search engines using directives in robots.txt and directly agreed limits with Google (who don't use the robots.txt directly). On a couple of occasions in the past we have had to deny access to the TDWG LSID resolver as it has been responsible for far more traffic than we can support (up to 10 times the crawl limits we have agreed with search engine bots) - this due to the pages on the GBIF portal and / or zipcodezoo being crawled by a search engine, which in turn triggers a high volume of requests from TDWG to IPNI. The crawler itself has no knowledge that it is in effect accessing data held at ipni.org rather than tdwg.org as the HTTP response is HTTP 200. One of Rod's emails recently mentioned that we need a resolver to act like a tinyurl or bit.ly. I have pasted below the HTTP headers for an HTTP request to the TDWG LSID resolver, and to tinyurl / bit.ly. To the end user it looks as though tdwg.org is the true location of the LSID resource, whereas with the tinyurl and bitly both just redirect traffic. I'm just posting this for discussion really - if we are to mandate use of a web based HTTP resolver/proxies, it should really issue 30* redirects so that established crawl delays between producer and consumer will be used. The alternative would be for the HTTP resolver to read and process the directives in robots.txt, but this would be difficult to implement as it is not in itself a crawler, just a gateway. I'm sure that if proxied forms of LSIDs become more prevalent this problem will become more widespread, so now - with the on-going attempt to define what services a GUID resolver should provide - might be a good time to plan how to fix this. cheers, Nicky [nn00kg@kvstage01 ~]$ curl -I http://lsid.tdwg.org/urn:lsid:ipni.org:names:783030-1 HTTP/1.1 200 OK Via: 1.1 KISA01 Connection: close Proxy-Connection: close Date: Mon, 27 Apr 2009 11:41:55 GMT Content-Type: application/xml Server: Apache/2.2.3 (CentOS) [nn00kg@kvstage01 ~]$ curl -I http://tinyurl.com/czkquy HTTP/1.1 301 Moved Permanently Via: 1.1 KISA01 Connection: close Proxy-Connection: close Date: Mon, 27 Apr 2009 12:16:38 GMT Location: http://www.ipni.org/ipni/plantNameByVersion.do?id=783030-1&version=1.4&outp… Content-type: text/html Server: TinyURL/1.6 X-Powered-By: PHP/5.2.9 [nn00kg@kvstage01 ~]$ curl -I http://bit.ly/KO1Ko HTTP/1.1 301 Moved Permanently Via: 1.1 KISA01 Connection: Keep-Alive Proxy-Connection: Keep-Alive Content-Length: 287 Date: Mon, 27 Apr 2009 12:19:48 GMT Location: http://www.ipni.org/ipni/plantNameByVersion.do?id=783030-1&version=1.4&outp… Content-Type: text/html;charset=utf-8 Server: nginx/0.7.42 Allow: GET, HEAD, POST - Nicola Nicolson - Science Applications Development, - Royal Botanic Gardens, Kew, - Richmond, Surrey, TW9 3AB, UK - email: n.nicolson(a)rbgkew.org.uk - phone: 020-8332-5766

13 23