December 2007 - tdwg-tag - lists.tdwg.org

[tdwg-tapir] TapirTester & new client library
by Renato De Giovanni 27 Dec '07

27 Dec '07

Hi again... As I anticipated in my previous message, now TapirTester has a new test suite for DarwinCore providers that use a specific database: http://tapirtester.dev.tdwg.org/ You can try the new test suite on this provider: http://test.cria.org.br/tapirlink/tapir.php/dwc I forgot to mention in the other message that as part of this work I also started to develop a TAPIR client library in PHP. tapirChirp is still not feature complete, but it covers the most important functionalities of the protocol. You can download the first version here: https://sourceforge.net/project/showfiles.php?group_id=38190&package_id=256… The development source code of this library is also publicly available in the following subversion repository address: https://digir.svn.sourceforge.net/svnroot/digir/tapirchirp/trunk Best Wishes and Happy data sharing in 2008, -- Renato

1 0

[tdwg-tapir] TapirTester
by Renato De Giovanni 21 Dec '07

21 Dec '07

Deal all, This is just to let you know that there's a new tool to help checking TAPIR providers in terms of compatibility with the protocol: http://tapirtester.dev.tdwg.org/ If you don't know any TAPIR endpoint, you can try with this one: http://test.cria.org.br/tapirlink/tapir.php/test TapirTester is open source and can be downloaded with a subversion client. The address for checking out is: https://digir.svn.sourceforge.net/svnroot/digir/tapirtester/trunk/ If you get the software, you'll find instructions about how to install it and how to write new test suites. Besides the general test suite, there will soon be another one for full TAPIR providers that have mapped a specific database using DarwinCore concepts. By the way, anyone willing to improve the existing tests or write new test suites is welcome. Please let me know if you have any ideas or if you find any issues. Best Regards, -- Renato

2 1

[tdwg-tapir] Re: AW: AW: TapirLite first steps
by Döring, Markus 19 Dec '07

19 Dec '07

Guido, I've been plaing with your TAPIR service and it seems it doesnt understand paging. The specs are not awfully clear about whether this is optional, but I think any service should support start & limit parameters. See http://www.tdwg.org/dav/subgroups/tapir/1.0/docs/TAPIRSpecification_2007-07- 18.html#toc54 For example: http://idaho.ipd.uka.de/GgSRS/tapir?op=i&t=names&genus=P%25&limit=10 Should only return the first 10 records, while this should return 10 records starting with the 20th: http://idaho.ipd.uka.de/GgSRS/tapir?op=i&t=names&genus=P%25&limit=10&start=2 0 Looking forward to the plazi and golden gate presentation today! Regards, Markus "Guido Sautter" wrote on 22.11.2007 23:25 Uhr: > Hi Renato, > > I've created the servlet now, and it seems to work ... could you check if > all the XML files are valid, or if I violated the protocol somewhere? I'm > especially nervous about the search template, since the inventory templates > seem less complex. My service URL is > http://idaho.ipd.uka.de/GgSrsTapir/tapir?, just append what you like and > tell me if the server's response is (a) sensible, and (b) valid according to > the TAPIR specification. > > Thanks al lot, > Guido > > -----Ursprüngliche Nachricht----- > Von: Renato De Giovanni [mailto:renato@cria.org.br] > Gesendet: Freitag, 16. November 2007 13:25 > An: Guido Sautter > Cc: Donat Agosti; Terry Catapano; Markus Döring > Betreff: Re: AW: TapirLite first steps > > > Hi Guido, > > Did you already start to implement something based on that TAPIRLite recipe? > > Although I still have doubts about which concepts to use (DarwinCore, TDWG > ontology, etc.) I'm pretty much convinced that this work is feasible. By > the way, this kind of doubt doesn't prevent implementation - it just > prevents how to formalize the TAPIR documents (xml schemas, output models, > query templates, capabilities response, etc.). But this should be easy to > do later - and I can certainly help. > > I'm attaching a document that I'm using to put notes and ideas. The first > part is what we call a TAPIR CNS file which should contain all definitions > that we need. I just listed the concepts that I understood from you and > added a couple of query templates. These are only aliases. As I said we > can leave the respective GUIDs and documents to the end. > > So, continuing from the last step in the TAPIRLite recipe: > >> 7- When the operation is search, you should get the value of the parameter >> "template" or "t", probably delegate processing to another part of your >> code according to the template, and then produce the response according to >> the format that we still need to define. > > The value for the "template" parameter will be the associated alias or the > guid. I'm already suggesting some aliases in the attached file for the > query template - for instance "get_publications". > > Each template will have a known set of parameters that can be passed. I > would suggest that we start with no parameters for the inventory > templates. For the search templates you could start with these parameters: > > bllat: bottom left latitude > bllong: bottom left longitude > trlat: top right latitude > trlong: top right longitude > taxonname: like comparison with the scientific name > loc: like comparison with the location > > So you should check if each parameter exists and in that case add the > respective filter condition to your local search. > > Each template will also have it's own response structure. I'm suggesting > different response structures for the two search templates (see the > attached document). > > I would recommend to start implementing something like this and then make > adjustments later. What do you think? > > Please note that you're the ones to conceive the API so that it best suit > your needs, so please feel free to make any changes in the suggested > names/parameters/structures (I'll let you know if the changes have some > incompatibility with the protocol). > > Hope this helps, > -- > Renato > > >> Hi Renato, >> >> thanks a lot for all the tips :-) I think I see TapirLite is what we >> actually can do, and we are way closer now to our goal. Our data model is >> as >> follows (very simplistic): >> - Each document is identified by some meta data (MODS header) and contains >> one or more treatments in its body >> - A treatment is a chunk of text describing exactly one given taxon, maybe >> comparable to a section or sub-section in other scientific papers >> - A treatment refers to one specific taxon, which is identified with its >> Linaean (scientific) name, from now on refered to as "the taxon" >> - A treatment may contain names and coordinates of locations where >> specimen >> of the taxon have been collected >> - A treatment may contain free text comprising morphological descriptions >> of >> the taxon, and other aspect like ecology, discussions, etc. >> >> So from a data point of view, a treatment is basically an atomic record in >> a >> biodiversity database, with the following attributes: >> - exactly one scientific name of the taxon treated, plus individual parts >> of >> that name, like genus, subgenus, species, and subspecies >> - exactly one set of the MODS meta data of the publication the treatment >> belongs to, plus the number of the first and the last page >> - zero to many locations with (at least) name, longitude and latitude as >> subordinate attributes >> - zero to many paragraphs of free text for further descriptions, >> discussions, etc. >> >> In the first place, we want to offer this data as pairs of taxon name and >> location data: >> - {<scientificName> (genus=G species=S), <locationName> (long=X lat=Y)} >> - {<scientificName> (genus=G species=S), <linkToPublication>} (the example >> you used in your last mail) >> - {<locationName> (long=X lat=Y), <linkToPublication>} >> >> but also as plain lists of either taxon names or locations. If I got it >> right, the latter would be the purpose of an invetory request. All the >> attributes could be identified as the respective concepts in DarwinCore. >> >> We want to offer search functionality using either locations (using name >> or >> long/lat), or taxon names (the parts, not only the scientificName as a >> whole) as the predicates, using LIKE for comparing string values, but >> karthesian distance for long/lat. This allows for: >> - Computing the dispersal of a taxon >> - Computing the fauna of a location or area >> - Getting all the literature on some taxon >> - Getting all the literature on some area >> >> Hope this explains a little more. If any questions remain, just ask. >> >> All the best, >> Guido >> >> -----Ursprüngliche Nachricht----- >> Von: Renato De Giovanni [mailto:renato@cria.org.br] >> Gesendet: Mittwoch, 31. Oktober 2007 19:55 >> An: Guido Sautter >> Cc: Döring; Markus; Donat Agosti; Terry Catapano >> Betreff: TapirLite first steps >> >> >> Guido, >> >> These are the main points that you need to consider when developing a >> TAPIRLite provider: >> >> 1- You don't need to worry about XML parsing, only KVP (key-value-pair >> requests in HTTP GET or HTTP POST) >> >> 2- Your script should first detect which operation was requested (check >> for a parameter named "operation" or "op" which could contain "metadata", >> "m", "capabilities", "c", "inventory", "i", "search", "s", "ping", "p", >> all case insensitive). When the parameter is not passed, assume metadata >> operation. After detecting the operation you can delegate response >> processing to different parts of your code. >> >> 3- TAPIR responses are almost always included in a TAPIR envelope (except >> search responses when the parameter "envelope" or "e" was passed and >> contains "0" or "false"). I'm attaching a template for the TAPIR envelope >> (tapir_envelope.xml). >> >> 4- When the operation is metadata, just get the content from a local XML >> file and put it into the response body. I'm attaching a sample TAPIR >> metadata content (metadata.xml). >> >> 5- When the operation is capabilities, just get the content from a local >> XML file and put it into the response body. I'm attaching a sample TAPIR >> capabilities content (capabilities.xml). >> >> 6- When the operation is inventory, you should get the value of the >> parameter "template" or "t", probably delegate processing to another part >> of your code according to the template, and then produce the response. You >> can find a sample inventory response here: >> http://wiki.tdwg.org/twiki/bin/view/TAPIR/ExampleTapirMessages#iresp >> >> 7- When the operation is search, you should get the value of the parameter >> "template" or "t", probably delegate processing to another part of your >> code according to the template, and then produce the response according to >> the format that we still need to define. >> >> To better help you defining the structure of search result I would need >> understand more about your data model (sorry I'm not a biologist). Do you > >> have any UML diagram or ER diagram that could help? I still have basic >> doubts such as "can a taxonomic treatment be related to more than one >> scientific name and locality"? >> >> Could you also come up with a flat list of things for which you want to be >> able to search and return content? Just use natural language and then >> we'll see how to create some GUID for each concept. >> >> I need to leave now... >> >> Regards, >> -- >> Renato >>

1 0

[tdwg-tapir] r1355 - in tapir: 1.0/schema dev/schema
by svn＠rs.tdwg.org 12 Dec '07

12 Dec '07

Author: RenatoDeGiovanni Date: 2007-12-12 12:04:30 +0100 (Wed, 12 Dec 2007) New Revision: 1355 Modified: tapir/1.0/schema/tapir.xsd tapir/dev/schema/tapir.xsd Log: XML Schema validation was failing when search results returned no matching records. Added minOccurs = 0 to the root search result element. Modified: tapir/1.0/schema/tapir.xsd =================================================================== --- tapir/1.0/schema/tapir.xsd 2007-12-07 13:59:24 UTC (rev 1354) +++ tapir/1.0/schema/tapir.xsd 2007-12-12 11:04:30 UTC (rev 1355) @@ -8,7 +8,7 @@ xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:vcard="http://www.w3.org/2001/vcard-rdf/3.0#" elementFormDefault="qualified" - attributeFormDefault="unqualified" version="1.0.4" xml:lang="EN" > + attributeFormDefault="unqualified" version="1.0.5" xml:lang="EN" > <xsd:annotation> <xsd:documentation xml:lang="en"> TAPIR XML Schema. TAPIR is an acronym for TDWG Access Protocol for Information Retrieval. @@ -1828,7 +1828,7 @@ <xsd:documentation>Format of search operation result.</xsd:documentation> </xsd:annotation> <xsd:sequence> - <xsd:any processContents="lax" namespace="##other"/> + <xsd:any processContents="lax" namespace="##other" minOccurs="0"/> <xsd:element name="summary" type="resultSummaryType"> <xsd:annotation> <xsd:documentation>Used to show information about the number of records for Modified: tapir/dev/schema/tapir.xsd =================================================================== --- tapir/dev/schema/tapir.xsd 2007-12-07 13:59:24 UTC (rev 1354) +++ tapir/dev/schema/tapir.xsd 2007-12-12 11:04:30 UTC (rev 1355) @@ -8,7 +8,7 @@ xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:vcard="http://www.w3.org/2001/vcard-rdf/3.0#" elementFormDefault="qualified" - attributeFormDefault="unqualified" version="1.0.4" xml:lang="EN" > + attributeFormDefault="unqualified" version="1.0.5" xml:lang="EN" > <xsd:annotation> <xsd:documentation xml:lang="en"> TAPIR XML Schema. TAPIR is an acronym for TDWG Access Protocol for Information Retrieval. @@ -1828,7 +1828,7 @@ <xsd:documentation>Format of search operation result.</xsd:documentation> </xsd:annotation> <xsd:sequence> - <xsd:any processContents="lax" namespace="##other"/> + <xsd:any processContents="lax" namespace="##other" minOccurs="0"/> <xsd:element name="summary" type="resultSummaryType"> <xsd:annotation> <xsd:documentation>Used to show information about the number of records for

1 0

[tdwg-guid] Biodiversity Heritage Library Name Services v.1.0 released
by Chris Freeland 05 Dec '07

05 Dec '07

*Apologies for cross-posts* The Biodiversity Heritage Library (BHL) has released its first suite of Name Services into production. Using these services application providers can retrieve "discovered bibliographies" for a given taxonomic name showing where that name occurs throughout the entire corpus of natural history literature available through BHL. To view an example response, go to: http://www.biodiversitylibrary.org/services/name/NameService.asmx?op=Nam eGetDetail Enter 6663950 in nameBankID: box and hit Invoke. Full documentation of the services, including examples, can be found at: http://docs.google.com/Doc?id=dgvjvvkz_1x5qbm3 These services complement the human-readable interface to "discovered bibliographies" currently deployed at http://www.biodiversitylibrary.org/NameSearch.aspx. See http://www.biodiversitylibrary.org/name/Tapirus_bairdi for an example bibliography for Baird's Tapir. How it works Each digitized page image in BHL has an accompanying OCR text file. As users navigate to a page, the uncorrected OCR file is sent to uBio's TaxonFinder, which identifies text strings that match the characteristics of Latin binomials. Those potential name strings are then compared to the 10.7 million+ names in uBio's NameBank, and the results, both matched and unmatched, are stored in the BHL database. BHL also has automated processes to reindex pages at regular intervals since NameBank is a growing repository. What we've found As of 20 Nov 2007 more than 6.8 million potential name strings have been identified throughout the BHL corpus, with more than 3.8 million matched to a corresponding NameBank identifier. There are more than 431,000 unique names within that 3.8 million set. Of those, more than 156,000 are known by a single occurrence. These results will be evaluated more thoroughly in the coming months to determine potential errors such as false positives and how to refine the TaxonFinder algorithm to reduce them. Caveat: These results are generated from uncorrected OCR, which range in quality from pretty good (contemporary publications, such as modern issues of Rhodora) to downright terrible (18th century Latin texts, such as Species Plantarum). Again, further evaluation is required to determine the full scope of this problem. Metadata in response The metadata currently returned from the services is not yet explicitly mapped to Dublin Core, MODS, or other schemata but will be in subsequent revisions. This current response is the most verbose offering possible, encompassing every piece of information we have to share about the digitized page and the book from which it was scanned. For this first release we wanted to expose all metadata possible then work to map to other schema according to community wants and needs. Where we're headed This service, and the metadata it serves, will be incorporated into the Encyclopedia of Life by its launch next year. Discussions are already underway with other data providers to demonstrate how these services can be used. To see a simple example of how the human-readable interface can be used from external sites, check out the 'External Links' at the bottom of the Wikipedia article for Mimosa pudica L., the sensitive plant: http://en.wikipedia.org/wiki/Mimosa_pudica Up next: Development of an OpenURL parser that will allow service providers to submit requests for citations and retrieve page images or resolve to appropriate pages in the BHL portal. Any and all comments welcome! E-mail directly to me, or leave them on the BHL blog at http://biodiversitylibrary.blogspot.com. Chris Freeland Technical Director, Biodiversity Heritage Library Application Development Manager, Missouri Botanical Garden

1 0

[tdwg-tapir] Interpretation of TAPIR filters
by Renato De Giovanni 04 Dec '07

04 Dec '07

Dear all, I came across an issue that I know it was discussed before, but I couldn't find anything about it in the spec and I couldn't find a message with a final agreement/solution. The issue is: How should TAPIR filters be interpreted if they reference concepts that are not understood by providers or if they reference parameters that were not passed? Query templates may often include parameterized filters with many conditions, such as: <filter> <and> <equals> <concept id="http://rs.tdwg.org/dwc/dwcore/Genus"/> <parameter name="genus"/> </equals> <equals> <concept id="http://rs.tdwg.org/dwc/dwcore/SpecificEpithet"/> <parameter name="species"/> </equals> <equals> <concept id="http://rs.tdwg.org/dwc/dwcore/Country"/> <parameter name="country"/> </equals> </and> </filter> And it's important to be able to use the same query template even when not all parameters are passed, such as allowing all these calls: http://myprovider.org/tapir?op=search&t=mytemplate&genus=x http://myprovider.org/tapir?op=search&t=mytemplate&genus=x&species=y http://myprovider.org/tapir?op=search&t=mytemplate&country=z and so on... Otherwise we would need a different template for each combination of parameters, which would certainly make clients very unhappy. I've been discussing this with Markus and we're leaning towards the following suggestion about how to interpret filters: 1- Comparative operators (COPs) that reference unspecified parameters should be simply dropped from the filter. 2- COPs must evaluate to false if they reference a concept that is not understood by the provider, unless they also reference an unspecified parameter - in which case the COP must be dropped from the filter. 3- After dropping a COP, if it was part of a logical operator (LOP) then the LOP should also be dropped if there are no boolean operators left inside the LOP. 4- After dropping a LOP, if it was part of another LOP, then the parent LOP should also be dropped if there are no boolean operators left inside the parent LOP. There's clearly a gap in the specification about this and we need to fix it, so please let me know if you have any thoughts about this. If there are no objections or better suggestions, then I'll probably use the proposal above to amend the spec. Many thanks! -- Renato

3 6

Current LSID resolvers
by Kevin Richards 03 Dec '07

03 Dec '07

Hi all I have been thinking about the LSID resolvers that have been set up within the TDWG community so far. Some of these resolvers have evolved from the prototype/experimental stage of the GUID and TAG subgroups. There are also a variety of resolvers that no longer seem to be up and running, eg LSIDs at http://lsid.biopathways.org/authorities.shtml Most of the example LSIDs at Rod Page's tester page fail to some degree - http://darwin.zoology.gla.ac.uk/~rpage/lsid/tester/ Also, different metadata is being returned for each resolver. I have had comments that these LSID resolvers are not working, and "isn't the intention of LSIDs for them to persist forever and always be available". I wonder if one of our goals for the next year should be to look at these existing resolvers and decide on / announce / commit to their continued existence, metadata formats, etc. I think the following probably need to be addressed: - decide if each resolver is experimental or permanent - perhaps we need some standard "metadata" about a resolver that describes the intention of that resolver (eg whether it is experimental or not) - decide if the resolver should be taken offline to avoid confusion - the metadata format for similar resolvers, eg resolvers dealing with Taxon Name objetcs, or Taxon Concepts, or Specimens, etc. Does this need to be the same for each resolver? always RDF? In some cases a single class of the LSID vocabularies have been implemented, in other cases a mixture of several classes and properties. Do wee need some guidelines, best practices for the "reccommended" fields? This obviously requires further work with associated ontologies. Any thoughts? Kevin

1 0

RE: [tdwg-guid] TDWG LSID Resolver broken?
by Richard Pyle 03 Dec '07

03 Dec '07

VERY strange! This LSID does not work: http://lsid.tdwg.org/urn:lsid:zoobank.org:act:20889795-7EC7-42F3-A4C3-D1D977 04A609 But this one does: http://lsid.tdwg.org/urn:lsid:bishopmuseum.org:tnu:20889795-7EC7-42F3-A4C3-D 1D97704A609 Even though these are the exact same record from the exact same database, and the latter resolver was created from the exact same source code as the former. Both seem to work fine on Rod's tester page, except the latter has one failure (5a), while in the former it is just a warning. Go figure.... Rich ________________________________ From: Paul Kirk [mailto:p.kirk@cabi.org] Sent: Wednesday, November 28, 2007 11:57 AM To: Richard Pyle Subject: RE: [tdwg-guid] TDWG LSID Resolver broken? Hi Rich, works OK for IF LSIDs http://lsid.tdwg.org/urn:lsid:indexfungorum.org:names:296755 <http://lsid.tdwg.org/urn:lsid:indexfungorum.org:names:296755> Cheers, Paul ________________________________ From: tdwg-guid-bounces(a)lists.tdwg.org on behalf of Richard Pyle Sent: Wed 28/11/2007 21:52 To: tdwg-guid(a)lists.tdwg.org Subject: [tdwg-guid] TDWG LSID Resolver broken? Hi All, I'm testing the ZooBank LSID resolver, and I can't seem to get any of my LSIDs to work with the TDWG LSID resolver (http://lsid.tdwg.org/) They used to work a couple months ago, but now they don't. For example: urn:lsid:zoobank.org:act:20889795-7EC7-42F3-A4C3-D1D97704A609 This works fine on Rod's Tester site: http://linnaeus.zoology.gla.ac.uk/~rpage/lsid/tester/ Any ideas??? Thanks, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html _______________________________________________ tdwg-guid mailing list tdwg-guid(a)lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-guid P Think Green - don't print this email unless you really need to ************************************************************************ The information contained in this e-mail and any files transmitted with it is confidential and is for the exclusive use of the intended recipient. If you are not the intended recipient please note that any distribution, copying or use of this communication or the information in it is prohibited. Whilst CAB International trading as CABI takes steps to prevent the transmission of viruses via e-mail, we cannot guarantee that any e-mail or attachment is free from computer viruses and you are strongly advised to undertake your own anti-virus precautions. If you have received this communication in error, please notify us by e-mail at cabi(a)cabi.org or by telephone on +44 (0)1491 829199 and then delete the e-mail and any copies of it. CABI is an International Organization recognised by the UK Government under Statutory Instrument 1982 No. 1071. **************************************************************************

15 43

OOSTethys
by Bob Morris 01 Dec '07

01 Dec '07

Has any cognizance been taken of the OOSTethys project? Some aspects of its architecture, whose semantically mediated service implementation is proceeding (???) might be adoptable for registries of ontologies extending the TDWG core ontology. See http://www.oostethys.org/overview [If it's already been examined and commented upon, can anyone point me to those comments? Thanks.] Bob -- Robert A. Morris Professor of Computer Science UMASS-Boston ram(a)cs.umb.edu http://bdei.cs.umb.edu/ http://www.cs.umb.edu/~ram http://www.cs.umb.edu/~ram/calendar.html phone (+1)617 287 6466

1 0