March 2012 - tdwg-tag - lists.tdwg.org

Proposed charter for a TAG Vocabulary Management Task Group (VOMAG)
by Dag Endresen (GBIF) 02 Nov '12

02 Nov '12

Dear TAG, After battling with the plans for a biodiversity knowledge organization (KOS) framework for biodiversity information resources we have identified the requirement to develop guidelines and best practices for the management of vocabularies of terms. Basic terms organized in vocabularies provides an essential element to underpin the biodiversity information standards. As introduced at the TDWG 2011 TAG meetings in New Orleans, we propose the formation of a new Vocabulary Management Task Group (VOMAG) [1] to be organized under the TDWG technical architecture group (TAG). Please find the draft charter available from the GBIF Community Site [2][3]. Here you will also find another draft document "Biodiversity Knowledge Organization System: Proposed Architecture; Version 0.1 February 2012" that provides an overview of the proposed KOS landscape and the context for the proposed work-plan of the Vocabulary Management Task Group Charter. This is the first draft so far only discussed with Greg Whitbread as convener of the TDWG TAG and with Steve Baskauf and Joel Sachs as convener of the TDWG RDF/OWL task group. We invite feedback and comments to the proposed formation of the task group including suggestions with regard to the work-plan. Please join the Vocabulary Management group at the GBIF Community Site [1]. You can start or participate in discussions or share suggestions using the GBIF Community Site. Feel also free to make contact with us to volunteer as a core member for this proposed task group! [1] http://community.gbif.org/pg/groups/21382/vocabulary-management/ [2] http://community.gbif.org/pg/file/read/21388/ [3] http://community.gbif.org/pg/blog/read/21387/ [4] http://community.gbif.org/pg/file/read/21582/ Best regards Dag, Eamonn and David -- Dag Endresen, PhD Knowledge Systems Engineer Global Biodiversity Information Facility (GBIF) Universitetsparken 15, DK-2100 Copenhagen, Denmark http://community.gbif.org/pg/profile/dag.endresen

13 21

Change genus definition or introduce genericName?
by Peter Desmet 22 Mar '12

22 Mar '12

Hi everyone, A spin-off of the discussion regarding canonicalScientificName (and possibly a solution to it as well) concerns the term "genus". As mentioned before, most people probably interpret this term as "The name of the genus of the scientificName." - scientificName: *Aster* simplex var. ramosissimus (Torrey & A. Gray) A. Cronquist - acceptedNameUsage: Symphyotrichum lanceolatum (Willdenow) G.L. Nesom subsp. lanceolatum var. lanceolatum - genus: *Aster* - specificEpithet: simplex - infraspecificEpithet: ramosissimus Even though it is defined as "The full scientific name of the genus in which the taxon is classified.", which leads to this for synonyms: - scientificName: Aster simplex var. ramosissimus (Torrey & A. Gray) A. Cronquist - acceptedNameUsage: *Symphyotrichum* lanceolatum (Willdenow) G.L. Nesom subsp. lanceolatum var. lanceolatum - genus: *Symphyotrichum Nees* - specificEpithet: simplex - infraspecificEpithet: ramosissimus For more background on this issue, please read: http://code.google.com/p/darwincore/issues/detail?id=151 & http://lists.tdwg.org/pipermail/tdwg-content/2012-March/002877.html There are now two possible solutions on the table (by Markus Döring, http://code.google.com/p/darwincore/issues/detail?id=151#c7): for sharing the genus name part we have 2 options: A) consistent dwc approach: We keep the dwc:genus definition as it is to be inline with the other classification terms including subgenus. All of these should contain the accepted classification, not necessarily the part of the name. Two new terms genericName and infragenericEpithet are then proposed to hold the genus part of a name and the infra generic part. This approach keeps the dwc terms consistent and does not mix classification terms with terms for atomized names. B) least change approach: Taking into account that many people already use dwc:genus for the name part we alter the definition of the term as proposed here. This results in the least impact on existing publishers and probably consumers too. But it has the drawback of i) dwc terms are inconsistent with family & subgenus being the classification, not the name part, ii) no option anymore to share the genus classification, iii) no way to share all infra generic names I would hope we can start a vote around those 2 options with A) being my clear favorite We need your opinion and input to move this issue forward. Thanks, Peter -- Peter Desmet Biodiversity Informatics Manager Canadensys - www.canadensys.net Université de Montréal Biodiversity Centre 4101 rue Sherbrooke est Montreal, QC, H1X2B2 Canada Phone: 514-343-6111 #82354 Fax: 514-343-2288 Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com Skype: anderhalv Public profile: http://www.linkedin.com/in/peterdesmet

1 0

Call for Challenge entries for conference on Informatics for Phylogenetics, Evolution, and Biodiversity (iEvoBio)
by Nico Cellinese 21 Mar '12

21 Mar '12

Many trees enter. Fresh analysis ensues. New insights emerge. The conference on Informatics for Phylogenetics, Evolution, and Biodiversity (iEvoBio) is again holding a Challenge competition in 2012, on the theme "Synthesizing Phylogenies". Somewhere, buried in large sets of trees, lies a stunning new revelation, a baffling discovery, the answer to a longstanding controversy, or simply something not obvious to the naked eye. The mission of the 2012 iEvoBio challenge is to find those revelations, discoveries and answers within your own data and/or within one of the datasets provided by the challenge. What new scientifically interesting results can you pull from these trees, using any combination of techniques at your disposal? Further information on the nature of challenge entries and how to submit them can be found on the iEvoBio website athttp://ievobio.org/challenge.html. Submissions are due by June 25, 2012. Selected candidates will make short oral presentations to demonstrate their work in a "Challenge session" at the conference, and if suitable will automatically be entered into the Software Bazaar track of the conference. Cash prizes will be awarded for first place (USD 1,500) and runner-up entries. The winning entries will be selected by a vote of the iEvoBio meeting participants. Alongside the iEvoBio Challenge, 2012 iEvoBio sponsor Biomatters Ltd is running the Geneious Challenge. The goal of this challenge is to develop a new plugin to Geneious Pro, using the public API, that enables a new and exciting visualization or analysis. The winning entry will receive a $1000 cash prize, and all entrants who submit by the deadline will receive a 12-month subscription license. The Geneious Challenge has the same open and closing dates as the iEvoBio Challenge. See http://ievobio.org/geneious_challenge.html for more information. Separate from iEvoBio, the Encyclopedia of Life (EOL) is leading a Phylogenetic Tree challenge to encourage not only the development of large phylogenies, but their practical use. Together with the Global Names project they will choose two winning entries: one for the best large tree and one for the most promising but not necessarily large tree -- both in Darwin Core Archive format suitable for use in major biodiversity websites such as EOL. The two winners will receive paid trips to the 2012 iEvoBio conference. See http://eol.org/info/tree_challenge for more information. More details about the iEvoBio conference and program are available at http://ievobio.org. You can also find continuous updates on the conference's Twitter feed at http://twitter.com/iEvoBio and Google+ page, or subscribe to the low-traffic iEvoBio announcements mailing list athttp://groups.google.com/group/ievobio-announce. iEvoBio 2012 is sponsored by the US National Evolutionary Synthesis Center (NESCent) and by Biomatters Ltd., in partnership with the Society for the Study of Evolution (SSE) and the Systematic Biologists (SSB). The iEvoBio 2012 Organizing Committee: Hilmar Lapp, US National Evolutionary Synthesis Center (chair) Robert Beiko, Dalhousie University Nico Cellinese, University of Florida Robert Guralnick, University of Colorado at Boulder Rebecca Kao, Denver Botanic Gardens Ellinor Michel, Natural History Museum, London Nadia Talent, Royal Ontario Museum Andrea Thomer, University of Illinois at Urbana-Champaign <><><><><><><><><><><><><><><><><><><><><><><><> Nico Cellinese, Ph.D. Assistant Curator, Botany & Informatics Adjunct Assistant Professor, Department of Biology Florida Museum of Natural History University of Florida 354 Dickinson Hall, PO Box 117800 Gainesville, FL 32611-7800, U.S.A. Tel. 352-273-1979 Fax 352-846-1861 http://cellinese.blogspot.com/

1 0

Re: [tdwg-tag] [tdwg-content] canonicalScientificName
by Roderic Page 16 Mar '12

16 Mar '12

Apart from the fact that I can barely bring myself to care about plants ;) I suspect that the vast majority of names do not present these problems. Why do we let edge cases determine what we do? Regards Rod On 14 Mar 2012, at 19:10, Dmitry Mozzherin wrote: > To add to Gregor's post > > scientificNameAuthorship common usage assumes that a name is a linear > construct which is true for many cases. However in general scientific > name is a tree, and it is most obvious with names of hybrids. So when > people try to apply solutions like splitting name to canonical form > and authorship, it won't work for everybody and everything > > Dima > > On Wed, Mar 14, 2012 at 2:58 PM, Gregor Hagedorn <g.m.hagedorn(a)gmail.com> wrote: >>> - When providers had a text blob for the name, separate from the text blob >>> for the authorship, they could concatenate the two for presentation in >>> scientificName, and also provide the authorship bit in >>> scientificNameAuthorship, and the consumers could easily strip the >>> authorship from scientificName to produce the functional equivalent of a >>> canonical name. >> >> >> this does not work for autonyms in botany, i.e. the infraspecific taxa where >> the infraspecific epithet and the specific epithet are the same. Here the >> author is before the infraspecific epithet. Inconvenient, but prevents the >> simplest solution. >> >> _______________________________________________ >> tdwg-content mailing list >> tdwg-content(a)lists.tdwg.org >> http://lists.tdwg.org/mailman/listinfo/tdwg-content >> > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content > --------------------------------------------------------- Roderic Page Professor of Taxonomy Institute of Biodiversity, Animal Health and Comparative Medicine College of Medical, Veterinary and Life Sciences Graham Kerr Building University of Glasgow Glasgow G12 8QQ, UK Email: r.page(a)bio.gla.ac.uk Tel: +44 141 330 4778 Fax: +44 141 330 2792 Skype: rdmpage AIM: rodpage1962(a)aim.com Facebook: http://www.facebook.com/profile.php?id=1112517192 Twitter: http://twitter.com/rdmpage Blog: http://iphylo.blogspot.com Home page: http://taxonomy.zoology.gla.ac.uk/rod/rod.html

6 7

GUID and LSID standards
by Bob Morris 16 Mar '12

16 Mar '12

http://www.tdwg.org/homepage-news-item/article/lsid-and-guid-applicability-… lists http://www.tdwg.org/standards/150/download/ as the location of the approved standards, but those documents indicate that their status is "TDWG Draft Standard" No doubt those are the drafts that were actually approved, but shouldn't the documents be corrected as to their status? Also, it would be far nicer if they were downloadable as separate documents. It is clumsy to cite just one of them. Speaking of citation, what if all approved standards carried a recommended way to cite them? Bob -- Robert A. Morris Emeritus Professor of Computer Science UMASS-Boston 100 Morrissey Blvd Boston, MA 02125-3390 IT Staff Filtered Push Project Harvard University Herbaria Harvard University email: morris.bob(a)gmail.com web: http://efg.cs.umb.edu/ web: http://etaxonomy.org/mw/FilteredPush http://www.cs.umb.edu/~ram === The content of this communication is made entirely on my own behalf and in no way should be deemed to express official positions of The University of Massachusetts at Boston or Harvard University.

4 3

Re: [tdwg-tag] [tdwg-content] Canonical name parsing
by Peter Desmet 15 Mar '12

15 Mar '12

Hi Donald, scientificName, with its current definition [1] is a great term and should be continued to used as such. As with most Darwin Core terms, it offers flexibility, so its not an impediment for publishing data. In the GBIF context, this term is considered mandatory: records without it are ignored during indexing (I believe). All of this can stay. canonicalScientificName would be an *additional* term with a *clear rule* (see my proposed definition [2]). This is the case for other Darwin Core terms as well, such as decimalLatitude [3], minimalElevationInMeters [4] or countryCode [5]. They serve as an ready-to-use addition/alternative to verbatimLatitude [6], verbatimElevation [7] and country [8] respectively. These terms don't stop anyone from publishing data, but data publishers who can provide this kind of information have the choice to do so. It would be the same for canonicalScientificName. And yes, an aggregator like GBIF can play an important role in providing consistent data to its users and figuring out what they really need, but not all data is consumed that way. In addition, I hope a user would be able to download cleaned data from the GBIF portal as Darwin Core. Wouldn't it be nice that the parsed canonicalScientificName created by GBIF can be provided in its proper term? There are users out there who want this! Regards, Peter [1] http://rs.tdwg.org/dwc/terms/index.htm#scientificName [2] http://code.google.com/p/darwincore/issues/detail?id=150 [3] http://rs.tdwg.org/dwc/terms/index.htm#decimalLatitude [4] http://rs.tdwg.org/dwc/terms/index.htm#minimumElevationInMeters [5] http://rs.tdwg.org/dwc/terms/index.htm#countryCode [6] http://rs.tdwg.org/dwc/terms/index.htm#verbatimLatitude [7] http://rs.tdwg.org/dwc/terms/index.htm#verbatimElevation [8] http://rs.tdwg.org/dwc/terms/index.htm#country On Wed, Mar 14, 2012 at 11:19, Donald Hobern (GBIF) <dhobern(a)gbif.org> wrote: > > Hi Peter. > > I certainly agree that aggregators only represent one use case here but, having seen a lot of the mess of real-world data, I don't believe that simply adding a new term will fix this problem for the users you describe. To get the results you want, we would need a sufficiently large majority of data sets to follow the rules perfectly that we could ignore those that were non-conformant. This would mean we should mandate that every data set must use the new element (with or without the existing scientificName element) and that they must present scientific names in the expected way (or else have their data considered non-compliant). Until now, the philosophy on publishing Darwin Core data has been to make it as easy as possible for data providers to expose their data, even at the expense of greater complexity for consumers. I suspect that we would have a lot less data available for use now if we had taken a more stringent approach. > > In some ways, this proposal reminds me of the structures in ABCD which seek to offer users verbatim and more normalised ways to represent several types of information. This actually makes consuming all the possible forms of such data very complex, since a record may contain all variant forms or just any one of them. If multiple forms are available, which one should be considered the primary version? > > I suspect that things may also get complicated as soon as you discuss botanical subspecies, varieties, subvarieties, forms and subforms. There are recommended ways to abbreviate the rank markers in these cases but some variation can be expected. > > Of course aggregators should be providing more robust services for accessing exactly what you want in a consistent, predictable way and I would suggest that the best place to attack the problem is to define exactly what a typical user needs to see and then for GBIF and similar projects to work on delivering predictable data downloads and web services that clean out all of these nomenclatural inconsistencies - and perhaps also add value in other ways such as augmenting the data with associated environmental values (as the Atlas of Living Australia does). This would allow us all to work together on developing a consistent and predictable algorithm for handling interpretation of name strings, including synonymy, misspellings, virus names and everything else that makes this such a difficult problem. > > Best wishes, > > Donald > > ---------------------------------------------------------------------- > Donald Hobern - GBIF Director - dhobern(a)gbif.org > Global Biodiversity Information Facility http://www.gbif.org/ > GBIF Secretariat, Universitetsparken 15, DK-2100 Copenhagen Ø, Denmark > Tel: +45 3532 1471 Mob: +45 2875 1471 Fax: +45 2875 1480 > ---------------------------------------------------------------------- > > > -----Original Message----- > From: peter.desmet.cubc(a)gmail.com [mailto:peter.desmet.cubc@gmail.com] On Behalf Of Peter Desmet > Sent: Wednesday, March 14, 2012 3:41 PM > To: Tim Robertson [GBIF] > Cc: Donald Hobern (GBIF); dev Developers; TDWG content mailing list; TDWG TAG mailing list; Christian Gendreau > Subject: Re: Canonical name parsing > > Hi Tim, > > I agree, aggregators like GBIF and Canadensys will have to deal with clean and dirty data in each field anyway: they need code libraries to deal with this and it is good that these are being developed. But, that doesn't help someone who wants to use data from a Darwin Core Archive with his data in Excel or a Roderic Page who wants to get things done for a prototype. > Having to use Java libraries or even the Name Parser [1] (though both > great) is a barrier to data use. Darwin Core (Archives) is not only used for machine to machine interaction, humans use it too, and I think we should allow easy hacking (I mean this in the good sense), especially for something as important as the scientific name. > In addition, as a data publisher (e.g. for our VASCAN checklist) I > *do* have the information to provide a clean and simple to use canonicalScientificName, but I just can't share it via the otherwise excellent biodiversity sharing standard Darwin Core. I think that's a pity. > > Peter > > [1] http://tools.gbif.org/nameparser/ > [2] http://data.canadensys.net/vascan > > PS: Yes, Canadensys will use the GBIF interpretation libraries. Since we develop in Java as well, using those libraries is as easy as the proverbial "one line of code". We're looking forward in testing them and providing patches to enhance them. Open source FTW! :-) > > > On Wed, Mar 14, 2012 at 07:32, Tim Robertson [GBIF] <trobertson(a)gbif.org> wrote: > > Hi Peter, > > > > I'm replying off the TDWG list, since it is a bit of a tangent to your discussion. If you feel it is relevant, please CC the list again. > > > > At GBIF as you know, we have to interpret all kinds of quality of content. I tend to agree with Donald that this would not really help in consumption, as in my experience we will have to deal with both clean and dirty data in each field *anyway* when this is used at network scale. I would rather see us evolve the interpretation libraries to handle all the corner cases, which we need to develop anyway. We already do a pretty decent job at extracting canonicals. This is further enhanced when you couple the extracted canonical with a fuzzy match against the "authoritative names" we can now index thanks to the availability of checklists in DwC-A format. > > > > I know you are a Java shop. Are you using the GBIF interpretation libraries [1] at the moment? If not, is there a reason why you don't? > > They are used in all GBIF projects (portal, checklistbank etc), and the more we enhance them, the better it is for everyone. We have a significant test coverage [2,3] and there have been quite some man months (years?) spent already in their development and with some real regular expression experts (most notably Markus D. and Dave M.). All our work is Maven-ized, versioned and available in our Maven repository [4]. > > > > I hope these are interesting to you. We would welcome any patches to enhance them, or assistance in identifying the corner cases and capturing those as unit tests. > > > > Hope this helps, > > Tim > > > > [1] > > http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src > > /main/java/org/gbif/ecat/parser/NameParser.java > > [2] > > http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src > > /test/java/org/gbif/ecat/parser/NameParserTest.java > > [3] > > http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src > > /#src%2Ftest%2Fresources [4] > > http://repository.gbif.org/index.html#nexus-search;quick~ecat-common > > > > > > -- > Peter Desmet > Biodiversity Informatics Manager > Canadensys - www.canadensys.net > > Université de Montréal Biodiversity Centre > 4101 rue Sherbrooke est > Montreal, QC, H1X2B2 > Canada > > Phone: 514-343-6111 #82354 > Fax: 514-343-2288 > Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com > Skype: anderhalv > Public profile: http://www.linkedin.com/in/peterdesmet > > > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content -- Peter Desmet Biodiversity Informatics Manager Canadensys - www.canadensys.net Université de Montréal Biodiversity Centre 4101 rue Sherbrooke est Montreal, QC, H1X2B2 Canada Phone: 514-343-6111 #82354 Fax: 514-343-2288 Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com Skype: anderhalv Public profile: http://www.linkedin.com/in/peterdesmet

6 9

Re: [tdwg-tag] [tdwg-content] Canonical name parsing
by Peter Desmet 14 Mar '12

14 Mar '12

Thanks everyone for the feedback so far! Now, if you want to +1 the proposal, become a friend of Timon lepidus: https://plus.google.com/114672072317054763788/posts/Nph2ksggNZW :-) Peter On Wed, Mar 14, 2012 at 16:15, Dmitry Mozzherin <dmozzherin(a)eol.org> wrote: > mmm where is +1 button? :) > > On Wed, Mar 14, 2012 at 4:11 PM, Peter Desmet <peter.desmet(a)umontreal.ca> > wrote: > > Hi Paul, > > > > Higher taxon: "Magnoliidae Novák ex Takhtajan" (a subclass). > > - scientificName: Magnoliidae Novák ex Takhtajan > > - taxonRank: subclass > > But there are no terms to share the canonical name "Magnoliidae". The > only > > available options are kingdom, phylum, class, order, family, genus, > > subgenus, specificEpithet, infraspecificEpithet, none of which are > > appropriate. > > > > Solution: > > - canonicalScientificName: Magnoliidae > > > > Infrageneric taxon: "Abies sect. Amabilis (Matzenko) Farjon & Rushforth" > (a > > section) > > - scientificName: Abies sect. Amabilis (Matzenko) Farjon & Rushforth > > - taxonRank: section > > - genus: Abies > > But there are no terms to share "Abies Amabilis", "Abies sect. Amabilis", > > "Abies section Amabilis" or even "Amabilis". The only available options > are > > kingdom, phylum, class, order, family, genus, subgenus, specificEpithet, > > infraspecificEpithet, none of which are appropriate. Why we have > subgenus, > > but not infragenericEpithet is another issue. I would at least be able to > > share "Amabilis". > > > > Solution: > > - canonicalScientificName: Abies Amabilis > > - taxonRank: section > > > > Peter > > > > There is no place to share the canonical name "Magnoliidae" for this > taxon. > > > > On Wed, Mar 14, 2012 at 14:37, Paul Kirk <p.kirk(a)cabi.org> wrote: > >> > >> 'For higher taxa or infrageneric taxa, these terms are not sufficient' > ... > >> why? > >> > >> > >> > >> Paul > >> > >> > >> > >> ________________________________ > >> > >> From: tdwg-tag-bounces(a)lists.tdwg.org [tdwg-tag-bounces(a)lists.tdwg.org] > on > >> behalf of Peter Desmet [peter.desmet(a)umontreal.ca] > >> Sent: 14 March 2012 18:26 > >> To: Richard Pyle > >> Cc: TDWG content mailing list; Donald Hobern (GBIF); dev Developers; > >> Christian Gendreau; TDWG TAG mailing list > >> > >> Subject: Re: [tdwg-tag] [tdwg-content] Canonical name parsing > >> > >> Rich, > >> > >> I wished those terms were sufficient, but as mentioned in the > >> justification for > http://code.google.com/p/darwincore/issues/detail?id=150: > >> > >> genus, specificEpithet, infraspecificEpithet: concatenated, this terms > are > >> identical to the canonicalScientificName for genera, species and > >> infraspecific taxa. For higher taxa or infrageneric taxa, these terms > are > >> not sufficient. In addition, there is some ambiguity regarding the genus > >> definition: for synonyms, is it the accepted genus or the genus that is > part > >> of the synonym name? See: > >> http://lists.tdwg.org/pipermail/tdwg-content/2010-November/002052.html. > In > >> the former case, the genus cannot be used to concatenate a > >> canonicalScientificName. > >> > >> To give an example for a higher taxon: > >> scientificName: Magnoliidae Novák ex Takhtajan > >> taxonRank: subclass > >> > >> There is no place to share the canonical name "Magnoliidae" for this > >> taxon. > >> > >> Peter > >> > >> On Wed, Mar 14, 2012 at 14:13, Richard Pyle <deepreef(a)bishopmuseum.org> > >> wrote: > >>> > >>> > >>> I guess the parts that confuse me are: > >>> > >>> 1) What providers are able to produce a canonicalScientificName as per > >>> Peter’s definition, but are unable to provide the pre-parsed elements > of > >>> genus | subgenus | specificEpithet | infraspecificEpithet? > >>> > >>> 2) What consumers could make use of a canonicalScientificName as per > >>> Peter’s definition, but are unable to make (even better) use of the > >>> pre-parsed elements of genus | subgenus | specificEpithet | > >>> infraspecificEpithet? > >>> > >>> Aloha, > >>> Rich > >>> > >>> > >>> > >>> > >>> From: tdwg-content-bounces(a)lists.tdwg.org > >>> [mailto:tdwg-content-bounces@lists.tdwg.org] On Behalf Of Peter Desmet > >>> Sent: Wednesday, March 14, 2012 7:03 AM > >>> To: Donald Hobern (GBIF) > >>> Cc: TDWG content mailing list; Christian Gendreau; Tim Robertson > [GBIF]; > >>> TDWG TAG mailing list; dev Developers > >>> Subject: Re: [tdwg-content] Canonical name parsing > >>> > >>> Hi Donald, > >>> > >>> scientificName, with its current definition [1] is a great term and > >>> should be continued to used as such. As with most Darwin Core terms, it > >>> offers flexibility, so its not an impediment for publishing data. In > the > >>> GBIF context, this term is considered mandatory: records without it are > >>> ignored during indexing (I believe). All of this can stay. > >>> > >>> canonicalScientificName would be an additional term with a clear rule > >>> (see my proposed definition [2]). This is the case for other Darwin > Core > >>> terms as well, such as > >>> decimalLatitude [3], minimalElevationInMeters [4] or countryCode [5]. > >>> They serve as an ready-to-use addition/alternative to verbatimLatitude > [6], > >>> verbatimElevation [7] and country [8] respectively. These terms don't > stop > >>> anyone from publishing data, but data publishers who can provide this > kind > >>> of information have the choice to do so. It would be the same for > >>> canonicalScientificName. > >>> > >>> And yes, an aggregator like GBIF can play an important role in > providing > >>> consistent data to its users and figuring out what they really need, > but not > >>> all data is consumed that way. In addition, I hope a user would be > able to > >>> download cleaned data from the GBIF portal as Darwin Core. Wouldn't it > be > >>> nice that the parsed canonicalScientificName created by GBIF can be > provided > >>> in its proper term? There are users out there who want this! > >>> > >>> Regards, > >>> > >>> Peter > >>> > >>> [1] http://rs.tdwg.org/dwc/terms/index.htm#scientificName > >>> [2] http://code.google.com/p/darwincore/issues/detail?id=150 > >>> [3] http://rs.tdwg.org/dwc/terms/index.htm#decimalLatitude > >>> [4] http://rs.tdwg.org/dwc/terms/index.htm#minimumElevationInMeters > >>> [5] http://rs.tdwg.org/dwc/terms/index.htm#countryCode > >>> [6] http://rs.tdwg.org/dwc/terms/index.htm#verbatimLatitude > >>> [7] http://rs.tdwg.org/dwc/terms/index.htm#verbatimElevation > >>> [8] http://rs.tdwg.org/dwc/terms/index.htm#country > >>> > >>> On Wed, Mar 14, 2012 at 11:19, Donald Hobern (GBIF) <dhobern(a)gbif.org> > >>> wrote: > >>> > > >>> > Hi Peter. > >>> > > >>> > I certainly agree that aggregators only represent one use case here > >>> > but, having seen a lot of the mess of real-world data, I don't > believe that > >>> > simply adding a new term will fix this problem for the users you > describe. > >>> > To get the results you want, we would need a sufficiently large > majority of > >>> > data sets to follow the rules perfectly that we could ignore those > that were > >>> > non-conformant. This would mean we should mandate that every data > set must > >>> > use the new element (with or without the existing scientificName > element) > >>> > and that they must present scientific names in the expected way (or > else > >>> > have their data considered non-compliant). Until now, the philosophy > on > >>> > publishing Darwin Core data has been to make it as easy as possible > for data > >>> > providers to expose their data, even at the expense of greater > complexity > >>> > for consumers. I suspect that we would have a lot less data > available for > >>> > use now if we had taken a more stringent approach. > >>> > > >>> > In some ways, this proposal reminds me of the structures in ABCD > which > >>> > seek to offer users verbatim and more normalised ways to represent > several > >>> > types of information. This actually makes consuming all the > possible forms > >>> > of such data very complex, since a record may contain all variant > forms or > >>> > just any one of them. If multiple forms are available, which one > should be > >>> > considered the primary version? > >>> > > >>> > I suspect that things may also get complicated as soon as you discuss > >>> > botanical subspecies, varieties, subvarieties, forms and subforms. > There > >>> > are recommended ways to abbreviate the rank markers in these cases > but some > >>> > variation can be expected. > >>> > > >>> > Of course aggregators should be providing more robust services for > >>> > accessing exactly what you want in a consistent, predictable way and > I would > >>> > suggest that the best place to attack the problem is to define > exactly what > >>> > a typical user needs to see and then for GBIF and similar projects > to work > >>> > on delivering predictable data downloads and web services that clean > out all > >>> > of these nomenclatural inconsistencies - and perhaps also add value > in other > >>> > ways such as augmenting the data with associated environmental > values (as > >>> > the Atlas of Living Australia does). This would allow us all to work > >>> > together on developing a consistent and predictable algorithm for > handling > >>> > interpretation of name strings, including synonymy, misspellings, > virus > >>> > names and everything else that makes this such a difficult problem. > >>> > > >>> > Best wishes, > >>> > > >>> > Donald > >>> > > >>> > > ---------------------------------------------------------------------- > >>> > Donald Hobern - GBIF Director - dhobern(a)gbif.org > >>> > Global Biodiversity Information Facility http://www.gbif.org/ > >>> > GBIF Secretariat, Universitetsparken 15, DK-2100 Copenhagen Ø, > Denmark > >>> > Tel: +45 3532 1471 Mob: +45 2875 1471 Fax: +45 2875 1480 > >>> > > ---------------------------------------------------------------------- > >>> > > >>> > > >>> > -----Original Message----- > >>> > From: peter.desmet.cubc(a)gmail.com [mailto: > peter.desmet.cubc(a)gmail.com] > >>> > On Behalf Of Peter Desmet > >>> > Sent: Wednesday, March 14, 2012 3:41 PM > >>> > To: Tim Robertson [GBIF] > >>> > Cc: Donald Hobern (GBIF); dev Developers; TDWG content mailing list; > >>> > TDWG TAG mailing list; Christian Gendreau > >>> > Subject: Re: Canonical name parsing > >>> > > >>> > Hi Tim, > >>> > > >>> > I agree, aggregators like GBIF and Canadensys will have to deal with > >>> > clean and dirty data in each field anyway: they need code libraries > to deal > >>> > with this and it is good that these are being developed. But, that > doesn't > >>> > help someone who wants to use data from a Darwin Core Archive with > his data > >>> > in Excel or a Roderic Page who wants to get things done for a > prototype. > >>> > Having to use Java libraries or even the Name Parser [1] (though both > >>> > great) is a barrier to data use. Darwin Core (Archives) is not only > >>> > used for machine to machine interaction, humans use it too, and I > think we > >>> > should allow easy hacking (I mean this in the good sense), > especially for > >>> > something as important as the scientific name. > >>> > In addition, as a data publisher (e.g. for our VASCAN checklist) I > >>> > *do* have the information to provide a clean and simple to use > >>> > canonicalScientificName, but I just can't share it via the otherwise > >>> > excellent biodiversity sharing standard Darwin Core. I think that's > a pity. > >>> > > >>> > Peter > >>> > > >>> > [1] http://tools.gbif.org/nameparser/ > >>> > [2] http://data.canadensys.net/vascan > >>> > > >>> > PS: Yes, Canadensys will use the GBIF interpretation libraries. Since > >>> > we develop in Java as well, using those libraries is as easy as the > >>> > proverbial "one line of code". We're looking forward in testing them > and > >>> > providing patches to enhance them. Open source FTW! :-) > >>> > > >>> > > >>> > On Wed, Mar 14, 2012 at 07:32, Tim Robertson [GBIF] > >>> > <trobertson(a)gbif.org> wrote: > >>> > > Hi Peter, > >>> > > > >>> > > I'm replying off the TDWG list, since it is a bit of a tangent to > >>> > > your discussion. If you feel it is relevant, please CC the list > again. > >>> > > > >>> > > At GBIF as you know, we have to interpret all kinds of quality of > >>> > > content. I tend to agree with Donald that this would not really > help in > >>> > > consumption, as in my experience we will have to deal with both > clean and > >>> > > dirty data in each field *anyway* when this is used at network > scale. I > >>> > > would rather see us evolve the interpretation libraries to handle > all the > >>> > > corner cases, which we need to develop anyway. We already do a > pretty > >>> > > decent job at extracting canonicals. This is further enhanced > when you > >>> > > couple the extracted canonical with a fuzzy match against the > "authoritative > >>> > > names" we can now index thanks to the availability of checklists > in DwC-A > >>> > > format. > >>> > > > >>> > > I know you are a Java shop. Are you using the GBIF interpretation > >>> > > libraries [1] at the moment? If not, is there a reason why you > don't? > >>> > > They are used in all GBIF projects (portal, checklistbank etc), and > >>> > > the more we enhance them, the better it is for everyone. We have a > >>> > > significant test coverage [2,3] and there have been quite some man > months > >>> > > (years?) spent already in their development and with some real > regular > >>> > > expression experts (most notably Markus D. and Dave M.). All our > work is > >>> > > Maven-ized, versioned and available in our Maven repository [4]. > >>> > > > >>> > > I hope these are interesting to you. We would welcome any patches > to > >>> > > enhance them, or assistance in identifying the corner cases and > capturing > >>> > > those as unit tests. > >>> > > > >>> > > Hope this helps, > >>> > > Tim > >>> > > > >>> > > [1] > >>> > > > >>> > > > http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src > >>> > > /main/java/org/gbif/ecat/parser/NameParser.java > >>> > > [2] > >>> > > > >>> > > > http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src > >>> > > /test/java/org/gbif/ecat/parser/NameParserTest.java > >>> > > [3] > >>> > > > >>> > > > http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src > >>> > > /#src%2Ftest%2Fresources [4] > >>> > > > http://repository.gbif.org/index.html#nexus-search;quick~ecat-common > >>> > > > >>> > > >>> > > >>> > > >>> > -- > >>> > Peter Desmet > >>> > Biodiversity Informatics Manager > >>> > Canadensys - www.canadensys.net > >>> > > >>> > Université de Montréal Biodiversity Centre > >>> > 4101 rue Sherbrooke est > >>> > Montreal, QC, H1X2B2 > >>> > Canada > >>> > > >>> > Phone: 514-343-6111 #82354 > >>> > Fax: 514-343-2288 > >>> > Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com > >>> > Skype: anderhalv > >>> > Public profile: http://www.linkedin.com/in/peterdesmet > >>> > > >>> > > >>> > _______________________________________________ > >>> > tdwg-content mailing list > >>> > tdwg-content(a)lists.tdwg.org > >>> > http://lists.tdwg.org/mailman/listinfo/tdwg-content > >>> > >>> > >>> > >>> > >>> -- > >>> Peter Desmet > >>> Biodiversity Informatics Manager > >>> Canadensys - www.canadensys.net > >>> > >>> Université de Montréal Biodiversity Centre > >>> 4101 rue Sherbrooke est > >>> Montreal, QC, H1X2B2 > >>> Canada > >>> > >>> Phone: 514-343-6111 #82354 > >>> Fax: 514-343-2288 > >>> Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com > >>> Skype: anderhalv > >>> Public profile: http://www.linkedin.com/in/peterdesmet > >>> > >>> > >>> This message is only intended for the addressee named above. Its > >>> contents may be privileged or otherwise protected. Any unauthorized > use, > >>> disclosure or copying of this message or its contents is prohibited. > If you > >>> have received this message by mistake, please notify us immediately by > reply > >>> mail or by collect telephone call. Any personal opinions expressed in > this > >>> message do not necessarily represent the views of the Bishop Museum. > >>> _______________________________________________ > >>> tdwg-content mailing list > >>> tdwg-content(a)lists.tdwg.org > >>> http://lists.tdwg.org/mailman/listinfo/tdwg-content > >> > >> > >> > >> > >> -- > >> Peter Desmet > >> Biodiversity Informatics Manager > >> Canadensys - www.canadensys.net > >> > >> Université de Montréal Biodiversity Centre > >> 4101 rue Sherbrooke est > >> Montreal, QC, H1X2B2 > >> Canada > >> > >> Phone: 514-343-6111 #82354 > >> Fax: 514-343-2288 > >> Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com > >> Skype: anderhalv > >> Public profile: http://www.linkedin.com/in/peterdesmet > >> > >> P Think Green - don't print this email unless you really need to > >> > >> ************************************************************************ > >> The information contained in this e-mail and any files transmitted with > it > >> is confidential and is for the exclusive use of the intended recipient. > If > >> you are not the intended recipient please note that any distribution, > >> copying or use of this communication or the information in it is > >> prohibited. > >> > >> Whilst CAB International trading as CABI takes steps to prevent the > >> transmission of viruses via e-mail, we cannot guarantee that any e-mail > or > >> attachment is free from computer viruses and you are strongly advised to > >> undertake your own anti-virus precautions. > >> > >> If you have received this communication in error, please notify us by > >> e-mail at cabi(a)cabi.org or by telephone on +44 (0)1491 832111 and then > >> delete the e-mail and any copies of it. > >> > >> CABI is an International Organization recognised by the UK Government > >> under Statutory Instrument 1982 No. 1071... > >> > >> > ************************************************************************** > >> > >> > >> _______________________________________________ > >> tdwg-tag mailing list > >> tdwg-tag(a)lists.tdwg.org > >> http://lists.tdwg.org/mailman/listinfo/tdwg-tag > >> > > > > > > > > -- > > Peter Desmet > > Biodiversity Informatics Manager > > Canadensys - www.canadensys.net > > > > Université de Montréal Biodiversity Centre > > 4101 rue Sherbrooke est > > Montreal, QC, H1X2B2 > > Canada > > > > Phone: 514-343-6111 #82354 > > Fax: 514-343-2288 > > Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com > > Skype: anderhalv > > Public profile: http://www.linkedin.com/in/peterdesmet > > > > _______________________________________________ > > tdwg-content mailing list > > tdwg-content(a)lists.tdwg.org > > http://lists.tdwg.org/mailman/listinfo/tdwg-content > > > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content > -- Peter Desmet Biodiversity Informatics Manager Canadensys - www.canadensys.net Université de Montréal Biodiversity Centre 4101 rue Sherbrooke est Montreal, QC, H1X2B2 Canada Phone: 514-343-6111 #82354 Fax: 514-343-2288 Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com Skype: anderhalv Public profile: http://www.linkedin.com/in/peterdesmet

1 0

Re: [tdwg-tag] canonicalScientificName
by Peter Desmet 14 Mar '12

14 Mar '12

Sorry, forgot to include TAG mailing list. On Tue, Mar 13, 2012 at 15:27, Peter Desmet <peter.desmet(a)umontreal.ca> wrote: > Hi everyone, > > In a recent tweet [1] Roderic Page reminded me that in Darwin Core, we > don't have a ready-to-use scientificName field. The definition for > scientificName [2] asks for the full verbose name, including authors. > I think this is a good definition (see below), but it also means that > in a lot of use cases, names need to be parsed before they can be used > or matched. I am currently helping collections publish their data for > Candensys [3] and as a data producer I am happy we can provide all the > information we have in scientificName, but as a data user, I get > frustrated every time I see those long verbose botanical names with > multiple authors. I am convinced that our data would be more usable if > we had an additional canonicalScientificName term. > > Which is why I am now officially requesting it on the Darwin Core code > site: http://code.google.com/p/darwincore/issues/detail?id=150 (see > below). This has been discussed in detail before [4], but no consensus > was reached. I hope we can get our act together this time! > > Regards, > > Peter Desmet > > -- > > [1] https://twitter.com/#!/rustyrussell22/status/179500954901692417 > [2] http://rs.tdwg.org/dwc/terms/index.htm#scientificName > [3] http://www.canadensys.net > [4] http://lists.tdwg.org/pipermail/tdwg-content/2010-November/thread.html#1976 > > > ==New Term Recommendation== > Submitter: Peter Desmet > > Justification: The scientific name is probably the most used element > of an occurrence/taxon, but currently Darwin Core does not provide a > single ready-to-use-field for this. A canonicalScientificName with the > scientific name as a uninomial, binomial or trinomial could solve this > problem. > The current terms are not sufficient: > - scientificName: verbose, used to record all components of a > scientific name (if available), including authorship(s) and > rankmarker(s). It is critical to keep this definition, as this term is > sometimes the only place to share certain information, e.g.: > quadrinomials, intermediate botanical authors, hybrid formulas, etc. > The disadvantage of only having this verbose notation is that the user > needs to parse the name before he/she can use or match it. > - genus, specificEpithet, infraspecificEpithet: concatenated, this > terms are identical to the canonicalScientificName for genera, species > and infraspecific taxa. For higher taxa or infrageneric taxa, these > terms are not sufficient. In addition, there is some ambiguity > regarding the genus definition: for synonyms, is it the accepted genus > or the genus that is part of the synonym name? See: > http://lists.tdwg.org/pipermail/tdwg-content/2010-November/002052.html. > In the former case, the genus cannot be used to concatenate a > canonicalScientificName. > > The need for this term has been discussed thoroughly already (see: > http://lists.tdwg.org/pipermail/tdwg-content/2010-November/thread.html#1976), > but no consensus was reached. I'd like to reopen the discussion and I > hope that a consensus can be reached quickly, so our data can be used > more easily. > > Definition: The scientific name as a uninomial, binomial or trinomial. > When forming part of an Identification, this should be the name in > lowest level taxonomic rank that can be determined. This term should > not contain authorship(s), rankmarker(s) or identification > qualifications. If the scientific name cannot be expressed as a uni-, > bi- or trinomial (e.g. hybrid formulas), do not use this term (use > scientificName instead). > > Comment: Examples: "Carex" (genus), "Vulpes vulpes" (species), > "Anaphalis margaritacea occidentalis" (plant variety) > > Refines: > > Has Domain: > > Has Range: > > Replaces: > > ABCD 2.06: > > -- > Peter Desmet > Biodiversity Informatics Manager > Canadensys - www.canadensys.net > > Université de Montréal Biodiversity Centre > 4101 rue Sherbrooke est > Montreal, QC, H1X2B2 > Canada > > Phone: 514-343-6111 #82354 > Fax: 514-343-2288 > Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com > Skype: anderhalv > Public profile: http://www.linkedin.com/in/peterdesmet

5 5

Re: [tdwg-tag] [tdwg-content] canonicalScientificName
by Peter Desmet 14 Mar '12

14 Mar '12

Hi Donald, I just don't understand why an additional field canonicalScientificName (much like minimumElevationInMeters or countryCode) would create or shift the problem. It has a clear and easy to understand definition [1]. As a data provider I can assess if its too much of a bother to populate the field in addition to the more lenient scientificName (just like countryCode, etc.) and as a data user I can complain or ignore the data if I see the provider didn't follow the definition. This discussion resurfaces every time: clearly users would like to have this. So why do we make it difficult for those users and the data providers who can provide it? Peter [1] http://code.google.com/p/darwincore/issues/detail?id=150 I could even add "[...] it will contain one, two or three words" (see email Jessie Kennedy) PS: Yes, some people probably interpret scientificName as "Give us the cleanest representation of the scientific name you have", but that is definitely not the Darwin Core definition. It's more "give us the most verbose representation of the scientific name you have", which is what I use and what I advise all data publishers in my network to use. The difference doesn't matter though, the important thing is that a data user cannot expect this term to be a canonical representation of the name. On Wed, Mar 14, 2012 at 13:36, Donald Hobern (GBIF) <dhobern(a)gbif.org>wrote: > Hi Rod.**** > > ** ** > > Given the two alternatives you offer, I fully agree. However (and I may > be very wrong on this), I don’t actually believe that what you describe is > what is happening in most cases. I believe that the scientificName field > should always be the “give us the cleanest representation of the scientific > name you have” and we could certainly provide a more useful set of > priorities for how that is defined. Regardless of what the DwC guidelines > say, I think many data providers simply map their most closely aligned > database column into the DwC view and we get whatever that happens to be, > with whatever authorship it may contain.**** > > ** ** > > I’d agree that the clean name is much closer to what a web-enabled > linked-data world needs and would happily endorse a move to make that the > recommended form. I just honestly believe that many providers will always > give us something different. Adding a new field will just shift the > problem.**** > > ** ** > > At least that’s my perspective…**** > > ** ** > > All the best,**** > > ** ** > > Donald**** > > ** ** > > ** ** > > ----------------------------------------------------------------------**** > > Donald Hobern - GBIF Director - dhobern(a)gbif.org **** > > Global Biodiversity Information Facility http://www.gbif.org/ **** > > GBIF Secretariat, Universitetsparken 15, DK-2100 Copenhagen Ø, Denmark**** > > Tel: +45 3532 1471 Mob: +45 2875 1471 Fax: +45 2875 1480**** > > ----------------------------------------------------------------------**** > > ** ** > > *From:* tdwg-content-bounces(a)lists.tdwg.org [mailto: > tdwg-content-bounces(a)lists.tdwg.org] *On Behalf Of *Roderic Page > *Sent:* Wednesday, March 14, 2012 5:50 PM > *To:* TDWG Content Mailing List > *Subject:* Re: [tdwg-content] canonicalScientificName**** > > ** ** > > Dear Donald,**** > > ** ** > > I couldn't disagree more!**** > > ** ** > > It seems to me that this is one case where needs of consumers and > providers align pretty well.**** > > ** ** > > If I'm publishing data I want to avoid hassle, and one hassle is finding > the taxonomic authorities for names. Then there is the issue of how to > write the authority. There are so many variables: do I include diacritic > characters? is the person's name abbreviated? what is the correct date? > should I use parentheses? should I use commas? If I can just publish the > canonical name life is simpler.**** > > ** ** > > As a consumer I can't trust people to get the authority right. Publishers > get the taxonomic names wrong, and they will certainly make a mess of the > authority.**** > > ** ** > > So, if we mandate clean names we are saying to providers "give me this"*** > * > > ** ** > > <taxonomic name>**** > > [some scope for crap]**** > > ** ** > > Instead, we've mandated "give me this"**** > > ** ** > > <taxonomic name> + <authority>**** > > [some scope for crap] + [huge scope for crap]**** > > ** ** > > Why? Why would we do this to ourselves? Why do we think it's OK to have > databases full of duplicates such as these (from the ION database)?:**** > > ** ** > > Pseudopaludicola Miranda Ribeiro 1926**** > > Pseudopaludicola Mir. Ribeiro 1926**** > > Pseudopaludicola Miranda-Ribeiro 1926**** > > ** ** > > One consequence of this is that we have projects like > http://globalnames.org project, which is essentially collecting endless > variations on authority strings. In other words, trying to clean up a mess > essentially of our own making. **** > > ** ** > > By all means have a field for taxonomic authority, but keep that separate > from the canonical taxonomic name. In the real world, the canonical name is > what people use. If we want people to make data available, make it simple. > If we want people to use data make it simple.**** > > ** ** > > Regards**** > > ** ** > > Rod**** > > ** ** > > ** ** > > ** ** > > On 14 Mar 2012, at 11:16, Donald Hobern wrote:**** > > > > **** > > Hi Peter.**** > > ** ** > > I certainly sympathise with the desire for a readily-consumed naked > scientific name field. However, unless the canonicalScientificName element > is enforced as a mandatory field (which would in itself impact some data > publishers and may prevent them validly sharing their data without extra > work to provide clean scientific names), it will be yet another element > which data consumers must check. If canonicalScientificName is supplied, > consumers will still need to handle cases where it is malformed. If is not > supplied, they will need to ignore the record or else do precisely what > they do today with the scientificName field. **** > > ** ** > > I therefore worry that adding this field could in fact make the task more > complex, rather than simpler, for data consumers.**** > > ** ** > > Thanks,**** > > ** ** > > Donald**** > > **** > > ----------------------------------------------------------------------**** > > Donald Hobern - GBIF Director - dhobern(a)gbif.org**** > > Global Biodiversity Information Facility http://www.gbif.org/**** > > GBIF Secretariat, Universitetsparken 15, DK-2100 Copenhagen Ø, Denmark**** > > Tel: +45 3532 1471 Mob: +45 2875 1471 Fax: +45 2875 1480**** > > ----------------------------------------------------------------------**** > > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content**** > > ** ** > > --------------------------------------------------------- > Roderic Page > Professor of Taxonomy > Institute of Biodiversity, Animal Health and Comparative Medicine > College of Medical, Veterinary and Life Sciences > Graham Kerr Building > University of Glasgow > Glasgow G12 8QQ, UK > > Email: r.page(a)bio.gla.ac.uk > Tel: +44 141 330 4778 > Fax: +44 141 330 2792**** > > Skype: rdmpage > AIM: rodpage1962(a)aim.com > Facebook: http://www.facebook.com/profile.php?id=1112517192 > Twitter: http://twitter.com/rdmpage > Blog: http://iphylo.blogspot.com > Home page: http://taxonomy.zoology.gla.ac.uk/rod/rod.html**** > > ** ** > > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content > > -- Peter Desmet Biodiversity Informatics Manager Canadensys - www.canadensys.net Université de Montréal Biodiversity Centre 4101 rue Sherbrooke est Montreal, QC, H1X2B2 Canada Phone: 514-343-6111 #82354 Fax: 514-343-2288 Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com Skype: anderhalv Public profile: http://www.linkedin.com/in/peterdesmet

1 0

Re: [tdwg-tag] Canonical name parsing
by Peter Desmet 14 Mar '12

14 Mar '12

Hi Tim, I agree, aggregators like GBIF and Canadensys will have to deal with clean and dirty data in each field anyway: they need code libraries to deal with this and it is good that these are being developed. But, that doesn't help someone who wants to use data from a Darwin Core Archive with his data in Excel or a Roderic Page who wants to get things done for a prototype. Having to use Java libraries or even the Name Parser [1] (though both great) is a barrier to data use. Darwin Core (Archives) is not only used for machine to machine interaction, humans use it too, and I think we should allow easy hacking (I mean this in the good sense), especially for something as important as the scientific name. In addition, as a data publisher (e.g. for our VASCAN checklist) I *do* have the information to provide a clean and simple to use canonicalScientificName, but I just can't share it via the otherwise excellent biodiversity sharing standard Darwin Core. I think that's a pity. Peter [1] http://tools.gbif.org/nameparser/ [2] http://data.canadensys.net/vascan PS: Yes, Canadensys will use the GBIF interpretation libraries. Since we develop in Java as well, using those libraries is as easy as the proverbial "one line of code". We're looking forward in testing them and providing patches to enhance them. Open source FTW! :-) On Wed, Mar 14, 2012 at 07:32, Tim Robertson [GBIF] <trobertson(a)gbif.org> wrote: > Hi Peter, > > I'm replying off the TDWG list, since it is a bit of a tangent to your discussion. If you feel it is relevant, please CC the list again. > > At GBIF as you know, we have to interpret all kinds of quality of content. I tend to agree with Donald that this would not really help in consumption, as in my experience we will have to deal with both clean and dirty data in each field *anyway* when this is used at network scale. I would rather see us evolve the interpretation libraries to handle all the corner cases, which we need to develop anyway. We already do a pretty decent job at extracting canonicals. This is further enhanced when you couple the extracted canonical with a fuzzy match against the "authoritative names" we can now index thanks to the availability of checklists in DwC-A format. > > I know you are a Java shop. Are you using the GBIF interpretation libraries [1] at the moment? If not, is there a reason why you don't? > They are used in all GBIF projects (portal, checklistbank etc), and the more we enhance them, the better it is for everyone. We have a significant test coverage [2,3] and there have been quite some man months (years?) spent already in their development and with some real regular expression experts (most notably Markus D. and Dave M.). All our work is Maven-ized, versioned and available in our Maven repository [4]. > > I hope these are interesting to you. We would welcome any patches to enhance them, or assistance in identifying the corner cases and capturing those as unit tests. > > Hope this helps, > Tim > > [1] http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src/main… > [2] http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src/test… > [3] http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/src/#src… > [4] http://repository.gbif.org/index.html#nexus-search;quick~ecat-common > -- Peter Desmet Biodiversity Informatics Manager Canadensys - www.canadensys.net Université de Montréal Biodiversity Centre 4101 rue Sherbrooke est Montreal, QC, H1X2B2 Canada Phone: 514-343-6111 #82354 Fax: 514-343-2288 Email: peter.desmet(a)umontreal.ca / peter.desmet.cubc(a)gmail.com Skype: anderhalv Public profile: http://www.linkedin.com/in/peterdesmet

1 0