October 2015 - tdwg-content

Re: [tdwg-content] [IPT] Reverting the process of DwC standardization
by Menashe' Eliezer 29 Oct '15

29 Oct '15

Hello, Resending the same message due to a subscription problem. -- Menashè 2015-10-29 12:15 GMT+01:00 Menashe' Eliezer <menashe.eliezer(a)gmail.com>: > Hello, > Please see my updated suggestion at > https://github.com/gbif/ipt/issues/1165 > IMHO Open Refine is not the right tool. One can simply use org.apache.poi > in his Java application for reading all the information from the different > files inside the DwC, and create an ODS file with the combined matrix, > which takes into consideration also possible parentEventID. I'm sorry I > don't have time to do it myself. > I hope it's clear. > -- > Menashè > > > 2015-10-28 18:57 GMT+01:00 Shorthouse, David < > david.shorthouse(a)umontreal.ca>: > >> All, >> >> Is part of the issue being expressed here because the raw ecological data >> sets we're discussing are small-ish matrices rather than occurrences, with >> site codes as columns, taxa as rows and measures of density/abundance as >> cells (and similar for environmental variables)? Such structures are often >> used as input for software that executes eg ordinations, classification & >> regression trees, species richness estimates. The shortcoming of such a >> structure is the inherent idiosyncratic nature of "site codes", with >> variable numbers of them, i.e. an arbitrary number of columns. I doubt it >> was ever designed for ease of dataset integration, but rather for ease of >> computation. Representing this structure as Event core requires significant >> transposition & potential for error if it were manual. Open Refine is one >> such tool that could permit bi-directional transpositions (DwC -> matrix >> and then matrix -> DwC), but it is still clunky and accommodation of >> extensions is virtually non-existent. But, perhaps Open Refine recipes and >> guides gets us one step closer to finding a balance between the need for >> standardized representation & efficient transport (DwC) vs. end-users who >> want matrices for ease of computation. >> >> David P. Shorthouse >> >> On Tue, Oct 27, 2015 at 7:36 AM, David Valentim Dias <dvdias(a)sibbr.gov.br >> > wrote: >> >>> Hi again, >>> >>> I think the problem target both. DwC because is a solution to a problem >>> creating another problem to researchers less "skilled" in table >>> manipulation. Ecological data with occurrence is resulting in three tables >>> and manipulation of these are getting harder with the number of core or >>> extensions used. >>> Two possible solutions comes in mind: create a new term describing the >>> original layout of the columns (so we can use csvjoin like Menashe suggest) >>> or ipt with option to store the original table associated with resource. >>> We can always use external links in eml and save the file somewhere but >>> this means creating another service and managing more login (aka resource >>> cost and new problems). >>> >>> I think any solution will need ipt changes. >>> >>> 2015-10-27 9:08 GMT-02:00 Menashe' Eliezer <menashe.eliezer(a)gmail.com>: >>> >>>> Hi Tim, >>>> I believe that the IPT feature I've requested long ago could be helpful >>>> for David: https://github.com/gbif/ipt/issues/1165 >>>> Consumers and also the data providers don't have a DwC-A viewer, and >>>> they need to join the separate csv files for having one table in a >>>> worksheet. >>>> Web applications like the one at OBIS website do let end users download >>>> one big table. >>>> >>>> Best regards, >>>> Menashè >>>> >>>> >>>> 2015-10-27 9:53 GMT+01:00 Tim Robertson <trobertson(a)gbif.org>: >>>> >>>>> Hi David >>>>> (CC’ing the IPT list as this might be an IPT specific thread - >>>>> http://lists.gbif.org/mailman/listinfo/ipt) >>>>> >>>>> For clarification - is your question specific to the DwC-A standard >>>>> which is possible as Alex says or is it specific to the IPT tool please? >>>>> >>>>> Do you imagine a scenario where you’d effectively map the same >>>>> extension 2 times - once to interpreted and once to verbatim - or do you >>>>> envisage a different data schema for each? >>>>> >>>>> Thanks, >>>>> Tim >>>>> >>>>> >>>>> >>>>> On 23 Oct 2015, at 16:00, Alex Thompson <godfoder(a)acis.ufl.edu> wrote: >>>>> >>>>> David, >>>>> >>>>> It's certainly possible, within the context of a Darwin Core Archive, >>>>> to include other files within the ZIP file that lie outside the schema of >>>>> the archive. Both GBIF and iDigBio do this when generating downloads for >>>>> various reasons (RIGHTS & LICENSE files, additional EML metadata, etc). >>>>> However, I do not believe it is possible to do this within IPT. You might >>>>> submit an issue on the IPT issue tracker ( >>>>> https://github.com/gbif/ipt/issues) for potential inclusion of this >>>>> feature in a future version of IPT. >>>>> >>>>> There are workarounds you can use to include additional data in Darwin >>>>> Core archives, but none of them will exactly match your old format. For >>>>> instance, including an additional Occurrence file with the values as JSON >>>>> in dynamicProperties or in some other verbatim format in the >>>>> occurrenceRemarks field. Both of those would at least give some method of >>>>> single-row access (vs joining multiple measurementOrFacts to a single event >>>>> id) if that is the primary concern, even if they would require additional >>>>> parsing steps to be useful. >>>>> >>>>> Alex Thompson >>>>> iDigBio Infrastructure >>>>> >>>>> >>>>> On 10/23/2015 09:40 AM, David Valentim Dias wrote: >>>>> >>>>> Dear colleagues, >>>>> >>>>> Here on SiBBr we're using the new eventCore and measurementOrFacts and >>>>> after the process of standardization to DwC and publishing we think some >>>>> users/researchers will want the "original" table format because of multiple >>>>> reasons. >>>>> >>>>> Is possible to have a vertabimTable or some place where we can store >>>>> the original table/column format? >>>>> >>>>> Regards >>>>> >>>>> >> _______________________________________________ >> IPT mailing list >> IPT(a)lists.gbif.org >> http://lists.gbif.org/mailman/listinfo/ipt >> >> >

1 0

Re: [tdwg-content] [IPT] Reverting the process of DwC standardization
by David Valentim Dias 28 Oct '15

28 Oct '15

Hi again, I think the problem target both. DwC because is a solution to a problem creating another problem to researchers less "skilled" in table manipulation. Ecological data with occurrence is resulting in three tables and manipulation of these are getting harder with the number of core or extensions used. Two possible solutions comes in mind: create a new term describing the original layout of the columns (so we can use csvjoin like Menashe suggest) or ipt with option to store the original table associated with resource. We can always use external links in eml and save the file somewhere but this means creating another service and managing more login (aka resource cost and new problems). I think any solution will need ipt changes. 2015-10-27 9:08 GMT-02:00 Menashe' Eliezer <menashe.eliezer(a)gmail.com>: > Hi Tim, > I believe that the IPT feature I've requested long ago could be helpful > for David: https://github.com/gbif/ipt/issues/1165 > Consumers and also the data providers don't have a DwC-A viewer, and they > need to join the separate csv files for having one table in a worksheet. > Web applications like the one at OBIS website do let end users download > one big table. > > Best regards, > Menashè > > > 2015-10-27 9:53 GMT+01:00 Tim Robertson <trobertson(a)gbif.org>: > >> Hi David >> (CC’ing the IPT list as this might be an IPT specific thread - >> http://lists.gbif.org/mailman/listinfo/ipt) >> >> For clarification - is your question specific to the DwC-A standard which >> is possible as Alex says or is it specific to the IPT tool please? >> >> Do you imagine a scenario where you’d effectively map the same extension >> 2 times - once to interpreted and once to verbatim - or do you envisage a >> different data schema for each? >> >> Thanks, >> Tim >> >> >> >> On 23 Oct 2015, at 16:00, Alex Thompson <godfoder(a)acis.ufl.edu> wrote: >> >> David, >> >> It's certainly possible, within the context of a Darwin Core Archive, to >> include other files within the ZIP file that lie outside the schema of the >> archive. Both GBIF and iDigBio do this when generating downloads for >> various reasons (RIGHTS & LICENSE files, additional EML metadata, etc). >> However, I do not believe it is possible to do this within IPT. You might >> submit an issue on the IPT issue tracker ( >> https://github.com/gbif/ipt/issues) for potential inclusion of this >> feature in a future version of IPT. >> >> There are workarounds you can use to include additional data in Darwin >> Core archives, but none of them will exactly match your old format. For >> instance, including an additional Occurrence file with the values as JSON >> in dynamicProperties or in some other verbatim format in the >> occurrenceRemarks field. Both of those would at least give some method of >> single-row access (vs joining multiple measurementOrFacts to a single event >> id) if that is the primary concern, even if they would require additional >> parsing steps to be useful. >> >> Alex Thompson >> iDigBio Infrastructure >> >> >> On 10/23/2015 09:40 AM, David Valentim Dias wrote: >> >> Dear colleagues, >> >> Here on SiBBr we're using the new eventCore and measurementOrFacts and >> after the process of standardization to DwC and publishing we think some >> users/researchers will want the "original" table format because of multiple >> reasons. >> >> Is possible to have a vertabimTable or some place where we can store the >> original table/column format? >> >> Regards >> >> -- >> >> >> >> [image: ass_sibbr] >> >> *David Valentim Dias* >> >> *Biodiversity Data Management* >> >> SiBBr | MCTI | PNUMA (UNEP) >> >> Phone: +55 61 3329 6045 >> Phone: +55 61 9359 6151 >> Skype: ctenidae >> >> dvdias(a)sibbr.gov.br >> >> www.sibbr.gov.br >> >> >> _______________________________________________ >> tdwg-content mailing listtdwg-content@lists.tdwg.orghttp://lists.tdwg.org/mailman/listinfo/tdwg-content >> >> >> _______________________________________________ >> tdwg-content mailing list >> tdwg-content(a)lists.tdwg.org >> http://lists.tdwg.org/mailman/listinfo/tdwg-content >> >> >> >> _______________________________________________ >> IPT mailing list >> IPT(a)lists.gbif.org >> http://lists.gbif.org/mailman/listinfo/ipt >> >> > > _______________________________________________ > IPT mailing list > IPT(a)lists.gbif.org > http://lists.gbif.org/mailman/listinfo/ipt > > -- [image: ass_sibbr] *David Valentim Dias* *Biodiversity Data Management* SiBBr | MCTI | PNUMA (UNEP) Phone: +55 61 3329 6045 Phone: +55 61 9359 6151 Skype: ctenidae dvdias(a)sibbr.gov.br www.sibbr.gov.br

2 1

Reverting the process of DwC standardization
by David Valentim Dias 27 Oct '15

27 Oct '15

Dear colleagues, Here on SiBBr we're using the new eventCore and measurementOrFacts and after the process of standardization to DwC and publishing we think some users/researchers will want the "original" table format because of multiple reasons. Is possible to have a vertabimTable or some place where we can store the original table/column format? Regards -- [image: ass_sibbr] *David Valentim Dias* *Biodiversity Data Management* SiBBr | MCTI | PNUMA (UNEP) Phone: +55 61 3329 6045 Phone: +55 61 9359 6151 Skype: ctenidae dvdias(a)sibbr.gov.br www.sibbr.gov.br

3 2

Use of taxonomic concept labels in a revision
by Nico Franz 22 Oct '15

22 Oct '15

Dear TDWGers: This revision uses taxonomic concept labels to integrate past, currently, and presumably future (differing) views on a (so-perceived) group of weevils. Part of an effort (however partial) to concretize solutions to some well known challenges. http://zookeys.pensoft.net/articles.php?id=6001 Cheers, Nico

1 0

Re: [tdwg-content] status of uBio
by Chuck Miller 16 Oct '15

16 Oct '15

I’d be glad to host it here as well. Chuck From: jonathan.rees(a)gmail.com [mailto:jonathan.rees@gmail.com] On Behalf Of Jonathan A Rees Sent: Thursday, October 15, 2015 11:39 AM To: Chuck Miller Cc: Steve Baskauf; tdwg-content(a)lists.tdwg.org Subject: Re: [tdwg-content] status of uBio Even just providing a dump file of it somewhere would be a big advance over the current situation. I'd be happy to host it, if I can get ahold of it (of course assuming the blessing of whomever is responsible).

8 9

Re: [tdwg-content] status of uBio
by Steve Baskauf 16 Oct '15

16 Oct '15

Nicky, Thanks, I'm looking forward to checking it out! But no animals. :-( Steve Nicky Nicolson wrote: > > Hello Steve, > > Thanks for triggering an interesting thread. > > Just waving the IPNI flag for a minute: > > > > 1. They have managed to stick around for a long time and are stable in > their format (as LSIDs and HTTP proxied LSIDs). > > At ipni.org we support our own HTTP proxy for LSIDs: > > http://ipni.org/urn:lsid:ipni.org:names:12345-1 > > … and it’s been an age since I tried to resolve an LSID using the > formal LSID spec but a quick run-through today shows that all the > steps appear to be in working order. > > > 2. The coverage of names is really good for plants, animals, different > geographic locations, etc. I also use ITIS identifiers but it's > fairly common for me to not be able to find one for the name I need, > which almost never happens with uBio. > > > > IPNI is comprehensive for vascular plants (at species level). We’ll be > addressing the data gaps at infra-specific level – but it’s very > useful for us to be armed with reasons (from users like yourself) as > to why we should spend time doing this. > > > 3. It's somewhat clear what uBio identifiers refer to: names vs. > something more nebulous involving taxa or ... something. (Not trying > to push your button, Rich Pyle). > > IPNI only serves data about names, no taxa here. > > We are pushing IPNI IDs into our taxonomic resources so that a user > can flexibly match a name and get an IPNI identifier (the > nomenclatural part), and then as a later step query a taxonomic > resource for their current view as to the taxonomic status of that > name. We are aiming for a clean separation of names matching from the > (multiple, potentially different) uses of those names to form taxonomies. > > > 4. You can actually get RDF associated with the LSID version of the > uBio identifiers. I was wanting to download some to play with in our > new triplestore (http://rdf.library.vanderbilt.edu) when I discovered > that the server was down. The RDF is somewhat ad hoc, but hey, it's > there. > > > > The HTTP proxy above returns RDF metadata for the specified record. > > New developments (most of the functionality outlined above is 8-10 > years old), I’ve been working on exposing IPNI data through standard > match services – namely the Open Refine reconciliation API, which > permits much more flexible names look-ups using a lot of heuristics > gathered from matching data port exercises carried out at Kew over the > past few years. Currently the main hitch with this service is that it > includes duplicate records. This is something we’re working on > resolving right now. > > The service is outlined here: http://data1.kew.org/reconciliation/ and > details of the IPNI service in particular are here: > http://data1.kew.org/reconciliation/about/IpniName > > > > If you’re interested in using IPNI, I’d be happy to hear any comments > on the functionality above and / or any requests as to how we can make > it more useful for you. > > > > cheers, > > Nicky > > > > > > *From:* tdwg-content-bounces(a)lists.tdwg.org > [mailto:tdwg-content-bounces@lists.tdwg.org] *On Behalf Of *Steve Baskauf > *Sent:* 16 October 2015 14:09 > *To:* Dmitry Mozzherin <dmozzherin(a)gmail.com> > *Cc:* Chuck Miller (Contact) <chuck.miller(a)mobot.org>; > tdwg-content(a)lists.tdwg.org; Jonathan A Rees <rees(a)mumble.net>; > Shorthouse, David <david.shorthouse(a)umontreal.ca> > *Subject:* Re: [tdwg-content] status of uBio > > > > Thanks all for the information and comments about the status of uBio. > I'm glad to hear that the server will probably come back up. If not, > then I hope the data will be made available to those who said they > would be willing to host it. > > I have been interested in using the uBio identifiers for several reasons: > 1. They have managed to stick around for a long time and are stable in > their format (as LSIDs and HTTP proxied LSIDs). > 2. The coverage of names is really good for plants, animals, different > geographic locations, etc. I also use ITIS identifiers but it's > fairly common for me to not be able to find one for the name I need, > which almost never happens with uBio. > 3. It's somewhat clear what uBio identifiers refer to: names vs. > something more nebulous involving taxa or ... something. (Not trying > to push your button, Rich Pyle). > 4. You can actually get RDF associated with the LSID version of the > uBio identifiers. I was wanting to download some to play with in our > new triplestore (http://rdf.library.vanderbilt.edu) when I discovered > that the server was down. The RDF is somewhat ad hoc, but hey, it's > there. > > There isn't really any other source that has all of these > characteristics. So please keep uBio going indefinitely, if at all > possible. > Steve > > Dmitry Mozzherin wrote: > > I had been administering uBio for the last year, but now I am > moving from MBL. uBio machine is in a bad shape, and it crashes > after a few hours of work. My plan is to create Docker containers > for database, code and data, which should make whole system much > more stable, and much more manageable. Good news I will definitely > try my best to do it, the bad news I am spread thinner than usual > with move, transferring hardware and grant, GN things, EOL things, > and figuring out what to do with the house etc. uBio 'code' part > is about 35 Gb, which makes the task more complicated, but I am > quite optimistic that I will be able to make containers and put > them either on an MBL machine, run it from University of > Illinois, or give it to Naturalis -- depending on what will make > more sense for Dave Remsen, MBL and all interested in the project. > > > > > > -- > Steven J. Baskauf, Ph.D., Senior Lecturer > Vanderbilt University Dept. of Biological Sciences > > postal mail address: > PMB 351634 > Nashville, TN 37235-1634, U.S.A. > > delivery address: > 2125 Stevenson Center > 1161 21st Ave., S. > Nashville, TN 37235 > > office: 2128 Stevenson Center > phone: (615) 343-4582, fax: (615) 322-4942 > If you fax, please phone or email so that I will know to look for it. > http://bioimages.vanderbilt.edu > http://vanderbilt.edu/trees > -- Steven J. Baskauf, Ph.D., Senior Lecturer Vanderbilt University Dept. of Biological Sciences postal mail address: PMB 351634 Nashville, TN 37235-1634, U.S.A. delivery address: 2125 Stevenson Center 1161 21st Ave., S. Nashville, TN 37235 office: 2128 Stevenson Center phone: (615) 343-4582, fax: (615) 322-4942 If you fax, please phone or email so that I will know to look for it. http://bioimages.vanderbilt.edu http://vanderbilt.edu/trees

2 1

status of uBio
by Steve Baskauf 15 Oct '15

15 Oct '15

Does anybody know the status of uBio? The web server has been down now for weeks. I've been using it for a long time to find uBio ID numbers to use as a locally unique identifier for names. If the uBio server is not coming back up, does anybody know if there is a data dump somewhere? Steve -- Steven J. Baskauf, Ph.D., Senior Lecturer Vanderbilt University Dept. of Biological Sciences postal mail address: PMB 351634 Nashville, TN 37235-1634, U.S.A. delivery address: 2125 Stevenson Center 1161 21st Ave., S. Nashville, TN 37235 office: 2128 Stevenson Center phone: (615) 343-4582, fax: (615) 322-4942 If you fax, please phone or email so that I will know to look for it. http://bioimages.vanderbilt.edu http://vanderbilt.edu/trees

3 2