May 2008 - tdwg-tag - lists.tdwg.org

[tdwg-tapir] DwC extensions
by John R. WIECZOREK 04 Jun '08

04 Jun '08

I got sidetracked on this days ago, but feel in the light of recent star schema discussions on the original caching thread that the time is again right to submit this new discussion. Tradition has DwC discussions on this Tapir mailing list. I'm starting this new thread based on Markus' recent posting (below) about an Identification extension to DwC. I'm motivated to pull together the time and energy to finally push the pending DwC through the standards process, with a goal of having that whole process finished by the TDWG Meeting this year. I've been thinking about how to conduct the Request for Comment required to move the standard forward. I propose to put together a survey with Survey Monkey or something akin to actually test for reasonble concensus. Any comments or suggestions about this idea are welcome. However, I see benefits to having further discussion about some key issues before doing that, as I believe we now have enough accumulated experience to make some good decisions that will affect the design and guidelines for further development of the Darwin core and extensions. In the past, most Darwin Core discussions have revolved about whether to include a particular concept, and where. I think it will be much more useful to concentrate on a few key issues at a higher level, resolve them at that level, then make any necessary changes to the schemas based on the consensus guiding principles. It should be easy and fast to accomplish this if the principles are clear and simple. It should be possible to complete this work soon if we can easily achieve a concensus. Here are some seed questions and recommendations to facilitate the resolution if this next step in the process. 1) Is species occurrence in nature and in collections the right scope for the Core? 2) Should the general philosophy of the Core be inclusive or minimalist? What are the characteristics of a concept that allow it to be in the Core? What are the characteristics of a concept that allow it to be added to an existing extension? 3) What are the defining characteristics of a group of related concepts that justify the creation of a new extension? Should extensions be based on abstract conceptual groupings/objects (events, identifications/determinations, places)? Or on special interests (paleo, curation, interaction)? Or on the stability of the concepts (core contains the proven stable concepts, extensions are more volatile)? 4) Should there be elements in the Core and extensions to hold GUIDs linking them to instances of related classes of objects, such as an occurrence to a TaxonConceptGUID, or an occurrence to a CoreGatheringGUID? Should every extension have a non-mandatory GUID allowing for the external resolution of the object? 5) What should the Darwin Tapir application schema look like? 6) Is it the right approach to have restrictions on content at the concept definition level? Where should the line be drawn? Arguments have been raised in the past about the DwC and extensions' content with respect to being restrictive versus open to incorrect content. For example, DayOfYear in the current DwC 1.4 (http://rs.tdwg.org/dwc/tdwg_dw_core.xsd) is typed as a dwc:dayOfYearDataType, which is defined in http://rs.tdwg.org/dwc/tdwg_basetypes.xsd as: <xs:simpleType name="dayOfYearDataType"> <xs:restriction base="xs:integer"> <xs:minInclusive value="1" /> <xs:maxInclusive value="366" /> </xs:restriction> </xs:simpleType> John's two-cent opinions: 1) Yes 2) Tough question. Inclusive, but with a well-defined path from testing to inclusion. I see no merit in minimalism for its own sake. Candidate concepts should be in scope (the discovery or retrieval of occurrence information), have a demonstrated audience, and be stable following testing. New and untested concepts can go into test extensions and application schemas that import the core, other extensions and the test extension. 3) Alignment with the Core Ontology ( http://wiki.tdwg.org/twiki/bin/view/TAG/CoreOntology) is a good guiding principle for the design of extensions. To me this suggests, for example, that an Identification Extension is appropriate as a model for the CoreIdentification. The tough bit is to decide which objects constitute extensions. All of them? Higher-level ones? Ones that are likely to have services built around them? For example, should there be an extension for a CorePlace (Geospatial extension) or a CoreGathering (Geospatial extension with event information)? Another tough bit is to decide if objects that can have a one-to-many relationship with the Core should have Status concepts. For example, given an Identification Extension to the Core, should that extension have an IdentificationStatus concept in which to label a uniquely "accepted" identification? Sorry, more questions than answers here. The same recommendations for inclusion of concepts in the core (above) apply to extensions - that they should be tested and stable. 4) Tough question. We don't seem to be completely prepared from the implementation perspective to apply GUIDs to occurrences, let alone other objects whose nature may change over time. Is there a convincing argument one way or another? In the absence of an argument in favor, I guess the default response is "No new GUID concepts". 5) The existing Darwin Record Application Schema for Tapir ( http://rs.tdwg.org/dwc/tdwg_dw_record_tapir.xsd) is a good model. It is working well in practice so far. The concepts will have to change to accommodate any changes in the Core or extensions, but the structure and method of composition of the application schema seem sound. 6) No, it isn't the right approach to overly restrict content at the concept definition level for the simple reason that if we do that, we will remove the need and value of applications or services built on top of the distributed networks (or caches built from them) to help collections validate or do error detection on their data. That would be a great loss as an incentive to participate. Besides, application schemas can be built from the existing concept definitions and may further restrict them for specialized purposes. Anything short of a flamethrower in response is welcome. ---------- Forwarded message ---------- From: Markus Döring <mdoering(a)gbif.org> Date: Fri, May 16, 2008 at 1:29 AM Subject: Re: [tdwg-tapir] Fwd: Tapir protocol - Harvest methods? To: Renato De Giovanni <renato(a)cria.org.br> Cc: tdwg-tapir(a)lists.tdwg.org Renato, <snip> I have created an identification extension for darwin core that holds the historical list of identification events and their outcome. This is a YAML section of the metafile describing the columns for this extension through fully qualified concepts ala TAPIR: identification: - http://rs.tdwg.org/dwc/dwcore/ScientificName - http://rs.tdwg.org/dwc/dwcore/AuthorYearOfScientificName - http://rs.tdwg.org/dwc/dwcore/Family - http://rs.tdwg.org/dwc/dwcore/IdentificationQualifier - http://rs.tdwg.org/dwc/curatorial/DateIdentified - http://rs.tdwg.org/dwc/curatorial/IdentifiedBy When creating this I realised that pretty much all concepts I was interested in already existed in darwin core or the curatorial extension. Wouldnt it be wise to reuse those concepts? Or are they strictly tight to the idea of a current identification and therefore cant be used for historical ones? This is probably more of a darwin core question than TAPIR, but we are all on this list anyway ... The xml in that case would look sth like this: <record uri="http://mygarden.com/specimen/plants/54321-423-43-54-6-3-24-44 "> <dwc:ScientificName>Aster alpinus subsp. parviceps<dwc:ScientificName> ... <ident:record> <dwc:ScientificName>Aster alpinus<dwc:ScientificName> <dwc:AuthorYearOfScientificName>L.</dwc:AuthorYearOfScientificName> <dwc:Family>Asteraceae<dwc:Family> <cur:DateIdentified>1913-03-12</cur:DateIdentified> <cur:IdentifiedBy>Karl Marx</cur:IdentifiedBy> </ident:record> <ident:record> <dwc:ScientificName>Aster alpinus subsp. parviceps<dwc:ScientificName> <dwc:AuthorYearOfScientificName>Novopokr.</ dwc:AuthorYearOfScientificName> <dwc:Family>Asteraceae<dwc:Family> <cur:DateIdentified>2003-09-07</cur:DateIdentified> <cur:IdentifiedBy>Keith Richards</cur:IdentifiedBy> </ident:record> <record> Markus </snip>

7 6

Re: [tdwg-tapir] Fwd: Tapir protocol - Harvest methods?
by trobertson＠gbif.org 26 May '08

26 May '08

Hi All, This is very interesting too me, as I came up with the same conclusion while harvesting for GBIF. As a "harvester of all records" it is best described with an example: - Complete Inventory of ScientificNames: 7 minutes @ the limited 200 records per page - Complete Harvesting of records: - 260,000 records - 9 hours harvesting duration - 500MB TAPIR+DwC XML returned (DwC 1.4 with geospatial and curatorial extensions) - Extraction of DwC records from harvested XML: <2 minutes - Resulting file size 32MB, Gzipped to <3MB I spun hard drives for 9 hours, and took up bandwidth that is paid for, to retrieve something that could have been generated provider side in minutes and transferred in seconds (3MB). I sent a proposal to TDWG last year termed "datamaps" which was effectively what you are describing, and I based it on the Sitemaps protocol, but I got nowhere with it. With Markus, we are making more progress and I have spoken with several GBIF data providers about a proposed new standard for full dataset harvesting and it has been received well. So Markus and I have started a new proposal and have a working name of 'Localised DwC Index' file generation (it is an index if you have more than DwC data, and DwC is still standards compliant) which is really a GZipped Tab file dump of the data, which is slightly extensible. The document is not ready to circulate yet but the benefits section reads currently: - Provider database load reduced, allowing it to serve real distributed queries rather than "full datasource" harvesters - Providers can choose to publish their index as it suits them, giving control back to the provider - Localised index generation can be built into tools not yet capable of integrating with TDWG protocol networks such as GBIF - Harvesters receive a full dataset view in one request, making it very easy to determine what records are eligible for deletion - It becomes very simple to write clients that consume entire datasets. E.g. data cleansing tools that the provider can run: - Give me ISO Country Codes for my dataset - The application pulls down the providers index file, generates ISO country code, returns a simple table using the providers own identifier - Check my names for spelling mistakes - Application skims over the records and provides a list that are not known to the application - Providers such as UK NBN cannot serve 20 million records to the GBIF index using the existing protocols efficiently. - They have the ability to generate a localised index however - Harvesters can very quickly build up searchable indexes and it is easy to create large indices. - Node Portal can easily aggregate index data files - true index to data, not an illusion of a cache. More like Google sitemaps It is the ease at which one can offer tools to data providers that really interests me. The technical threshold required to produce services that offer reporting tools on peoples data is really very low with this mechanism. This and the fact that large datasets will be harvestable - we have even considered the likes of bit-torrent for the large ones although I think this is overkill. As a consumer therefore I fully support this move as a valuable addition to the wrapper tools. Cheers Tim (wrote the GBIF harvesting, and new to this list) > > > Begin forwarded message: > >> From: "Aaron D. Steele" <eightysteele(a)gmail.com> >> Date: 13 de mayo de 2008 22:40:09 GMT+02:00 >> To: tdwg-tapir(a)lists.tdwg.org >> Cc: Aaron Steele <asteele(a)berkeley.edu> >> Subject: Re: [tdwg-tapir] Tapir protocol - Harvest methods? >> >> at berkeley we've recently prototyped a simple php program that uses >> an existing tapirlink installation to periodically dump tapir >> resources into a csv file. the solution is totally generic and can >> dump darwin core (and technically abcd schema, although it's currently >> untested). the resulting csv files are zip archived and made >> accessible using a web service. it's a simple approach that has proven >> to be, at least internally, quite reliable and useful. >> >> for example, several of our caching applications use the web service >> to harvest csv data from tapirlink resources using the following >> process: >> 1) download latest csv dump for a resource using the web service. >> 2) flush all locally cached records for the resource. >> 3) bulk load the latest csv data into the cache. >> >> in this way, cached data are always synchronized with the resource and >> there's no need to track new, deleted, or changed records. as an >> aside, each time these cached data are queried by the caching >> application or selected in the user interface, log-only search >> requests are sent back to the resource. >> >> after discussion with renato giovanni and john wieczorek, we've >> decided that merging this functionality into the tapirlink codebase >> would benefit the broader community. csv generation support would be >> declared through capabilities. although incremental harvesting >> wouldn't be immediately implemented, we could certainly extend the >> service to include it later. >> >> i'd like to pause here to gauge the consensus, thoughts, concerns, and >> ideas of others. anyone? >> >> thanks, >> aaron >> >> 2008/5/5 Kevin Richards <RichardsK(a)landcareresearch.co.nz>: >>> >>> >>> I think I agree here. >>> >>> The harvesting "procedure" is really defined outside the Tapir >>> protocol, is >>> it not? So it is really an agreement between the harvester and the >>> harvestees. >>> >>> So what is really needed here is the standard procedure for >>> maintaining a >>> "harvestable" dataset and the standard procedure for harvesting that >>> dataset. >>> We have a general rule at Landcare, that we never delete records in >>> our >>> datasets - they are either deprecated in favour of another record, >>> and so >>> the resolution of that record would point to the new record, or the >>> are set >>> to a state of "deleted", but are still kept in the dataset, and can >>> be >>> resolved (which would indicate a state of deleted). >>> >>> Kevin >>> >>> >>>>>> "Renato De Giovanni" <renato(a)cria.org.br> 6/05/2008 7:33 a.m. >>> >>> >>> >>> Hi Markus, >>> >>> I would suggest creating new concepts for incremental harvesting, >>> either in the data standards themselves or in some new extension. In >>> the case of TAPIR, GBIF could easily check the mapped concepts before >>> deciding between incremental or full harvesting. >>> >>> Actually it could be just one new concept such as "recordStatus" or >>> "deletionFlag". Or perhaps you could also want to create your own >>> definition for dateLastModified indicating which set of concepts >>> should be considered to see if something has changed or not, but I >>> guess this level of granularity would be difficult to be supported. >>> >>> Regards, >>> -- >>> Renato >>> >>> On 5 May 2008 at 11:24, Markus Döring wrote: >>> >>>> Phil, >>>> incremental harvesting is not implemented on the GBIF side as far >>>> as I >>>> am aware. And I dont think that will be a simple thing to >>>> implement on >>>> the current system. Also, even if we can detect only the changed >>>> records since the last harevesting via dateLastModified we still >>>> have >>>> no information about deletions. We could have an arrangement saying >>>> that you keep deleted records as empty records with just the ID and >>>> nothing else (I vaguely remember LSIDs were supposed to work like >>>> this >>>> too). But that also needs to be supported on your side then, never >>>> entirely removing any record. I will have a discussion with the >>>> others >>>> at GBIF about that. >>>> >>>> Markus >>> >>> _______________________________________________ >>> tdwg-tapir mailing list >>> tdwg-tapir(a)lists.tdwg.org >>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>> >>> >>> >>> >>> Please consider the environment before printing this email >>> >>> WARNING : This email and any attachments may be confidential and/or >>> privileged. They are intended for the addressee only and are not to >>> be read, >>> used, copied or disseminated by anyone receiving them in error. If >>> you are >>> not the intended recipient, please notify the sender by return >>> email and >>> delete this message and any attachments. >>> >>> The views expressed in this email are those of the sender and do not >>> necessarily reflect the >>> official views of Landcare Research. http:// >>> www.landcareresearch.co.nz >>> _______________________________________________ >>> tdwg-tapir mailing list >>> tdwg-tapir(a)lists.tdwg.org >>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>> >>> >> _______________________________________________ >> tdwg-tapir mailing list >> tdwg-tapir(a)lists.tdwg.org >> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > >

15 36

Re: [tdwg-tapir] combine TAPIR provider with cell phone networks?
by trobertson＠gbif.org 16 May '08

16 May '08

Hi Frank, Would you not just use HTTP over the network? WAP became obsolete because technology (mainly screen and battery advancement in the last 10 years) allowed the devices to be capable of having an embedded browser - that and bandwidth went up massively. WAP was a very simple compressed text format that takes me back and makes me shudder at the early Nokia applications I used to put out in 1999... > Do you know if we could use any protocol which could be installed on top > of a Web-Service to use a GSM network as carrier for Internet information? So basically the traditions "comms stack" looks like, TAPIR as XML over HTTP over the cell network (GSM, CDMA etc) I would assume that the network would be both costly and low bandwidth though. Given that a lot of this data does not change on a very rapid basis, have you considered snapshot dumps (as we have been discussing) and then regular postal service? Even google ships hard drives around the world - stick a few 10 terrabyte drives in a fireproof jiffy bag and do a DHL overnight shipping, and work out the bandwidth of DHL ;o) Thanks Tim > > > Hello everybody, > > > > I recently subscribed to the mailing list and read the last messages about > the possibility to dump data from GBIF by using csv format. > > > > This type of questions interests me a lot because I'm currently > contributing to a new project called "SABIN" (Sub-Saharan African > Biodiversity Information Network) , which could lead (in the medium-term) > to the installation of a "cache" copy of the GBIF main database hosted by > the Kinshasa University. This copy could act as a relay-server for GBIF > data in Central Africa. > > We would also like to encourage institutions from Central Africa to > install TAPIR providers if they have they have the technical capacity to > do so, and then provide data to GBIF. > > > > I contacted people working in Congo, who told me that the Internet > facilities are good in Kinshasa but rather poor in the Eastern part of the > country (in the Great Lakes Region). > > For that reason, nine Congolese universities are currently studying the > possibility to use the cellular-phone network to exchange electronic > information, within the framework of a project which is called Eb@lé. > > > > Do you know if we could use any protocol which could be installed on top > of a Web-Service to use a GSM network as carrier for Internet information? > > > > -I would like to know if it's possible to use a protocol and/or a tool > that could be installed on top on a web service to connect to a cellular > network. I've heard about WAP 5/6 years ago but it seems now an obsolete > technology. > > -Have you hear about this kind of infrastructure (cell phone network) > involving a TAPIR provider? > > > > Thanks in advance > > > > Franck > > Franck Theeten > Database Manager > Royal Museum for Central Africa <http://www.africamuseum.be/> > Department of African Zoology > Leuvensesteenweg, 13 > 3080 Tervuren, Belgium > Tel.: +32 2 769 56 24 > > ########################################### > > This message has been scanned by ICT - Africa Museum > > ________________________________________ > 16/5/2008 - Filtered through antispam by ICT > > _______________________________________________ > tdwg-tapir mailing list > tdwg-tapir(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >

1 0

[tdwg-tapir] combine TAPIR provider with cell phone networks?
by THEETEN Franck 16 May '08

16 May '08

Hello everybody, I recently subscribed to the mailing list and read the last messages about the possibility to dump data from GBIF by using csv format. This type of questions interests me a lot because I'm currently contributing to a new project called "SABIN" (Sub-Saharan African Biodiversity Information Network) , which could lead (in the medium-term) to the installation of a "cache" copy of the GBIF main database hosted by the Kinshasa University. This copy could act as a relay-server for GBIF data in Central Africa. We would also like to encourage institutions from Central Africa to install TAPIR providers if they have they have the technical capacity to do so, and then provide data to GBIF. I contacted people working in Congo, who told me that the Internet facilities are good in Kinshasa but rather poor in the Eastern part of the country (in the Great Lakes Region). For that reason, nine Congolese universities are currently studying the possibility to use the cellular-phone network to exchange electronic information, within the framework of a project which is called Eb@lé. Do you know if we could use any protocol which could be installed on top of a Web-Service to use a GSM network as carrier for Internet information? -I would like to know if it's possible to use a protocol and/or a tool that could be installed on top on a web service to connect to a cellular network. I've heard about WAP 5/6 years ago but it seems now an obsolete technology. -Have you hear about this kind of infrastructure (cell phone network) involving a TAPIR provider? Thanks in advance Franck Franck Theeten Database Manager Royal Museum for Central Africa <http://www.africamuseum.be/> Department of African Zoology Leuvensesteenweg, 13 3080 Tervuren, Belgium Tel.: +32 2 769 56 24 ########################################### This message has been scanned by ICT - Africa Museum ________________________________________ 16/5/2008 - Filtered through antispam by ICT

1 0

Re: [tdwg-tapir] Fwd: Tapir protocol - Harvest methods?[SEC=UNCLASSIFIED]
by Holetschek, Jörg 15 May '08

15 May '08

Hi guys, sorry for the late reaction, but I put off reading all the mails until today. Using CSV and tab delimited files will cause problems when the dumps contains freetext data, e.g. locality description or notes. When I pushed our BioCASE cache (50 million occurrence records) between different DBMS using tab delimited files, I had to experience that people are very eager to use tabs and new lines in freetext fields. Any character you choose for delimiting contents you will find in freetext fields... Cheers from Berlin, Jörg -----Ursprüngliche Nachricht----- Von: tdwg-tapir-bounces(a)lists.tdwg.org [mailto:tdwg-tapir-bounces@lists.tdwg.org]Im Auftrag von Markus Döring Gesendet: Mittwoch, 14. Mai 2008 15:35 An: Aaron D. Steele Cc: TAPIR mailing list Betreff: Re: [tdwg-tapir] Fwd: Tapir protocol - Harvest methods?[SEC=UNCLASSIFIED] it would keep the relations, but we dont really want any relational structure to be served up. And using sqlite binaries for the dwc star scheme would not be easier to work with than plain text files. they can even be loaded into excel straight away, can be versioned with svn and so on. If there is a geospatial extension file which has the GUID in the first column, applications might grab that directly and not even touch the central core file if they only want location data. I'd prefer to stick with a csv or tab delimited file. The simpler the better. And it also cant get corrupted as easily. Markus On 14 May, 2008, at 15:25, Aaron D. Steele wrote: > for preserving relational data, we could also just dump tapirlink > resources to an sqlite database file (http://www.sqlite.org), zip it > up, and again make it available via the web service. we use sqlite > internally for many projects, and it's both easy to use and well > supported by jdbc, php, python, etc. > > would something like this be a useful option? > > thanks, > aaron > > On Wed, May 14, 2008 at 2:21 AM, Markus Döring <mdoering(a)gbif.org> > wrote: >> Interesting that we all come to the same conclusions... >> The trouble I had with just a simple flat csv file is repeating >> properties like multiple image urls. ABCD clients dont use ABCD just >> because its complex, but because they want to transport this >> relational data. We were considering 2 solutions to extending this >> csv >> approach. The first would be to have a single large denormalised csv >> file with many rows for the same record. It would require knowledge >> about the related entities though and could grow in size rapidly. The >> second idea which we think to adopt is allowing a single level of 1- >> many related entities. It is basically a "star" design with the core >> dwc table in the center and any number of extension tables around it. >> Each "table" aka csv file will have the record id as the first >> column, >> so the files can be related easily and it only needs a single >> identifier per record and not for the extension entities. This would >> give a lot of flexibility while keeping things pretty simple to deal >> with. It would even satisfy the ABCD needs as I havent yet seen >> anyone >> requiring 2 levels of related tables (other than lookup tables). >> Those >> extensions could even be a simple 1-1 relation, but would keep things >> semantically together just like a xml namespace. The darwin core >> extensions would be good for example. >> >> So we could have a gzipped set of files, maybe with a simple metafile >> indicating the semantics of the columns for each file. >> An example could look like this: >> >> >> # darwincore.csv >> 102 Aster alpinus subsp. parviceps ... >> 103 Polygala vulgaris ... >> >> # curatorial.csv >> 102 Kew Herbarium >> 103 Reading Herbarium >> >> # identification.csv >> 102 2003-05-04 Karl Marx Aster alpinus L. >> 102 2007-01-11 Mark Twain Aster korshinskyi Tamamsch. >> 102 2007-09-13 Roger Hyam Aster alpinus subsp. parviceps >> Novopokr. >> 103 2001-02-21 Steve Bekow Polygala vulgaris L. >> >> >> >> I know this looks old fashioned, but it is just so simple and gives >> us >> so much flexibility. >> Markus >> >> >> >> >> On 14 May, 2008, at 24:39, Greg Whitbread wrote: >> >>> We have used a very similar protocol to assemble the latest AVH >>> cache. >>> It should be noted that this is an as-well-as protocol that only >>> works >>> because we have an established semantic standard (hispid/abcd). >>> >>> greg >>> >>> trobertson(a)gbif.org wrote: >>>> Hi All, >>>> >>>> This is very interesting too me, as I came up with the same >>>> conclusion >>>> while harvesting for GBIF. >>>> >>>> As a "harvester of all records" it is best described with an >>>> example: >>>> >>>> - Complete Inventory of ScientificNames: 7 minutes @ the limited >>>> 200 >>>> records per page >>>> - Complete Harvesting of records: >>>> - 260,000 records >>>> - 9 hours harvesting duration >>>> - 500MB TAPIR+DwC XML returned (DwC 1.4 with geospatial and >>>> curatorial >>>> extensions) >>>> - Extraction of DwC records from harvested XML: <2 minutes >>>> - Resulting file size 32MB, Gzipped to <3MB >>>> >>>> I spun hard drives for 9 hours, and took up bandwidth that is paid >>>> for, to >>>> retrieve something that could have been generated provider side in >>>> minutes >>>> and transferred in seconds (3MB). >>>> >>>> I sent a proposal to TDWG last year termed "datamaps" which was >>>> effectively what you are describing, and I based it on the Sitemaps >>>> protocol, but I got nowhere with it. With Markus, we are making >>>> more >>>> progress and I have spoken with several GBIF data providers about a >>>> proposed new standard for full dataset harvesting and it has been >>>> received >>>> well. So Markus and I have started a new proposal and have a >>>> working name >>>> of 'Localised DwC Index' file generation (it is an index if you >>>> have more >>>> than DwC data, and DwC is still standards compliant) which is >>>> really a >>>> GZipped Tab file dump of the data, which is slightly extensible. >>>> The >>>> document is not ready to circulate yet but the benefits section >>>> reads >>>> currently: >>>> >>>> - Provider database load reduced, allowing it to serve real >>>> distributed >>>> queries rather than "full datasource" harvesters >>>> - Providers can choose to publish their index as it suits them, >>>> giving >>>> control back to the provider >>>> - Localised index generation can be built into tools not yet >>>> capable of >>>> integrating with TDWG protocol networks such as GBIF >>>> - Harvesters receive a full dataset view in one request, making it >>>> very >>>> easy to determine what records are eligible for deletion >>>> - It becomes very simple to write clients that consume entire >>>> datasets. >>>> E.g. data cleansing tools that the provider can run: >>>> - Give me ISO Country Codes for my dataset >>>> - The application pulls down the providers index file, >>>> generates ISO >>>> country code, returns a simple table using the providers own >>>> identifier >>>> - Check my names for spelling mistakes >>>> - Application skims over the records and provides a list that >>>> are not >>>> known to the application >>>> - Providers such as UK NBN cannot serve 20 million records to the >>>> GBIF >>>> index using the existing protocols efficiently. >>>> - They have the ability to generate a localised index however >>>> - Harvesters can very quickly build up searchable indexes and it is >>>> easy >>>> to create large indices. >>>> - Node Portal can easily aggregate index data files >>>> - true index to data, not an illusion of a cache. More like Google >>>> sitemaps >>>> >>>> It is the ease at which one can offer tools to data providers that >>>> really >>>> interests me. The technical threshold required to produce services >>>> that >>>> offer reporting tools on peoples data is really very low with this >>>> mechanism. This and the fact that large datasets will be >>>> harvestable - we >>>> have even considered the likes of bit-torrent for the large ones >>>> although >>>> I think this is overkill. >>>> >>>> As a consumer therefore I fully support this move as a valuable >>>> addition >>>> to the wrapper tools. >>>> >>>> Cheers >>>> >>>> Tim >>>> (wrote the GBIF harvesting, and new to this list) >>>> >>>> >>>>> >>>>> Begin forwarded message: >>>>> >>>>>> From: "Aaron D. Steele" <eightysteele(a)gmail.com> >>>>>> Date: 13 de mayo de 2008 22:40:09 GMT+02:00 >>>>>> To: tdwg-tapir(a)lists.tdwg.org >>>>>> Cc: Aaron Steele <asteele(a)berkeley.edu> >>>>>> Subject: Re: [tdwg-tapir] Tapir protocol - Harvest methods? >>>>>> >>>>>> at berkeley we've recently prototyped a simple php program that >>>>>> uses >>>>>> an existing tapirlink installation to periodically dump tapir >>>>>> resources into a csv file. the solution is totally generic and >>>>>> can >>>>>> dump darwin core (and technically abcd schema, although it's >>>>>> currently >>>>>> untested). the resulting csv files are zip archived and made >>>>>> accessible using a web service. it's a simple approach that has >>>>>> proven >>>>>> to be, at least internally, quite reliable and useful. >>>>>> >>>>>> for example, several of our caching applications use the web >>>>>> service >>>>>> to harvest csv data from tapirlink resources using the following >>>>>> process: >>>>>> 1) download latest csv dump for a resource using the web service. >>>>>> 2) flush all locally cached records for the resource. >>>>>> 3) bulk load the latest csv data into the cache. >>>>>> >>>>>> in this way, cached data are always synchronized with the >>>>>> resource and >>>>>> there's no need to track new, deleted, or changed records. as an >>>>>> aside, each time these cached data are queried by the caching >>>>>> application or selected in the user interface, log-only search >>>>>> requests are sent back to the resource. >>>>>> >>>>>> after discussion with renato giovanni and john wieczorek, we've >>>>>> decided that merging this functionality into the tapirlink >>>>>> codebase >>>>>> would benefit the broader community. csv generation support would >>>>>> be >>>>>> declared through capabilities. although incremental harvesting >>>>>> wouldn't be immediately implemented, we could certainly extend >>>>>> the >>>>>> service to include it later. >>>>>> >>>>>> i'd like to pause here to gauge the consensus, thoughts, >>>>>> concerns, and >>>>>> ideas of others. anyone? >>>>>> >>>>>> thanks, >>>>>> aaron >>>>>> >>>>>> 2008/5/5 Kevin Richards <RichardsK(a)landcareresearch.co.nz>: >>>>>>> >>>>>>> I think I agree here. >>>>>>> >>>>>>> The harvesting "procedure" is really defined outside the Tapir >>>>>>> protocol, is >>>>>>> it not? So it is really an agreement between the harvester and >>>>>>> the >>>>>>> harvestees. >>>>>>> >>>>>>> So what is really needed here is the standard procedure for >>>>>>> maintaining a >>>>>>> "harvestable" dataset and the standard procedure for harvesting >>>>>>> that >>>>>>> dataset. >>>>>>> We have a general rule at Landcare, that we never delete records >>>>>>> in >>>>>>> our >>>>>>> datasets - they are either deprecated in favour of another >>>>>>> record, >>>>>>> and so >>>>>>> the resolution of that record would point to the new record, or >>>>>>> the >>>>>>> are set >>>>>>> to a state of "deleted", but are still kept in the dataset, and >>>>>>> can >>>>>>> be >>>>>>> resolved (which would indicate a state of deleted). >>>>>>> >>>>>>> Kevin >>>>>>> >>>>>>> >>>>>>>>>> "Renato De Giovanni" <renato(a)cria.org.br> 6/05/2008 7:33 a.m. >>>>>>>>>>>>> >>>>>>> >>>>>>> Hi Markus, >>>>>>> >>>>>>> I would suggest creating new concepts for incremental >>>>>>> harvesting, >>>>>>> either in the data standards themselves or in some new >>>>>>> extension. In >>>>>>> the case of TAPIR, GBIF could easily check the mapped concepts >>>>>>> before >>>>>>> deciding between incremental or full harvesting. >>>>>>> >>>>>>> Actually it could be just one new concept such as "recordStatus" >>>>>>> or >>>>>>> "deletionFlag". Or perhaps you could also want to create your >>>>>>> own >>>>>>> definition for dateLastModified indicating which set of concepts >>>>>>> should be considered to see if something has changed or not, >>>>>>> but I >>>>>>> guess this level of granularity would be difficult to be >>>>>>> supported. >>>>>>> >>>>>>> Regards, >>>>>>> -- >>>>>>> Renato >>>>>>> >>>>>>> On 5 May 2008 at 11:24, Markus Döring wrote: >>>>>>> >>>>>>>> Phil, >>>>>>>> incremental harvesting is not implemented on the GBIF side as >>>>>>>> far >>>>>>>> as I >>>>>>>> am aware. And I dont think that will be a simple thing to >>>>>>>> implement on >>>>>>>> the current system. Also, even if we can detect only the >>>>>>>> changed >>>>>>>> records since the last harevesting via dateLastModified we >>>>>>>> still >>>>>>>> have >>>>>>>> no information about deletions. We could have an arrangement >>>>>>>> saying >>>>>>>> that you keep deleted records as empty records with just the ID >>>>>>>> and >>>>>>>> nothing else (I vaguely remember LSIDs were supposed to work >>>>>>>> like >>>>>>>> this >>>>>>>> too). But that also needs to be supported on your side then, >>>>>>>> never >>>>>>>> entirely removing any record. I will have a discussion with the >>>>>>>> others >>>>>>>> at GBIF about that. >>>>>>>> >>>>>>>> Markus >>>>>>> _______________________________________________ >>>>>>> tdwg-tapir mailing list >>>>>>> tdwg-tapir(a)lists.tdwg.org >>>>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Please consider the environment before printing this email >>>>>>> >>>>>>> WARNING : This email and any attachments may be confidential >>>>>>> and/ >>>>>>> or >>>>>>> privileged. They are intended for the addressee only and are not >>>>>>> to >>>>>>> be read, >>>>>>> used, copied or disseminated by anyone receiving them in >>>>>>> error. If >>>>>>> you are >>>>>>> not the intended recipient, please notify the sender by return >>>>>>> email and >>>>>>> delete this message and any attachments. >>>>>>> >>>>>>> The views expressed in this email are those of the sender and do >>>>>>> not >>>>>>> necessarily reflect the >>>>>>> official views of Landcare Research. http:// >>>>>>> www.landcareresearch.co.nz >>>>>>> _______________________________________________ >>>>>>> tdwg-tapir mailing list >>>>>>> tdwg-tapir(a)lists.tdwg.org >>>>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> tdwg-tapir mailing list >>>>>> tdwg-tapir(a)lists.tdwg.org >>>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>>>> >>>> >>>> >>>> _______________________________________________ >>>> tdwg-tapir mailing list >>>> tdwg-tapir(a)lists.tdwg.org >>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>> >>> -- >>> >>> Australian Centre for Plant BIodiversity >>> Research<------------------+ >>> National greg whitBread voice: +61 2 62509 >>> 482 >>> Botanic Integrated Botanical Information System fax: +61 2 62509 >>> 599 >>> Gardens S........ I.T. happens.. >>> ghw(a)anbg.gov.au >>> +----------------------------------------->GPO Box 1777 Canberra >>> 2601 >>> >>> >>> >>> ------ >>> If you have received this transmission in error please notify us >>> immediately by return e-mail and delete all copies. If this e-mail >>> or any attachments have been sent to you in error, that error does >>> not constitute waiver of any confidentiality, privilege or copyright >>> in respect of information in the e-mail or attachments. >>> >>> >>> >>> Please consider the environment before printing this email. >>> >>> ------ >>> >>> _______________________________________________ >>> tdwg-tapir mailing list >>> tdwg-tapir(a)lists.tdwg.org >>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>> >> >> _______________________________________________ >> tdwg-tapir mailing list >> tdwg-tapir(a)lists.tdwg.org >> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >> > _______________________________________________ > tdwg-tapir mailing list > tdwg-tapir(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > _______________________________________________ tdwg-tapir mailing list tdwg-tapir(a)lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tapir

3 2

Re: [tdwg-tapir] tdwg-tapir Digest, Vol 31, Issue 9
by Chuck Miller 15 May '08

15 May '08

Using CSV files is certainly simple and compact. And a one-level deep star structure around DwC makes a lot of sense to me. It is simple and simple works. ABCD, TCS are complex and suffer from the confining heirarchical, pyramid structures of XML in a world that is mainly relational. Star structures are the old stuff of data warehousing where efficiency and straightforward data updating is required. But, isn't the age old problem with CSV files the weakness in integrity? One botched comma and you have a mess to troubleshoot. On a global scale, with thousands of providers, and hundreds of millions of records, any potential crack in the data integrity is potentially going to lead to a full-time job for someone to be tracking down the unexplainable errors. Working at these scales as Tim has pointed out forces consideration of small things that ordinarily are a breeze to contend with. Simple flat, tagged XML files (instead of CSV) in the same star structure could be a compromise. The files would be a little larger but have better integrity, and I would think they would compress well. But, I like the general idea of using flat, one-level star structures. Could we consider the same idea for harvesting name/concept data? A flat core with a one-level, one-to-many star. Rather than TCS. Chuck -----Original Message----- From: tdwg-tapir-bounces(a)lists.tdwg.org [mailto:tdwg-tapir-bounces@lists.tdwg.org] On Behalf Of tdwg-tapir-request(a)lists.tdwg.org Sent: Wednesday, May 14, 2008 4:59 AM To: tdwg-tapir(a)lists.tdwg.org Subject: tdwg-tapir Digest, Vol 31, Issue 9 Send tdwg-tapir mailing list submissions to tdwg-tapir(a)lists.tdwg.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.tdwg.org/mailman/listinfo/tdwg-tapir or, via email, send a message with subject or body 'help' to tdwg-tapir-request(a)lists.tdwg.org You can reach the person managing the list at tdwg-tapir-owner(a)lists.tdwg.org When replying, please edit your Subject line so it is more specific than "Re: Contents of tdwg-tapir digest..." Today's Topics: 1. Re: Fwd: Tapir protocol - Harvest methods? [SEC=UNCLASSIFIED] (Roger Hyam (TDWG)) 2. Re: [Hiscom-l] Fwd: Tapir protocol - Harvest methods? [SEC=UNCLASSIFIED] (Greg Whitbread) ---------------------------------------------------------------------- Message: 1 Date: Wed, 14 May 2008 10:57:20 +0100 From: "Roger Hyam (TDWG)" <rogerhyam(a)mac.com> Subject: Re: [tdwg-tapir] Fwd: Tapir protocol - Harvest methods? [SEC=UNCLASSIFIED] To: Markus D?ring <mdoering(a)gbif.org> Cc: "Hiscom-L Mailing List \(\(E-mail\)\)" <hiscom-l(a)chah.org.au>, tdwg-tapir(a)lists.tdwg.org Message-ID: <16026031-3900-41BF-960D-17429360EFE6(a)mac.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Generally if we are going to have csv files for data transfer we don't need to have software implementations just some documentation on what the csv files should contain. Something along the lines of: 1) Make a report from your database as a csv file(s) with the following columns... 2) Zip it up. 3) Either put it on a webserver and send us the URL or upload it using this webform. We don't need to bother with TAPIR etc. You could even only produce a CSV file of the records that have changed so big data sets needn't be a problem. I worry that we are working out how to move data about quickly and forgetting that the real goal is to integrate data and that will only come if people have GUIDs on the stuff they own and use other peoples GUIDs in their data. Solutions based around CSV files do nothing to move people in that direction and I would suspect lead to making matters worse. Finding ourselves in hole digging quicker may not be the best option. Roger ------------------------------------------------------------- Roger Hyam Roger(a)BiodiversityCollectionsIndex.org http://www.BiodiversityCollectionsIndex.org ------------------------------------------------------------- Royal Botanic Garden Edinburgh 20A Inverleith Row, Edinburgh, EH3 5LR, UK Tel: +44 131 552 7171 ext 3015 Fax: +44 131 248 2901 http://www.rbge.org.uk/ ------------------------------------------------------------- On 14 May 2008, at 10:21, Markus D?ring wrote: > Interesting that we all come to the same conclusions... > The trouble I had with just a simple flat csv file is repeating > properties like multiple image urls. ABCD clients dont use ABCD just > because its complex, but because they want to transport this > relational data. We were considering 2 solutions to extending this csv > approach. The first would be to have a single large denormalised csv > file with many rows for the same record. It would require knowledge > about the related entities though and could grow in size rapidly. The > second idea which we think to adopt is allowing a single level of 1- > many related entities. It is basically a "star" design with the core > dwc table in the center and any number of extension tables around it. > Each "table" aka csv file will have the record id as the first column, > so the files can be related easily and it only needs a single > identifier per record and not for the extension entities. This would > give a lot of flexibility while keeping things pretty simple to deal > with. It would even satisfy the ABCD needs as I havent yet seen anyone > requiring 2 levels of related tables (other than lookup tables). Those > extensions could even be a simple 1-1 relation, but would keep things > semantically together just like a xml namespace. The darwin core > extensions would be good for example. > > So we could have a gzipped set of files, maybe with a simple metafile > indicating the semantics of the columns for each file. > An example could look like this: > > > # darwincore.csv > 102 Aster alpinus subsp. parviceps ... > 103 Polygala vulgaris ... > > # curatorial.csv > 102 Kew Herbarium > 103 Reading Herbarium > > # identification.csv > 102 2003-05-04 Karl Marx Aster alpinus L. > 102 2007-01-11 Mark Twain Aster korshinskyi Tamamsch. > 102 2007-09-13 Roger Hyam Aster alpinus subsp. parviceps > Novopokr. > 103 2001-02-21 Steve Bekow Polygala vulgaris L. > > > > I know this looks old fashioned, but it is just so simple and gives us > so much flexibility. > Markus > > > On 14 May, 2008, at 24:39, Greg Whitbread wrote: > >> We have used a very similar protocol to assemble the latest AVH >> cache. >> It should be noted that this is an as-well-as protocol that only >> works >> because we have an established semantic standard (hispid/abcd). >> >> greg >> >> trobertson(a)gbif.org wrote: >>> Hi All, >>> >>> This is very interesting too me, as I came up with the same >>> conclusion >>> while harvesting for GBIF. >>> >>> As a "harvester of all records" it is best described with an >>> example: >>> >>> - Complete Inventory of ScientificNames: 7 minutes @ the limited 200 >>> records per page >>> - Complete Harvesting of records: >>> - 260,000 records >>> - 9 hours harvesting duration >>> - 500MB TAPIR+DwC XML returned (DwC 1.4 with geospatial and >>> curatorial >>> extensions) >>> - Extraction of DwC records from harvested XML: <2 minutes >>> - Resulting file size 32MB, Gzipped to <3MB >>> >>> I spun hard drives for 9 hours, and took up bandwidth that is paid >>> for, to >>> retrieve something that could have been generated provider side in >>> minutes >>> and transferred in seconds (3MB). >>> >>> I sent a proposal to TDWG last year termed "datamaps" which was >>> effectively what you are describing, and I based it on the Sitemaps >>> protocol, but I got nowhere with it. With Markus, we are making >>> more >>> progress and I have spoken with several GBIF data providers about a >>> proposed new standard for full dataset harvesting and it has been >>> received >>> well. So Markus and I have started a new proposal and have a >>> working name >>> of 'Localised DwC Index' file generation (it is an index if you >>> have more >>> than DwC data, and DwC is still standards compliant) which is >>> really a >>> GZipped Tab file dump of the data, which is slightly extensible. >>> The >>> document is not ready to circulate yet but the benefits section >>> reads >>> currently: >>> >>> - Provider database load reduced, allowing it to serve real >>> distributed >>> queries rather than "full datasource" harvesters >>> - Providers can choose to publish their index as it suits them, >>> giving >>> control back to the provider >>> - Localised index generation can be built into tools not yet >>> capable of >>> integrating with TDWG protocol networks such as GBIF >>> - Harvesters receive a full dataset view in one request, making it >>> very >>> easy to determine what records are eligible for deletion >>> - It becomes very simple to write clients that consume entire >>> datasets. >>> E.g. data cleansing tools that the provider can run: >>> - Give me ISO Country Codes for my dataset >>> - The application pulls down the providers index file, >>> generates ISO >>> country code, returns a simple table using the providers own >>> identifier >>> - Check my names for spelling mistakes >>> - Application skims over the records and provides a list that >>> are not >>> known to the application >>> - Providers such as UK NBN cannot serve 20 million records to the >>> GBIF >>> index using the existing protocols efficiently. >>> - They have the ability to generate a localised index however >>> - Harvesters can very quickly build up searchable indexes and it is >>> easy >>> to create large indices. >>> - Node Portal can easily aggregate index data files >>> - true index to data, not an illusion of a cache. More like Google >>> sitemaps >>> >>> It is the ease at which one can offer tools to data providers that >>> really >>> interests me. The technical threshold required to produce services >>> that >>> offer reporting tools on peoples data is really very low with this >>> mechanism. This and the fact that large datasets will be >>> harvestable - we >>> have even considered the likes of bit-torrent for the large ones >>> although >>> I think this is overkill. >>> >>> As a consumer therefore I fully support this move as a valuable >>> addition >>> to the wrapper tools. >>> >>> Cheers >>> >>> Tim >>> (wrote the GBIF harvesting, and new to this list) >>> >>> >>>> >>>> Begin forwarded message: >>>> >>>>> From: "Aaron D. Steele" <eightysteele(a)gmail.com> >>>>> Date: 13 de mayo de 2008 22:40:09 GMT+02:00 >>>>> To: tdwg-tapir(a)lists.tdwg.org >>>>> Cc: Aaron Steele <asteele(a)berkeley.edu> >>>>> Subject: Re: [tdwg-tapir] Tapir protocol - Harvest methods? >>>>> >>>>> at berkeley we've recently prototyped a simple php program that >>>>> uses >>>>> an existing tapirlink installation to periodically dump tapir >>>>> resources into a csv file. the solution is totally generic and can >>>>> dump darwin core (and technically abcd schema, although it's >>>>> currently >>>>> untested). the resulting csv files are zip archived and made >>>>> accessible using a web service. it's a simple approach that has >>>>> proven >>>>> to be, at least internally, quite reliable and useful. >>>>> >>>>> for example, several of our caching applications use the web >>>>> service >>>>> to harvest csv data from tapirlink resources using the following >>>>> process: >>>>> 1) download latest csv dump for a resource using the web service. >>>>> 2) flush all locally cached records for the resource. >>>>> 3) bulk load the latest csv data into the cache. >>>>> >>>>> in this way, cached data are always synchronized with the >>>>> resource and >>>>> there's no need to track new, deleted, or changed records. as an >>>>> aside, each time these cached data are queried by the caching >>>>> application or selected in the user interface, log-only search >>>>> requests are sent back to the resource. >>>>> >>>>> after discussion with renato giovanni and john wieczorek, we've >>>>> decided that merging this functionality into the tapirlink >>>>> codebase >>>>> would benefit the broader community. csv generation support would >>>>> be >>>>> declared through capabilities. although incremental harvesting >>>>> wouldn't be immediately implemented, we could certainly extend the >>>>> service to include it later. >>>>> >>>>> i'd like to pause here to gauge the consensus, thoughts, >>>>> concerns, and >>>>> ideas of others. anyone? >>>>> >>>>> thanks, >>>>> aaron >>>>> >>>>> 2008/5/5 Kevin Richards <RichardsK(a)landcareresearch.co.nz>: >>>>>> >>>>>> I think I agree here. >>>>>> >>>>>> The harvesting "procedure" is really defined outside the Tapir >>>>>> protocol, is >>>>>> it not? So it is really an agreement between the harvester and >>>>>> the >>>>>> harvestees. >>>>>> >>>>>> So what is really needed here is the standard procedure for >>>>>> maintaining a >>>>>> "harvestable" dataset and the standard procedure for harvesting >>>>>> that >>>>>> dataset. >>>>>> We have a general rule at Landcare, that we never delete records >>>>>> in >>>>>> our >>>>>> datasets - they are either deprecated in favour of another >>>>>> record, >>>>>> and so >>>>>> the resolution of that record would point to the new record, or >>>>>> the >>>>>> are set >>>>>> to a state of "deleted", but are still kept in the dataset, and >>>>>> can >>>>>> be >>>>>> resolved (which would indicate a state of deleted). >>>>>> >>>>>> Kevin >>>>>> >>>>>> >>>>>>>>> "Renato De Giovanni" <renato(a)cria.org.br> 6/05/2008 7:33 a.m. >>>>>>>>>>>> >>>>>> >>>>>> Hi Markus, >>>>>> >>>>>> I would suggest creating new concepts for incremental harvesting, >>>>>> either in the data standards themselves or in some new >>>>>> extension. In >>>>>> the case of TAPIR, GBIF could easily check the mapped concepts >>>>>> before >>>>>> deciding between incremental or full harvesting. >>>>>> >>>>>> Actually it could be just one new concept such as "recordStatus" >>>>>> or >>>>>> "deletionFlag". Or perhaps you could also want to create your own >>>>>> definition for dateLastModified indicating which set of concepts >>>>>> should be considered to see if something has changed or not, >>>>>> but I >>>>>> guess this level of granularity would be difficult to be >>>>>> supported. >>>>>> >>>>>> Regards, >>>>>> -- >>>>>> Renato >>>>>> >>>>>> On 5 May 2008 at 11:24, Markus D?ring wrote: >>>>>> >>>>>>> Phil, >>>>>>> incremental harvesting is not implemented on the GBIF side as >>>>>>> far >>>>>>> as I >>>>>>> am aware. And I dont think that will be a simple thing to >>>>>>> implement on >>>>>>> the current system. Also, even if we can detect only the changed >>>>>>> records since the last harevesting via dateLastModified we still >>>>>>> have >>>>>>> no information about deletions. We could have an arrangement >>>>>>> saying >>>>>>> that you keep deleted records as empty records with just the ID >>>>>>> and >>>>>>> nothing else (I vaguely remember LSIDs were supposed to work >>>>>>> like >>>>>>> this >>>>>>> too). But that also needs to be supported on your side then, >>>>>>> never >>>>>>> entirely removing any record. I will have a discussion with the >>>>>>> others >>>>>>> at GBIF about that. >>>>>>> >>>>>>> Markus >>>>>> _______________________________________________ >>>>>> tdwg-tapir mailing list >>>>>> tdwg-tapir(a)lists.tdwg.org >>>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Please consider the environment before printing this email >>>>>> >>>>>> WARNING : This email and any attachments may be confidential and/ >>>>>> or >>>>>> privileged. They are intended for the addressee only and are not >>>>>> to >>>>>> be read, >>>>>> used, copied or disseminated by anyone receiving them in error. >>>>>> If >>>>>> you are >>>>>> not the intended recipient, please notify the sender by return >>>>>> email and >>>>>> delete this message and any attachments. >>>>>> >>>>>> The views expressed in this email are those of the sender and do >>>>>> not >>>>>> necessarily reflect the >>>>>> official views of Landcare Research. http:// >>>>>> www.landcareresearch.co.nz >>>>>> _______________________________________________ >>>>>> tdwg-tapir mailing list >>>>>> tdwg-tapir(a)lists.tdwg.org >>>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> tdwg-tapir mailing list >>>>> tdwg-tapir(a)lists.tdwg.org >>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >>>> >>> >>> >>> _______________________________________________ >>> tdwg-tapir mailing list >>> tdwg-tapir(a)lists.tdwg.org >>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >> >> -- >> >> Australian Centre for Plant BIodiversity Research<------------------+ >> National greg whitBread voice: +61 2 62509 482 >> Botanic Integrated Botanical Information System fax: +61 2 62509 599 >> Gardens S........ I.T. happens.. ghw(a)anbg.gov.au >> +----------------------------------------->GPO Box 1777 Canberra 2601 >> >> >> >> ------ >> If you have received this transmission in error please notify us >> immediately by return e-mail and delete all copies. If this e-mail >> or any attachments have been sent to you in error, that error does >> not constitute waiver of any confidentiality, privilege or copyright >> in respect of information in the e-mail or attachments. >> >> >> >> Please consider the environment before printing this email. >> >> ------ >> >> _______________________________________________ >> tdwg-tapir mailing list >> tdwg-tapir(a)lists.tdwg.org >> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir >> > > _______________________________________________ > tdwg-tapir mailing list > tdwg-tapir(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-tapir ------------------------------ Message: 2 Date: Wed, 14 May 2008 19:59:12 +1000 From: "Greg Whitbread" <ghw(a)anbg.gov.au> Subject: Re: [tdwg-tapir] [Hiscom-l] Fwd: Tapir protocol - Harvest methods? [SEC=UNCLASSIFIED] To: "Markus D?ring " <mdoering(a)gbif.org> Cc: "Hiscom-L Mailing List \(\(E-mail\)\)" <hiscom-l(a)chah.org.au>, tdwg-tapir(a)lists.tdwg.org Message-ID: <1210759152.18436.414.camel@goshawk> Content-Type: text/plain; charset=utf-8 Marcus, We have used the star technique with abcd with some success. The core is pretty close to dwc if the current determination goes there too. It can then be delivered with headers appropriate to the harvester's preference. It gets harder when we try to do tcs. On Wed, 2008-05-14 at 19:21, Markus D?ring wrote: > Interesting that we all come to the same conclusions... > The trouble I had with just a simple flat csv file is repeating > properties like multiple image urls. ABCD clients dont use ABCD just > because its complex, but because they want to transport this > relational data. We were considering 2 solutions to extending this csv > approach. The first would be to have a single large denormalised csv > file with many rows for the same record. It would require knowledge > about the related entities though and could grow in size rapidly. The > second idea which we think to adopt is allowing a single level of 1- > many related entities. It is basically a "star" design with the core > dwc table in the center and any number of extension tables around it. > Each "table" aka csv file will have the record id as the first column, > so the files can be related easily and it only needs a single > identifier per record and not for the extension entities. This would > give a lot of flexibility while keeping things pretty simple to deal > with. It would even satisfy the ABCD needs as I havent yet seen anyone > requiring 2 levels of related tables (other than lookup tables). Those > extensions could even be a simple 1-1 relation, but would keep things > semantically together just like a xml namespace. The darwin core > extensions would be good for example. > > So we could have a gzipped set of files, maybe with a simple metafile > indicating the semantics of the columns for each file. > An example could look like this: > > > # darwincore.csv > 102 Aster alpinus subsp. parviceps ... > 103 Polygala vulgaris ... > > # curatorial.csv > 102 Kew Herbarium > 103 Reading Herbarium > > # identification.csv > 102 2003-05-04 Karl Marx Aster alpinus L. > 102 2007-01-11 Mark Twain Aster korshinskyi Tamamsch. > 102 2007-09-13 Roger Hyam Aster alpinus subsp. parviceps > Novopokr. > 103 2001-02-21 Steve Bekow Polygala vulgaris L. > > > > I know this looks old fashioned, but it is just so simple and gives us > so much flexibility. > Markus > > > On 14 May, 2008, at 24:39, Greg Whitbread wrote: > > > We have used a very similar protocol to assemble the latest AVH cache. > > It should be noted that this is an as-well-as protocol that only works > > because we have an established semantic standard (hispid/abcd). > > > > greg > > > > trobertson(a)gbif.org wrote: > >> Hi All, > >> > >> This is very interesting too me, as I came up with the same > >> conclusion > >> while harvesting for GBIF. > >> > >> As a "harvester of all records" it is best described with an example: > >> > >> - Complete Inventory of ScientificNames: 7 minutes @ the limited 200 > >> records per page > >> - Complete Harvesting of records: > >> - 260,000 records > >> - 9 hours harvesting duration > >> - 500MB TAPIR+DwC XML returned (DwC 1.4 with geospatial and > >> curatorial > >> extensions) > >> - Extraction of DwC records from harvested XML: <2 minutes > >> - Resulting file size 32MB, Gzipped to <3MB > >> > >> I spun hard drives for 9 hours, and took up bandwidth that is paid > >> for, to > >> retrieve something that could have been generated provider side in > >> minutes > >> and transferred in seconds (3MB). > >> > >> I sent a proposal to TDWG last year termed "datamaps" which was > >> effectively what you are describing, and I based it on the Sitemaps > >> protocol, but I got nowhere with it. With Markus, we are making more > >> progress and I have spoken with several GBIF data providers about a > >> proposed new standard for full dataset harvesting and it has been > >> received > >> well. So Markus and I have started a new proposal and have a > >> working name > >> of 'Localised DwC Index' file generation (it is an index if you > >> have more > >> than DwC data, and DwC is still standards compliant) which is > >> really a > >> GZipped Tab file dump of the data, which is slightly extensible. The > >> document is not ready to circulate yet but the benefits section reads > >> currently: > >> > >> - Provider database load reduced, allowing it to serve real > >> distributed > >> queries rather than "full datasource" harvesters > >> - Providers can choose to publish their index as it suits them, > >> giving > >> control back to the provider > >> - Localised index generation can be built into tools not yet > >> capable of > >> integrating with TDWG protocol networks such as GBIF > >> - Harvesters receive a full dataset view in one request, making it > >> very > >> easy to determine what records are eligible for deletion > >> - It becomes very simple to write clients that consume entire > >> datasets. > >> E.g. data cleansing tools that the provider can run: > >> - Give me ISO Country Codes for my dataset > >> - The application pulls down the providers index file, > >> generates ISO > >> country code, returns a simple table using the providers own > >> identifier > >> - Check my names for spelling mistakes > >> - Application skims over the records and provides a list that > >> are not > >> known to the application > >> - Providers such as UK NBN cannot serve 20 million records to the > >> GBIF > >> index using the existing protocols efficiently. > >> - They have the ability to generate a localised index however > >> - Harvesters can very quickly build up searchable indexes and it is > >> easy > >> to create large indices. > >> - Node Portal can easily aggregate index data files > >> - true index to data, not an illusion of a cache. More like Google > >> sitemaps > >> > >> It is the ease at which one can offer tools to data providers that > >> really > >> interests me. The technical threshold required to produce services > >> that > >> offer reporting tools on peoples data is really very low with this > >> mechanism. This and the fact that large datasets will be > >> harvestable - we > >> have even considered the likes of bit-torrent for the large ones > >> although > >> I think this is overkill. > >> > >> As a consumer therefore I fully support this move as a valuable > >> addition > >> to the wrapper tools. > >> > >> Cheers > >> > >> Tim > >> (wrote the GBIF harvesting, and new to this list) > >> > >> > >>> > >>> Begin forwarded message: > >>> > >>>> From: "Aaron D. Steele" <eightysteele(a)gmail.com> > >>>> Date: 13 de mayo de 2008 22:40:09 GMT+02:00 > >>>> To: tdwg-tapir(a)lists.tdwg.org > >>>> Cc: Aaron Steele <asteele(a)berkeley.edu> > >>>> Subject: Re: [tdwg-tapir] Tapir protocol - Harvest methods? > >>>> > >>>> at berkeley we've recently prototyped a simple php program that > >>>> uses > >>>> an existing tapirlink installation to periodically dump tapir > >>>> resources into a csv file. the solution is totally generic and can > >>>> dump darwin core (and technically abcd schema, although it's > >>>> currently > >>>> untested). the resulting csv files are zip archived and made > >>>> accessible using a web service. it's a simple approach that has > >>>> proven > >>>> to be, at least internally, quite reliable and useful. > >>>> > >>>> for example, several of our caching applications use the web > >>>> service > >>>> to harvest csv data from tapirlink resources using the following > >>>> process: > >>>> 1) download latest csv dump for a resource using the web service. > >>>> 2) flush all locally cached records for the resource. > >>>> 3) bulk load the latest csv data into the cache. > >>>> > >>>> in this way, cached data are always synchronized with the > >>>> resource and > >>>> there's no need to track new, deleted, or changed records. as an > >>>> aside, each time these cached data are queried by the caching > >>>> application or selected in the user interface, log-only search > >>>> requests are sent back to the resource. > >>>> > >>>> after discussion with renato giovanni and john wieczorek, we've > >>>> decided that merging this functionality into the tapirlink codebase > >>>> would benefit the broader community. csv generation support would > >>>> be > >>>> declared through capabilities. although incremental harvesting > >>>> wouldn't be immediately implemented, we could certainly extend the > >>>> service to include it later. > >>>> > >>>> i'd like to pause here to gauge the consensus, thoughts, > >>>> concerns, and > >>>> ideas of others. anyone? > >>>> > >>>> thanks, > >>>> aaron > >>>> > >>>> 2008/5/5 Kevin Richards <RichardsK(a)landcareresearch.co.nz>: > >>>>> > >>>>> I think I agree here. > >>>>> > >>>>> The harvesting "procedure" is really defined outside the Tapir > >>>>> protocol, is > >>>>> it not? So it is really an agreement between the harvester and > >>>>> the > >>>>> harvestees. > >>>>> > >>>>> So what is really needed here is the standard procedure for > >>>>> maintaining a > >>>>> "harvestable" dataset and the standard procedure for harvesting > >>>>> that > >>>>> dataset. > >>>>> We have a general rule at Landcare, that we never delete records > >>>>> in > >>>>> our > >>>>> datasets - they are either deprecated in favour of another record, > >>>>> and so > >>>>> the resolution of that record would point to the new record, or > >>>>> the > >>>>> are set > >>>>> to a state of "deleted", but are still kept in the dataset, and > >>>>> can > >>>>> be > >>>>> resolved (which would indicate a state of deleted). > >>>>> > >>>>> Kevin > >>>>> > >>>>> > >>>>>>>> "Renato De Giovanni" <renato(a)cria.org.br> 6/05/2008 7:33 a.m. > >>>>>>>> >>> > >>>>> > >>>>> Hi Markus, > >>>>> > >>>>> I would suggest creating new concepts for incremental harvesting, > >>>>> either in the data standards themselves or in some new > >>>>> extension. In > >>>>> the case of TAPIR, GBIF could easily check the mapped concepts > >>>>> before > >>>>> deciding between incremental or full harvesting. > >>>>> > >>>>> Actually it could be just one new concept such as "recordStatus" > >>>>> or > >>>>> "deletionFlag". Or perhaps you could also want to create your own > >>>>> definition for dateLastModified indicating which set of concepts > >>>>> should be considered to see if something has changed or not, but I > >>>>> guess this level of granularity would be difficult to be > >>>>> supported. > >>>>> > >>>>> Regards, > >>>>> -- > >>>>> Renato > >>>>> > >>>>> On 5 May 2008 at 11:24, Markus D?ring wrote: > >>>>> > >>>>>> Phil, > >>>>>> incremental harvesting is not implemented on the GBIF side as far > >>>>>> as I > >>>>>> am aware. And I dont think that will be a simple thing to > >>>>>> implement on > >>>>>> the current system. Also, even if we can detect only the changed > >>>>>> records since the last harevesting via dateLastModified we still > >>>>>> have > >>>>>> no information about deletions. We could have an arrangement > >>>>>> saying > >>>>>> that you keep deleted records as empty records with just the ID > >>>>>> and > >>>>>> nothing else (I vaguely remember LSIDs were supposed to work like > >>>>>> this > >>>>>> too). But that also needs to be supported on your side then, > >>>>>> never > >>>>>> entirely removing any record. I will have a discussion with the > >>>>>> others > >>>>>> at GBIF about that. > >>>>>> > >>>>>> Markus > >>>>> _______________________________________________ > >>>>> tdwg-tapir mailing list > >>>>> tdwg-tapir(a)lists.tdwg.org > >>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Please consider the environment before printing this email > >>>>> > >>>>> WARNING : This email and any attachments may be confidential and/ > >>>>> or > >>>>> privileged. They are intended for the addressee only and are not > >>>>> to > >>>>> be read, > >>>>> used, copied or disseminated by anyone receiving them in error. If > >>>>> you are > >>>>> not the intended recipient, please notify the sender by return > >>>>> email and > >>>>> delete this message and any attachments. > >>>>> > >>>>> The views expressed in this email are those of the sender and do > >>>>> not > >>>>> necessarily reflect the > >>>>> official views of Landcare Research. http:// > >>>>> www.landcareresearch.co.nz > >>>>> _______________________________________________ > >>>>> tdwg-tapir mailing list > >>>>> tdwg-tapir(a)lists.tdwg.org > >>>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > >>>>> > >>>>> > >>>> _______________________________________________ > >>>> tdwg-tapir mailing list > >>>> tdwg-tapir(a)lists.tdwg.org > >>>> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > >>> > >> > >> > >> _______________________________________________ > >> tdwg-tapir mailing list > >> tdwg-tapir(a)lists.tdwg.org > >> http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > > > > -- > > > > Australian Centre for Plant BIodiversity Research<------------------+ > > National greg whitBread voice: +61 2 62509 482 > > Botanic Integrated Botanical Information System fax: +61 2 62509 599 > > Gardens S........ I.T. happens.. ghw(a)anbg.gov.au > > +----------------------------------------->GPO Box 1777 Canberra 2601 > > > > > > > > ------ > > If you have received this transmission in error please notify us > > immediately by return e-mail and delete all copies. If this e-mail > > or any attachments have been sent to you in error, that error does > > not constitute waiver of any confidentiality, privilege or copyright > > in respect of information in the e-mail or attachments. > > > > > > > > Please consider the environment before printing this email. > > > > ------ > > > > _______________________________________________ > > tdwg-tapir mailing list > > tdwg-tapir(a)lists.tdwg.org > > http://lists.tdwg.org/mailman/listinfo/tdwg-tapir > > > > > _______________________________________________ > Hiscom-l mailing list > Hiscom-l(a)chah.org.au > http://chah.org.au/mailman/listinfo/hiscom-l_chah.org.au -- australian centre for plant bIodiversity research<------------------+ national greg whitBread voice: +61 2 62509 482 botanic Integrated Botanical Information System fax: +61 2 62509 599 gardens S........ I.T. happens.. ghw(a)anbg.gov.au +----------------------------------------->GPO Box 1777 Canberra 2601 ------ If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. Please consider the environment before printing this email. ------ ------------------------------ _______________________________________________ tdwg-tapir mailing list tdwg-tapir(a)lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tapir End of tdwg-tapir Digest, Vol 31, Issue 9 *****************************************

1 0

[tdwg-tapir] Tapir protocol - Harvest methods?
by Phil Cryer 14 May '08

14 May '08

Just starting with Tapir/DiGIR - I have 2 questions: * I would like to know if the Tapir protocol is the preferred method over DiGIR. We have a DiGIR implementation that we want to move away from, and bring up a Tapir one in its place. Is this normal, or do organizations run both to facilitate their older clients to do harvesting? * What is a method to harvest data from Tapir, and/or DiGIR - we want to do this internally to test our implementation before we open up to the world, how can I do this (we run Windows and Linux as clients) Thank you Phil -- Phil Cryer Open Source Development Missouri Botanical Garden

11 23

OASIS reference arcitecture for SOA availble for review
by Blum, Stan 10 May '08

10 May '08

This is likely to be very relevant to TDWG. -Stan ________________________________ From: Mary McRae on behalf of Mary McRae Sent: Fri 2008-05-09 9:13 AM To: members(a)lists.oasis-open.org; tc-announce(a)lists.oasis-open.org Cc: soa-rm(a)lists.oasis-open.org Subject: [members] Public Review of SOA-RA v1.0 To OASIS members, Public Announce Lists: The OASIS Service Oriented Architecture Reference Model (SOA-RM) TC has recently approved the following specification as a Committee Draft and approved the package for public review: Reference Architecture for Service Oriented Architecture Version 1.0 The public review starts today, 9 May 2008, and ends 8 July 2008. This is an open invitation to comment. We strongly encourage feedback from potential users, developers and others, whether OASIS members or not, for the sake of improving the interoperability and quality of OASIS work. Please feel free to distribute this announcement within your organization and to other appropriate mail lists. More non-normative information about the specification and the technical committee may be found at the public home page of the TC at http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=soa-rm. Comments may be submitted to the TC by any person through the use of the OASIS TC Comment Facility which can be located via the button marked "Send A Comment" at the top of that page, or directly at http://www.oasis-open.org/committees/comments/index.php?wg_abbrev=soa-rm. Submitted comments (for this work as well as other works of that TC) are publicly archived and can be viewed at http://lists.oasis-open.org/archives/soa-rm-comment/. All comments submitted to OASIS are subject to the OASIS Feedback License, which ensures that the feedback you provide carries the same obligations at least as the obligations of the TC members. The specification document and related files are available here: Editable Source: http://docs.oasis-open.org/soa-rm/soa-ra/v1.0/soa-ra-pr-01.doc PDF (Authoritative): http://docs.oasis-open.org/soa-rm/soa-ra/v1.0/soa-ra-pr-01.pdf HTML: http://docs.oasis-open.org/soa-rm/soa-ra/v1.0/soa-ra-pr-01.html Abstract: "This document specifies the OASIS Reference Architecture for Service Oriented Architecture. It follows from the concepts and relationships defined in the OASIS Reference Model for Service Oriented Architecture. While it remains abstract in nature, the current document describes one possible template upon which a SOA concrete architecture can be built. "Our focus in this architecture is on an approach to integrating business with the information technology needed to support it. The issues involved with integration are always present, but, we find, are thrown into clear focus when business integration involves crossing ownership boundaries. ... "The Reference Architecture has three main views: the Business via Service view which lays the foundation for conducting business in the context of Service Oriented Architecture; the Realizing Services view which addresses the requirements for constructing a Service Oriented Architecture; and the Owning Service Oriented Architecture view which focuses on the governance and management of SOA-based systems." OASIS and the Service Oriented Architecture Reference Model (SOA-RM) TC welcome your comments. --------------------------------------------------- Mary P McRae Manager of TC Administration, OASIS email: mary.mcrae(a)oasis-open.org web: www.oasis-open.org --------------------------------------------------------------------- This email list is used solely by OASIS for official consortium communications. Opt-out requests may be sent to member-services(a)oasis-open.org, however, all members are strongly encouraged to maintain a subscription to this list.

2 2

[tdwg-tapir] r1389 - tapir/dev/schema
by svn＠rs.tdwg.org 08 May '08

08 May '08

Author: root Date: 2008-05-08 14:28:26 +0200 (Thu, 08 May 2008) New Revision: 1389 Removed: tapir/dev/schema/test.txt Log: Rolling back changes made during tests. Deleted: tapir/dev/schema/test.txt =================================================================== --- tapir/dev/schema/test.txt 2008-05-08 12:24:32 UTC (rev 1388) +++ tapir/dev/schema/test.txt 2008-05-08 12:28:26 UTC (rev 1389) @@ -1 +0,0 @@ -This is a test. Please disregard. \ No newline at end of file

1 0

[tdwg-tapir] r1388 - tapir/dev/schema
by svn＠rs.tdwg.org 08 May '08

08 May '08

Author: root Date: 2008-05-08 14:24:32 +0200 (Thu, 08 May 2008) New Revision: 1388 Added: tapir/dev/schema/test.txt Log: This is a test. Please disregard. Added: tapir/dev/schema/test.txt =================================================================== --- tapir/dev/schema/test.txt (rev 0) +++ tapir/dev/schema/test.txt 2008-05-08 12:24:32 UTC (rev 1388) @@ -0,0 +1 @@ +This is a test. Please disregard. \ No newline at end of file

1 0