[tdwg] Ideas on having Harvesters like GBIF clean, flag inconsistencies, and add additional value to the data

[Peter DeVries]

Arthur Chapman sent me some good comments regarding Datums etc.
The discussion made me realize that there may be a need for two types of
formats. One for the providers and a second one that is output by the
harvesting service.

This is because the needs and abilities of the data providers are different
than the needs and abilities of those who would like to consume the data.

Consumers, who analyze and map the data, would like something that is easy
to process, standardized and as as error free as as possible.

It could work in the following way.

Data harvesters, like GBIF, collect the records. Run them through
cleaning algorithms that check attributes including that the lat and long
actually match the location described.

These harvesters would then expose this cleaned data via XML and RDF with
tags that flag possible inconsistencies. The harvesters would also add a
field for the lat and long in WGS84 if the original record contains a valid
Datum. Those records without a Datum would still be exposed but the added
geo:latitude and geo:longitude fields would be empty.

I can imagine that that data uploaded to GBIF and other harvester services
will be replete with typo's and inconsistencies that will frustrate people
trying to analyze or simply map the data, the harvester services could add
value by minimizing these frustrations.

Originally, it seemed that a global service should standardize on a global
Datum like WGS84. After all, we have standardized on meters? However, after
discussing this with Arthur, I realize that this is not possible for a
number of reasons. That said, I think the data would be much more valuable
and less likely to be misinterpreted if if a version of it was available in
WGS84. This solution would eventually encourage data providers to understand
what a Datum is and include it in their data. It would also help solve a
number of other data integration problems.

Respectfully,

Pete


---------------------------------------------------------------
Pete DeVries
Department of Entomology
University of Wisconsin - Madison
445 Russell Laboratories
1630 Linden Drive
Madison, WI 53706
------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.tdwg.org/pipermail/tdwg/attachments/20090508/481cc584/attachment.html