[tdwg-tapir] Fwd: Tapir protocol - Harvest methods?

[Renato De Giovanni]

I agree with Markus about using a simple data format. Relational database
dumps would require standard database structures or would expose specific
things that are already encapsulated by abstraction layers (conceptual
schemas).

I'm not sure about the best way to represent complex data structures like
ABCD, but for simpler providers such as TapirLink/Dwc, the idea was to
create a new script responsible for dumping all mapped concepts of a
specific data source into a single file. Providers could periodically call
this script from a cron job to regenerate the dump. The first line in the
dump file would indicate the concept identifiers (GUIDs) associated with
each column to make it a generic solution (and more compatible with
existing applications). Content could be tab-delimited and in the end
compressed.

Harvesters could use this "seed" file for the initial data import, and
then potentially use incremental harvesting to update the cache. But in
this case it would be necessary to know when the dump file was generated.

To use the existing TAPIR infrastructure, we would also need to know which
providers support the dump files. Aaron's idea, when he first discussed
with me, was to use a new custom operation. This makes sense to me, but
would require a small change in the protocol to add a custom slot in the
operations section of capabilities responses. Curiously, this approach
would allow the existence of TAPIR "static providers" - the simplest
possible category, even simpler than TapirLite. They would not support
inventories, searches or query templates, but would make the dump file
available through the new custom operation. Metadata, capabilities and
ping could be just static files served by a very simple script.

If this approach makes sense, I think these are the points that still need
to be addressed:

1) Decide about how to indicate the timestamp associated with the dump file.
2) Change the TAPIR schema (or figure out another solution to advertise
the new capability, but always remembering that in the TAPIR context a
single provider instance can host multiple data sources that are usually
distinguished by a query parameter in the URL, so I'm not sure how a
sitemaps approach could be used).
3) Decide about how to represent complex data such as ABCD (if using
multiple files, I would suggest to compress them together and serve as a
single file).
4) Write a short specification to describe the new custom operation and
the data format.

I'm happy to change the schema if there's consensus about this.

Best Regards,
--
Renato


> it would keep the relations, but we dont really want any relational
> structure to be served up.
> And using sqlite binaries for the dwc star scheme would not be easier
> to work with than plain text files. they can even be loaded into excel
> straight away, can be versioned with svn and so on. If there is a
> geospatial extension file which has the GUID in the first column,
> applications might grab that directly and not even touch the central
> core file if they only want location data.
>
> I'd prefer to stick with a csv or tab delimited file.
> The simpler the better. And it also cant get corrupted as easily.
>
> Markus
>
>
>
> On 14 May, 2008, at 15:25, Aaron D. Steele wrote:
>
>> for preserving relational data, we could also just dump tapirlink
>> resources to an sqlite database file (http://www.sqlite.org), zip it
>> up, and again make it available via the web service. we use sqlite
>> internally for many projects, and it's both easy to use and well
>> supported by jdbc, php, python, etc.
>>
>> would something like this be a useful option?
>>
>> thanks,
>> aaron