versioning for data versus metadata

[Matt Jones]

In my previous email I referred to problems in distinguishing data and
metadata.  I'd like to elaborate here.

Traditional definitions of metadata are 'data about data' or 'data
documentation'.  These are good definitions from a pragmatic standpoint,
but become somewhat less than helpful when trying to build real working
systems that utilize both data and metadata and try to preserve
replicability of analyses through versioning.  A simple example will
illustrate.

Sometimes people record repeating information about data as separate
metadata (for example, the date on which data were collected).  Other
times, they might include that information directly in their data model.

Take two entities, A and B:

Entity A:
---------
Metadata:
AttributeLabels = Site,Date,Abundance

Data:
Foo  20041010  19
Bar  20050712  20
Foo  20051010  20

Entity B:
-----------
Metadata:
AttributeLabels = Site,Abundance
CollectionDate = 20011002

Data:
Foo  24.3
Bar  21.3
Baz  20.4

Note that both entities contain the same information, but the second
places the date of collection as a metadata property, while the first
puts it in the data model. If one were to integrate these data entities
to produce a time-series plot of abundance by site, one would need to
extract the CollectionDate information from the metadata of entity B
before proceeding. Thus, for the purpose of an integrated analysis,
"CollectionDate" is really data.

Which way people model the information is somewhat of an arbitrary
decision, but typically comes down to looking at 1) rate of change of
the information across tuples, and 2) intended use of the final data.

So, to bring this back to the identifier issue.  If one were to assign
an identifer to Entity A and another to Entity B, resolving the
identifier should allow one to retrieve the data.  But in these two
cases the data that is returned will have different schemas and will
have different dependencies on the metadata.  To do integrated analyses
of the two entities, one really needs to be able to utilize both the
metadata and the data together and be assured that both are consistent.

LSIDs require that the data retrieved from an LSID never changes to
guarantee replicability and persistence, but allows the metadata to
change.  Clearly, if the 'CollectionDate' metadata for entity B were to
be changed any analyses that were performed using the original metadata
could no longer be replicated.  This causes a lot of trouble for
analytical systems that emphasize provenance and lineage for derived
data products.  It indicates that there is a strong case to be made that
metadata should be versioned as well and that both the metadata and data
associated with an identifier really should be immutable with respect to
the identifier.  Any changes to data or metadata should require updates
to the identifier revision.

Matt
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Matt Jones
jones at nceas.ucsb.edu                         Ph: 907-789-0496
National Center for Ecological Analysis and Synthesis (NCEAS)
UC Santa Barbara     http://www.nceas.ucsb.edu/ecoinformatics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~