I'm afraid this doesn't make much sense to me so I'll have to let the rest of you sort it out. When you come up with a solution we'll see if we can implement it in the morphology section of BioLink.
Sorry about that, Steve
Steve Shattuck
CSIRO Entomology
steve.shattuck(a)csiro.au
-----Original Message-----
From: Gregor Hagedorn
To: TDWG-SDD(a)USOBI.ORG
Sent: 9/2/2003 9:36 PM
Subject: Special states: stepping back to the generalized character definition and coding model
Steve Shattuck wrote:
> For "Special States", another way to look at it is that this doesn't
> really relate to a character but to the coding of a given character
> for a given taxon. It is really the status of the coding of that
> character for a taxon. Instead of storing it as a part of the
> character (by creating a new state for the character) why not create a
> new element for this information and attach it to the taxon by
> character intersection (or "cell", or "taxon description"). Call it
> "Coding Status" with an enumerated list of "Coded" (meaning the taxon
> has been coded for this character), "Not yet coded" (meaning it will
> be coded when I get around to it), "Not to be coded" (meaning it can
> be coded but I not going to) and "Can't be coded". Then add a text
> attribute to this element to allow an explanation of what's going on
> for the coding of this character for this taxon ("I haven't yet coded
> this character because suitable material of this taxon is currently
> unavailable"). This would seem to meet the needs of computer
> processing (through a defined, machine-readable enumerated list),
> extensibility (by supporting additions to this list such as "Coded but
> unreliable") and a full, human-readable explanation of what's going on
> (in the text comment).
I agree on the need to add an option for annotation text and to keep
the "coding status/missing data indicator" list relatively short.
I also can basically accept your list, except that the last example
usually applies to a coded state, not the character, i.e. one state
may be coded unreliably, the other ok. Currently we propose to use
uncertainty modifiers (probably, perhaps, etc.) for this case. This
is similar to the "by-misinterpretation" mechanism originally
proposed by Kevin, now proposed in SDD to be handled through a coding
modifier on an existing state.
General question to all: Any examples where having
uncertainty/unreliability should apply to the entire character rather
than to the states, i.e. where if implemented as a modifier this
would always have to be added to all states?
I wonder whether the "Coded" option is redundant. This should be
evident if no other CodingStatus is present and data are coded.
Except for redundancy (and therefore the need to enforce a rule that
"Coded" requires categorical or numerical data to be indeed coded;
else it would be a "lie"), I have no problem with it.
> We need to take a step back before we can move forward.
I am quite willing to take that step back. Let my try to explain the
reasoning why I think it is a good idea to generalize "coding
status/missing data indicators" to be handled akin to categorical
states:
The current SDD model for categorical characters is:
1. It is possible to defined shared state sets globally. Example:
color-states: red, green, blue, ...
2. The character definition list allows to define
a) local states
b) refer to 0 to several shared state sets (= inherit them)
alpha) as a whole set
beta) restricting the inheritance to states
specified in an enumeration
(Note: if the shared definition is expanded, alpha
automatically inherits the new character, beta not)
3. The item descriptions always refers to states defined in
the character definition list, never to global/shared
definitions directly.
----------
I view this as a structural generalization, and tried to model
statistical parameters like min/mean/max/s.d./s.e./sample size etc.
accordingly:
1. Global measures define the semantics of for the first version of
SDD perhaps 30 parameters. In the first SDD version to keep it
simply, the list would not yet be exandable, but it is modeled such
that it is easily expandable later on. Statistical parameters already
carry descriptive attributes that allows for most purposes to
interpret them without referring the state code.
2. Within a numeric character in Terminology: character definition it
is possible to select only specific parameters.
3. in the item description only parameters selected by the designer
of the terminology can be used.
----------
Now in that admittedly confusing "special state" paper from March I
tried to figure out whether we should really (as implemented in
Brazil) use this same generalization for "special states" or "coding
status/missing data indicators"
1. A shared state set defining special states with fixed codes (i.e.
not user expandable in the first SDD version, but since all
information is already in the data, easily expandable in future
versions.
2. Within a character in Terminology: define which of these
status/indicator states are to be enabled for a given character.
3. The item descriptions always refers to states defined in
the character definition list, never to global/shared
definitions directly.
----------
You propose to handle them separately. What would be the advantages
and disadvantages? Please add to the list if I overlook something:
Advantages:
a) avoid confusion in the definition of the terminology between
normal categorical states and "coding status/missing data
indicators". These things are structurally related in that both are
specific to a given character when the character is coded, but yield
information from different (although non-overlapping!) knowledge
domains, viz. biology for character states and knowledge meta data
for "coding status/missing data indicators".
Disadvantages:
a) the list is fixed and changes to the software are required if the
next SDD version extends another coding status.
b) reports can be generated only in the language of the program. If
the data are in German, but the software is English, the software
must add a special feature to translate the reporting of this
information into German.
c) when collating/compiling data from children, "coding
status/missing data indicators" must be handled identical with normal
data. That means especially, that the "coding status/missing data
indicators" must support multiple concurrent states:
Item 1, character 1:
state 1
Item 2, character 1:
coding status "excluded/scoped out"
Item 3, character 1:
coding status: "Not applicable"
if these 3 items are species within a genus, the compiled genus
description would be:
genus item, character 1:
state 1, or "excluded/scoped out", or "Not applicable"
In which type of report this information would be made visible is
another question, i.e. a difference between the types of states
remains. However, it would be clearly inappropriate to summarize this
into just "state 1".
Note that I was especially delighted to see that probably the entire
question of character scoping automatically falls into place, without
any additional structural complexity, if "scoped out" can be
inherited as a default value from the taxonomic parents! This is my
only excuse that I included preliminary notes on the issue of
default states in the special state paper, which made the paper a lot
more confusing!
---------------
This is my list, and on the basis of this list I would prefer to keep
the structural generalization, rather than implementing a completely
different mechanism. However, it is definitely good to take that step
back and rethink this.
To all: please respond if I currently overlook issues here!
However, even if "coding status/missing data indicators" are handled
analogous to shared categorical states and statistical parameters,
this does not mean, they have to be in one xml element. Whether a
database would store them in one table (would make sense to me) is
quite independent of this.
So, we could have (as an example, not necessarily the acutal schema):
Terminology
SharedStates
StatisticalParameters
MissingDataIndicators / or CodingStatus
CategoricalStates
CharacterDefinition
StatisticalParameters
MissingDataIndicators / or CodingStatus
CategoricalStates
Description
Character keyref=...
StatisticalParameters
MissingDataIndicators / or CodingStatus
CategoricalStates
----
Or:
Terminology
SharedStates
StatisticalParameters
MissingDataIndicators / or CodingStatus
CategoricalStates
CharacterDefinition
<StateDefinition xs:type="StatisticalParametersDef" key=...>
<StateDefinition xs:type="MissingDataIndicatorsDef" key=...>
<StateDefinition xs:type="CategoricalStatesDef" key=...>
Description
Character keyref=...
<StateReference xs:type="StatisticalParameterRef" keyref=...>
<StateReference xs:type="MissingDataIndicatorRef" keyref=...>
<StateReference xs:type="CategoricalStateRef" keyref=...>
------
Which is preferable?
------
Technical aside regarding the second version: I still don't fully
understand the implications of using the latter subtyping
("substitution group"?) version in regard to key/keyref constraints.
One thing is: is it possible to force the schema that xs:type is
required? It seems that this is only optional, and that if it is not
present it is up to the consuming application to figure out which
subtype is acutally used. However, whereas the CharacterDefininition
subtypes (StatisticalParametersDef) differ in their complex content
and can be recognized, the StatisticalParameterRef do not differ
EXCEPT in the keyref constraint. My guess is that therefore the above
does not work in xml schema, but please comment if you have insight
on this!
Gregor
----------------------------------------------------------
Gregor Hagedorn (G.Hagedorn(a)bba.de)
Institute for Plant Virology, Microbiology, and Biosafety
Federal Research Center for Agriculture and Forestry (BBA)
Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220
14195 Berlin, Germany Fax: +49-30-8304-2203
Often wrong but never in doubt!
