Special states

Steve at Steve at
Tue Mar 18 15:43:01 CET 2003

Hi All,

I've had a look at the excellent "Special States" document
l) put together by Gregor and a few comments are attached below.

These comments are in several sections (with a few notes at the bottom).
Have a look at the first and last sections and the long second section
if you are expecially interested.  This second section is pretty long
and rambles a bit, but covers a wide range of issues centred on Gregor's
document.  Hopefully it will help move the SDD forward.

Thanks, Steve


SDD Special States

At some level the details of the SDD structure are less important than
its content.  This is because few (if any) consuming applications will
use SDD other than as a source of information, this information being
translated into application-specific formats for processing and
analysis.  This requires mapping SDD items to local items and this can
be done for any given SDD structure.

Taking this approach, I would make only one suggestion: generalize,
Generalize, GENERALIZE!  What I mean is that the current Special States
document attempts to define all of the possible Special States that are
to be part of SDD and even goes so far as to exclude some potential
Special States.  I would strongly suggest that SDD have a general method
for handling these, with at most a list of suggested Special States
rather than trying to be too restrictive.  I firmly believe that it will
be extremely difficult (and short sighted) to try to define these as
part of the SDD Standard.  The list of useful Special States is much
longer than the few suggested in the current document, and an extension
is even suggested at the end of the document, a "ToDo" Special State.
Many more can be thought of with little effort.

This same difficulty (trying to be too specific) has arisen in related
activities within TDWG and has resulted in an unfortunate split among
participants.  There are two camps within the specimen-based standards
group, one arguing for a complete description of specimens, the other
looking for a set of common items across most groups.  The first group
has developed a "standard" with some 400 items while the second has
selected in the neighbourhood of 20 or so items.  My impression is that
SDD is taking the first approach: study the problem long enough and the
entire universe of data items and their relationships can be documented.
I'm not confident this is possible.  Biology is too diverse and needs
change too quickly to document all of life in any stable, meaningful and
useful way.  We can provide a framework and make suggestions, but we
won't be successful if we try to dictate too narrowly.

So my suggestion is that we focus on general needs, such as support for
Special States, but that we don't try to define other than at most a few
basic data items (such as general Special States).  What we provide is a
mechanism to support this functionality with the ability to extend it in
a controlled and flexible way.


OK, this is where it gets ugly.  I have a few specific comments on the
current Special States document, some relating to general issues, some
to specific comments.  These are less important than the message above
and largely represent a slightly different perspective on SDD from the
one given in the current document.  

First, the scope of SDD.  Currently SDD is trying to address three
separate and non-overlapping issues: (1) data, (2) business practices
and (3) application development.  Data is the specific information items
under consideration, business practices are how we think of this items
and how we handle them, and applications are specific pieces of software
used by manage data items.  The current Special States document mixes
these three areas, talking about data items, how people interpreter them
and how they might be displayed by a computer program.  The first area
is clearly within the scope of SDD, the second possibly but only in a
very general, non-specific way, and the third is clearly outside SDD
(and well it should be).  Bob made essentially this same comment when he
said that he thought "project management problems were not part of SDD."
We need to focus on DATA and nothing more.

The second serious problem I have with Special States is that they
aren't "states" in the normal, taxonomic sense.  Taxonomic "states" are
observations made about biological objects (be they taxa or specimens)
that are grouped together as "characters."  We use "states" to describe
items.  However, Special States aren't observable and aren't used to
describe items.  

What are "Special States"?  They are a mix of two separate sets of
information.  They document the status of data coding (for example
"unknown", "not interpretable" and "unfinished work") and they document
application-specific business rules (for example "use character default
state" - get the data based on a flag set for a specific state of this
character).  These are fundamentally different things and show the
danger of mixing data, business practices and application development.  

Lack of appreciation for these differences as resulted in "use default"
being treated as a case of "data have not yet been entered" when in fact
a state has been clearly indicated (through a pointer): it's the default
state (however the data or application defines this) and is certainly
known.  It's a user-interface shortcut used by an application to make
data entry quicker (in the opinion of an application developer!).  The
end result is exactly the same as if the state had been entered directly
in the description (the taxon by character intersection, or what DELTA
calls the Attribute).

This thinking results in some very complex processing being required by
consumers of SDD documents.  For example, it is suggested that if a
description is null, then the "special state" called "use default"
should be inserted and that this "state" be set to "not yet evaluated."
This means that to interpret an SDD document you not only have to
understand the individual data items but also know the rules for
processing them (to know that if data is missing you need to insert a
special value and then look this value up in another list to translate
it into something meaningful).  There must be a cleaner way to do this!

The "unknown", "not interpretable" and similar "special states" are
another problem altogether.  These "states" have nothing to do with
characters and everything to do with descriptions.  This is why Mike
Dallwitz didn't include them as part of the Character List.  I can
understand why you might want to treat these as "states" of characters
as this is the basic DELTA paradigm.  However, there is no need to do
this and doing so only complicates things.  For example, to tell if a
description has not been coded you'll need to get its state(s) and then
check their translations to see if any are "unknown", "not
interpretable" or any one of who knows how many other special

A much better way to implement this functionality would be to store an
"uncoded" flag with the description along with an (encoded or
text-based) explanation ("unknown", "not interpretable", "too lazy to
code this", "don't have proper specimens", "To Do" or what ever).  This
is both direct and allows the explanation to change in a simple and
flexible way.

These are the most serious and fundamental problems I have with the
current document.  I would also add the following minor comments (some
of these expand on the above comments as well).

The meaning of the DELTA "U" in attributes:  It's stated in the document
that this means "attempts to research the information were made but
failed" and that "the state U in DELTA is also used for cases where data
are present, but the author is unable to interpret them in current
terminology."  The DELTA documentation states simply that "A missing
attribute is equivalent to an attribute with pseudo-value U" and that
"If the state of the character is unknown, then the character is
omitted, or the state value coded as U."  I can't find anything in the
DELTA documentation that gives a reason for the use of U or suggests
that "attempts to research the information were made but failed" or
anything similar.  It would seem that DELTA treats nulls and U as the
same, nothing more.

Remove all sections dealing with "performance", "user-interfaces" and
referring to "applications."  These deal with specific software
developments and should not influence the SDD standard.

Great care should be taken when developing "business rules" for
interpreting an SDD document.  For example see the discussion starting
with "One frequent situation is that characters are added to the
terminology.  Possible solutions to achieve a synchronization between
descriptions and a separate terminology in this case are:".  It may be
important for SDD to document this situation but HOW it is dealt with is
up to individual applications and has little to do with the data itself
(which is the focus of SDD).  The same problem occurs with the statement
"The omission of character coding in a description should be used to
express the "use default" state."  This mixes data with processing.  The
lack of data is simply that - it implies nothing more and NO assumptions
should be made about it.  If the default state should be used then the
author needs to state this.

What does "terminology" mean?  It seems to be the character/character
state list but this is unclear.  It might be good to develop a glossary
of terms so we all know what we're talking about.

We also need to define "schema."  My understanding is that a "schema" is
a model of the structure of the data and not the data itself.  If you
add data you don't change the schema (assuming that this addition is
permitted by the schema - it's like adding rows to a database table -
the schema doesn't change, only the data).  But at several points the
document says "... the case that the terminology is changing ("schema
evolution") is ..." or similar.  This sounds like changes to the data
change the schema.  Is this the general view of schema or am I on the
wrong track?

The section discussing "Data have not yet been entered", "Data cannot be
entered" and "Data could have been entered but a deliberate decision was
made not to enter them" seems to mix unrelated cases and is a mess.  The
options listed are (followed by my interpretation of their meaning):
1) "Use character default state": Data is coded and is not missing (see
2) "Unfinished work": Data exists but has yet to be captured.
3) "Not applicable": Data does not logically exist.
4) "Unknown": Data exists but has yet to be captured.
5) "Not interpretable": Data exists but has yet to be captured.
6) "Out of scope": Data exists but has yet to be captured.
7) "Do not need to score": Data is coded and is not missing.
The mix involves data that is being pointed to through a process (No. 1
and 7), data that is impossible to collect (No. 3) and data that can be
collected but hasn't (for a variety of reasons) (No. 2, 4, 5 and 6).
Keep it simple by recording if data is absent and if it is, the reason.
Don't confuse it with application-specific short cuts ("Use default",
"Inherit/Compile from Parent/Children") or personal decisions made by an
author ("it's too hard to code", "this character is unimportant here",
"appropriate specimens are unavailable", etc).  These are distinct

There are a number of cases where attempts are made to develop lists of
conditions or situations.  For example, it is asked "Should a general
'cannot score' (for any kind of reason) be differentiated into" and then
three situations are listed.  Again, generalise this into "uncoded" with
a text-based explanation ("observation method failed", "incomplete
specimen" or any one of a thousand other reasons).  All the information
is captured and the system is rich enough to handle unforseen
situations.  This approach also fits perfectly with the previous

There is a note that "a general method is planned (@but not yet
formalized!@) in SDD" and that "SDD is considering introducing computed
characters."  We've been at this for at least 2 years now and there are
things that are "planned" or are "being considered" for SDD that we
don't know about?!?!?  Or is there an overall planning document that I'm
not aware of?  WE are SDD, SDD can't be "considering" things in

The entire discussion of inheritance and compilation (or what ever we
call it) needs to be though of in a fresh light.  Again, the document
confuses data with process.  For example, it's stated that "it is
desirable that the assumption [of inheritance/compilation] rather than
the inherited data are recorded".  The author of the dataset MUST
present real data, not give a process to get to that data.  Yes, the
author can tell us how they collected the data ("inherited from direct
parent", "compiled using Gouldes Statistic from all coded children", "by
reference to the default state for this character") but they also must
give us the data, not make us go look for it.  The process of
inheritance/compilation is too complex to leave to chance and trying to
define exactly what it means is too error prone.  I would suggest that
inheritance/compilation is an application-based activity, not a
fundamental aspect of the data.  Again, give us the data and separately
tell us how it was derived, don't make us derive it ourselves.

This exact same problem exists with "computed data."  I don't see how or
why an SDD document can contain the value "CouldNotCompute" because the
"generator couldn't compute this".  The "generator" is a piece of
software, not a data representation (which is what SDD is).
"CouldNotCompute" is an error message returned from an application and
it should never end up in an SDD document.  Yet again, if you want to
explain how the data was generated, find, do it, but don't make us
calculate it, we don't know how.

The discussion about "Not supported as special states, but supported
through modifiers" is good but, as above, describes descriptions and not
characters.  That's why inheritance/compilation is included here.

The statement "Special states express knowledge about why data for a
given character are missing in a description and thus make a statement
about the entire character" is interesting.  It says that special states
are about a character in a description.  The point I would make is that
they are primarily about the description, not the character.  They say
nothing about the character's use for other items, and only relate to
this character in this taxon, and this is through the "description."  So
let's focus this document on the description and only discuss the
character when it's appropriate.  This is supported by the statement
that "DELTA does not define the special states in the "character list"
directive: they are implicitly present in each character."  They are not
in the character list because the have little to do with characters.
They are in the DELTA attributes because they have everything to do with
descriptions.  I'm sorry to say that the next statement shows the danger
of letting past developments influence current work (and I'm as guilty
as any): "When a new character is created, DeltaAccess automatically
creates the full set of special states."  In BioLink we attach this
information to the description: you do it when you're coding the data,
not developing the character list.  I don't know which way is better and
as noted above, it probably doesn't matter as each application will
translate the information into a local format anyway.  What we DO need
to do is make sure the SDD format is expressive, flexible and as
application-neutral as possible (and ideally simple).

Most of the discussion under "Relations between declarative character
dependency and special states" seems to deal with developing
applications rather than data representation and may well be outside
SDD.  Sure, document that these dependencies exist and use them to
produce high-quality and internally consistent data, but do this at
application-level, not SDD-level.

The same applies to "Responsibility for validation."  SDD should be
about representing data in a standardized way, not about enforcing
taxonomic business rules or data quality standards.  As SDD will be
represented in XML then this XML document must be well-formed and pass a
check with a DTD or XML-Schema checker.  But SDD shouldn't be
responsible for checking the content of individual data items, that's
the job of authors and their applications.


So, to summarize:

SDD is about data and only data.  These data include characters and
descriptions and (optionally) how these descriptions were developed and
their current status.

SDD is not about process.  Process information can be included but it is
optional and can be safely ignored without data loss.  [I shouldn't have
to process a "use default data" statement to access a complete
description, the appropriate state(s) should be inserted into the
description when the SDD document is prepared with a note on how it was

SDD is not about applications.  Discussions concerning user-interfaces
or processing methods should not be included.  [How an application
manages and represents these data is completely independent of how SDD
represents the data.]


Finally, I want to reconfirm that Gregor has done a fantastic job with
getting us this far and I support his efforts.  My comments are more
about a different perspective rather than any serious flaw in logic.
Most of what is in the document is well presented and relevant to
managing taxonomic descriptions, I'm just not sure it is directly
related to the goals of SDD.

And sorry for going on for so long.  I'll sit down now.


Steve Shattuck
CSIRO Entomology
Steve.shattuck at csiro.au

More information about the tdwg-content mailing list