- tdwg-content - lists.tdwg.org

Re: TDWG-SDD Digest - 15 Aug 2003 to 27 Aug 2003 (#2003-16)
by Eric Gouda 28 Aug '03

28 Aug '03

At Thursday 28-8-2003, you wrote: > <snip....> >2. However, several aspects of complex descriptive data sets need > an operational access to a taxonomic hierarchy. > -> default values for new items > -> observations assumed to be true for an entire group but not > actually observed should be noted as "higher taxa" descriptions > and the information should be treated as implied in lower taxa. > The situation is identical for implying information from > species to subspecies as it is from Family to species. > -> Note that in contrast to DELTA, SDD has no special > "subspecies mechanism" Just some thoughts: I think a transparent mechanism is needed in which you can indicate in the item description which rank a taxon (item) is and you can include as many taxon ranks as you like (the subspecies mechanism is included then). Taxa can link to higher (and maybe lower) taxa, so the program that is going to produce Natural Language Descriptions can include descriptive characters from several ranks (as indicated by the user). Identification programs can start at a chosen rank and continue to the next rank taxa when finished. In XML this would be easy to define. In this way you could key out specimen for a whole family or flora when included, from family to subforma level if you like. Eric +++ ____ ____ E.J.Gouda(a)bio.uu.nl http://www.uu.nl/botgard/ / ___|_ __(_) __ / ___| ___ _ _ __| | __ _ Faxnr: +31 30 2535177 | |__ | '__| |/ _| | | _ / _ \| | | |/ _` |/ _` | Prive: Jungfrau 107 | |___| | | | (_ | |_| | (_) | |_| | (_| | (_| | NL - 3524 WJ Utrecht \____|_| |_|\__| \____|\___/ \___/ \__,_|\__,_| THE NETHERLANDS (NL) Curator of UNIVERSITY BOTANIC GARDENS, PO Box 80.162, NL-3508 TD UTRECHT \ Listowner: Alpine-L, Arisaema-L, Bromelia-L, DELTA-L, Passiflora-L / \__ Penstemon-L, Trillium-L (at Nic.SurfNet.NL) __/

1 0

Special states: What term to use instead?
by Gregor Hagedorn 28 Aug '03

28 Aug '03

Steve Shattuck in March responded with a long and detailed criticism of my Special States proposal/Brazil minutes mix, for which I am very grateful. I currently try to revise the information. One important point Steve raised was that the term "Special States" is very confusing because, although they are documenting knowledge management metadata in the object description, they are not terminological states in the character definition. I was never happy with "special states", but neither in Brazil nor in Paris we found a better name. The current options are: <li>Management states (not "internal" data management)</li> <li>Data scoring</li> <li>Status states</li> <li>Status flag</li> <li>Data entry status</li> <li>Control states</li> <li>Special states</li> <li>Completeness states</li> <li>Incompleteness states</li> <li>Completion states</li> <li>Pseudo character values (used e. g. by L. Dodds in XDELTA proposal)</li> <li>Replacement values</li> <li>Meta states ("meta ..." was considered too unspecific, being only a relative statement)</li> <li>Missing data indicators</li> <li>Missing data flags</li> <li>Null states</li> <li>Null values</li> <li>Knowledge constants/values/flags</li> "Data entry status flags" seems appropriate, but awfully long. "Knowledge management state" seems to be a bit too general, because it is only part of Kn.man. "Missing data flags" can easily be misunderstood as "missing data- flags" rather than "missing-data flags" Has anybody a bright idea? ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Taxonomic hierarchy
by Gregor Hagedorn 27 Aug '03

27 Aug '03

Eric Gouda wrote: > I'm highly > interested in a DELTA like format that is XML based (done a lot with > XML lately) to provide hirarchical features in the items part. This is a part we have purposely excluded in the discussions in Brazil and Paris so far, and perhaps it is time to open a discussion about this on the list. The answer to this discussion looms in the background of the current SDD outline, since it may cause major structural rearrangments. I therefore believe it important that we deal with it. The reason for not taking on the object hierarchy of the described items is that it has two aspects: 1. Fundamental evolutionary hierarchy = taxonomic systematics -> in my opinion it is not the purpose of a descriptive data standard to cover this. The hierarchisation of taxa into an evolutionary tree is a complex and contentious process that is independent of the fundamental data that are being observed (rather than implied by the taxonomic hierarchy) -> just like taxon concepts, obligate and facultative synonymy, the the hierarchisation of taxa should be covered by a separate taxonomic standard. -> to produce complete Flora/Fauna treatments, these standards need to work together. -> Note that a new TDWG "Taxonomic Names" xml standards group has been formed. The convener is Jerry Cooper and it will meet the first time at the TDWG meeting in Portugal 2. However, several aspects of complex descriptive data sets need an operational access to a taxonomic hierarchy. -> default values for new items -> observations assumed to be true for an entire group but not actually observed should be noted as "higher taxa" descriptions and the information should be treated as implied in lower taxa. The situation is identical for implying information from species to subspecies as it is from Family to species. -> Note that in contrast to DELTA, SDD has no special "subspecies mechanism" -> scoping characters in/out for specific groups. Scoping means that characters are only applicable or desirable to code only in certain groups (in contrast to character dependency, which is a reflexive relationship within descriptive data) (What other data or features depend on the taxon hierarchy? Can anybody provide good discussion examples?) So, I have no good solution for this. One option would be to simply allow any operational hierarchization in SDD and not care about taxonomic codes, standards, etc. at all, only providing a linking mechanism to taxonomic data sets. Is that desirable? Gregor ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Complexity of SDD format
by Gregor Hagedorn 27 Aug '03

27 Aug '03

Hello, my apologies to all list members for dropping out of the discussion for almost 5 months. I was overwhelmed with other work and could not get out of just feeling guilty but not picking the work up again. I am back from other work and also just back from vacation. Steve Shattuck wrote: > Kevin is pretty much right that I see the main use of SDD as > documenting essentially finished data rather than a "work in > progress". My thinking is that most people will work on their dataset > until it's finished, then make it available through SDD to other > interested parties. My assumption is that most collaborative projects > will select a single tool to use while building the dataset (be it > DELTA, LucID, DeltaAccess or what ever) and will use the native format > of this application to share data. > Now, I have no problem with capturing details needed by specific > situations or applications and would say that these are important and > should be support by SDD. However, I would try to make them > extensions to the basic, core data rather than part of the core > itself. I agree with your concern about complexity. It already shows that the complexity present is difficult to document. We urgently need more introductory material. However, I disagree about the notion of "finished datasets". I may be wrong, but I think that this is a thing of the past. The current process of publishing scientific information in unchangeable batches is primarily a function of the process of publishing printed materials, which are difficult to revise. I believe in the future we will move into continuously updated and revised versions. The granularity in terms of time may be months or years, but not decades as it is currently the case. Especially when I am thinking about collaboration I am thinking not about 2 or 3 people working closely, but a loose network of collaborators. That also includes that datasets are passed on from one generation to the next. If some researcher created a very valuable dataset as a finished work, then retired and a new Ph.D. candidate revises this dataset, it will be irrelevant whether creation and intellectual property details are in the SDD set. However, the outcome of this will be a dataset perhaps 5% improved, 95% original material. Now I believe it is undesirable to pass the entire dataset on to the next reviser without any indication of what happened. In this second step, I believe it is the task of the SDD exchange format to record some fundamental attribution of intellectual property. I am, however receptive to wisdom that caring for this (web updated data etc.) makes our task too complicated. > This leads to two thoughts: > > First, it is crucial for SDD to be the product of a team with > differing views who work together towards some middle ground. > Otherwise it is unlikely that it will meet the needs of the broader > community. > > Second, I worry about SDD becoming too specialised and too complex. > The "joy" of the DELTA standard is its simplicity. And this > simplicity is, I would say, one of the reasons it has been so > successful. I would contrast this with the Nexus format. Nexus is > only seriously being used by the 3 or 4 applications involved in its > original development. It is hardly a "standard" outside this small > community. The power of Nexus is that it supports essentially all of > the features of these original applications. This seems to be the way > SDD is heading: support all of the features of character-based > programs that currently exist. My concern would be that if we do this > we will be the only people who use SDD, new players finding the format > so complex and so specialised that it's expensive to implement and has > little flexibility to support "extensions" outside of the strict > standard. If this is true and is realised we all lose. Here I would maintain that we do you existing programs to help us in finding the requirements, but that the presence of a feature in a program never had any bearing on whether it should become a feature of SDD. Rather the argument always is: Is there a need to exchange such data, do these data describe scientific information or management information that should be interoperable as metadata of the scientific process. Quite a number of DeltaAccess features are NOT or completely differently supported in SDD. The concern about complexity is correct, however. A call to everyone: Please do join the TDWD /SDD meeting in Lisbon, 21-26 October of this year, to help us finding a good compromise! (Please inform me by direct mail if you are likely to come) Gregor ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: SDD links not working
by Gregor Hagedorn 27 Aug '03

27 Aug '03

> most links at the site: http://160.45.63.11/Projects/TDWG-SDD/Docs/ > (except for the slide show). Please apologize, the index.html document in that folder was just an oversight, there should be no index there. The documents in the DOCS folder are available through links from the newer minute versions. Please use http://160.45.63.11/Projects/TDWG-SDD/ instead. Thanks, Gregor ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: SDD links not working
by Georgina MacKenzie 27 Aug '03

27 Aug '03

OK - the link ffrom the list of working subgroups goes directly to this link - so you are in full control!! Also, the link to the list archives works! On 27 Aug 2003 at 10:43, Gregor Hagedorn wrote: > > most links at the site: http://160.45.63.11/Projects/TDWG-SDD/Docs/ > > (except for the slide show). > > Please apologize, the index.html document in that folder was just an > oversight, there should be no index there. The documents in the DOCS > folder are available through links from the newer minute versions. > > Please use > > http://160.45.63.11/Projects/TDWG-SDD/ instead. > > Thanks, Gregor > ---------------------------------------------------------- > Gregor Hagedorn (G.Hagedorn(a)bba.de) > Institute for Plant Virology, Microbiology, and Biosafety > Federal Research Center for Agriculture and Forestry (BBA) > Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 > 14195 Berlin, Germany Fax: +49-30-8304-2203 > > Often wrong but never in doubt! ****************************************************************** This email is intended for the addressee(s) only. If you are not an intended recipient please inform the sender. BIOSIS UK accepts no responsibility for statements clearly those of the sender and not expressly made on behalf of the Company, or for damage caused by any transmitted viruses. ******************************************************************

1 0

SDD links not working
by Eric Goda 15 Aug '03

15 Aug '03

Yes, I did try to get some more information about SDD but most links at the site: http://160.45.63.11/Projects/TDWG-SDD/Docs/ did not work (except for the slide show). Eric

1 0

Re: Special states
by Eric Goda 15 Aug '03

15 Aug '03

On Wed, 26 Mar 2003 08:24:17 +1100, Steve Shattuck <Steve.Shattuck(a)CSIRO.AU> wrote: >The fundamental problem I see is making SDD truly extensible and yet >generic at the same time. The big problem with the DELTA standard is >that every time Mike needed new DELTA program functionality he extended >the DELTA standard by adding new directives. This meant that if you >wanted to support the DELTA standard you needed to follow Mike's lead. >It's hardly a "standard" if it's being pushed by one group. Hi there, I'm new to this group and was envolved with DELTA writing my TAXASOFT programs years ago. I agree that there was some mixture between the DELTA standard and special needs (directives) for the programs using it, which was a wrong choice (discussed these polutions with Mike years ago, to keep the chars, items and specs file free from this). The power of DELTA is its simple flat format and I agree that the new format must be as simple and transparant as possible. I do not know in what stage this SDD devellopment is, but I'm highly interested in a DELTA like format that is XML based (done a lot with XML lately) to provide hirarchical features in the items part. Eric Gouda

1 0

Re: Special states
by unknown＠example.com 26 Mar '03

26 Mar '03

Thanks to both Kevin and Gregor for the comments on my comments. Kevin is pretty much right that I see the main use of SDD as documenting essentially finished data rather than a "work in progress". My thinking is that most people will work on their dataset until it's finished, then make it available through SDD to other interested parties. My assumption is that most collaborative projects will select a single tool to use while building the dataset (be it DELTA, LucID, DeltaAccess or what ever) and will use the native format of this application to share data. Now, I have no problem with capturing details needed by specific situations or applications and would say that these are important and should be support by SDD. However, I would try to make them extensions to the basic, core data rather than part of the core itself. The fundamental problem I see is making SDD truly extensible and yet generic at the same time. The big problem with the DELTA standard is that every time Mike needed new DELTA program functionality he extended the DELTA standard by adding new directives. This meant that if you wanted to support the DELTA standard you needed to follow Mike's lead. It's hardly a "standard" if it's being pushed by one group. A specific example - "Use default state." We are looking hard at this in BioLink and will probably stay away from it, using different functionality to achieve the same thing (rapid data entry and global data changes). We obviously support this when importing from DELTA (and possibly from SDD) but it won't be core functionality for us and we won't export it. To enshrine this in SDD simply because DELTA had it (or some other application currently supports it) may not be the best approach (it may be, but I'm not willing to make that assumption unless we have to). The same seems to be true for describing uncertainty. Kevin suggests that there are only three "special states": It's logically possible to code and I intend to code it but haven't gotten around to it yet (unfinished business) It's logically possible to code and I intend not to do it (character scoped out) It's logically impossible to code (inapplicable) I only see two here: It's logically possible to code but I haven't done it It's logically impossible to code PLUS a reason: unfinished business character scoped out don't have specimens + a thousand other reasons. To overcome the "machine processing" problem you'll need to enumerate this list and that's fine. But don't build a structure that makes it hard to change that enumeration at any time or as any set of users sees fit. The problem with "special states" is that it overloads "states", which have nothing to do with the uncertainty of the data. I think this is a priority issue. It is FAR more important to know if this character has been scored for this taxon than to know the reason why it was or wasn't. If you want to then tell me the reason, great. But keep it simple and flexible. It also separates data from metadata along slightly different lines. I consider the state to be data, the reason for coding/not coding to be metadata and want to separate these as much as possible. Kevin's discussion of attaching uncertainly to states breaks Gregor's SDD model in more fundamentally than what I had proposed. As Gregor suggests, this would result in fundamental changes to the current SDD model. The use of "modifiers" might work, but again, they are fairly tightly associated with natural language representation (as I understand it) and may not generalize well to fulfil Kevin's needs. This leads to two thoughts: First, it is crucial for SDD to be the product of a team with differing views who work together towards some middle ground. Otherwise it is unlikely that it will meet the needs of the broader community. Second, I worry about SDD becoming too specialised and too complex. The "joy" of the DELTA standard is its simplicity. And this simplicity is, I would say, one of the reasons it has been so successful. I would contrast this with the Nexus format. Nexus is only seriously being used by the 3 or 4 applications involved in its original development. It is hardly a "standard" outside this small community. The power of Nexus is that it supports essentially all of the features of these original applications. This seems to be the way SDD is heading: support all of the features of character-based programs that currently exist. My concern would be that if we do this we will be the only people who use SDD, new players finding the format so complex and so specialised that it's expensive to implement and has little flexibility to support "extensions" outside of the strict standard. If this is true and is realised we all lose. Steve Steve Shattuck CSIRO Entomology Steve.shattuck(a)csiro.au

1 0

Re: Special states (2)
by Gregor Hagedorn 21 Mar '03

21 Mar '03

> Gregor writes: "All these special states apply to entire characters. > SDD currently provides no mechanism to express partial knowledge > within a character (e. g. to score one state as not applicable, but > another as present). Theoretically it is possible that a character is > only partly entered, but this is not likely in a well designed > terminology (it does happen where character clusters are combined into > pseudo-characters, but such a design is considered problematic)." > > Not so. One important difference between Lucid and DELTA is that DELTA > can only encode uncertainty for a whole character, Lucid only for a > character state. Often in Lucid we do encode uncertainty for an entire > character (for example, fruit colour is uncertain for a given taxon > because fruits are unknown) - we do this by (slightly messily) > encoding uncertainty for all the states of the character: > > Fruit colour (coding for taxon x) > red - UNCERTAIN > yellow - UNCERTAIN > pink - UNCERTAIN > white - UNCERTAIN > ....etc Regarding Uncertainty: I propose to use modifiers on states for this purpose (see the chapter in Special states: "Not supported as special state, but supported through modifiers" and the linked document on "Uncertainty modifiers"). That is, Uncertainty is NOT a special state as discussed in the paragraph quoted at the start. The question is: Is it necessary to code "unfinished work" or "not applicable" against states? This is not a trivial decision. Making Unknown/not applicable statements realtive to states means that any change in the state terminology requires a revision of all descriptions using that character. If the character is a true atomic entity, I believe it is easier to enforce that these statements are character level statements, not modifiers. I believe that the attempts to use "unfinished work" or "not applicable" usually show we are dealing with "pseudo-characters" like distribution. Does anybody have a good case with real descriptive, chemical or morphological data? Stuff like "enzyme tests" I really consider an assemblage of characters, which will be much less painful to handle appropriately as soon as character groups and hierarchy are present. Currently people like to keep these in one character, because that is the only hierarchy readily available for organizing the work. Gregor ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0