- tdwg-content - lists.tdwg.org

Trying it out
by P. Bryan Heidorn 10 Aug '00

10 Aug '00

My research group has finally been able to devote a couple of hours together discussing the DDST V1.1 as it relates to the published treatments of the Flora of North America and records that we are creating for the EcoWatch Project. First, we quickly dismissed trying to fit FNA into the spec since we ran into to many issues. We'll return to FNA another day but we did decide that the PrairieWatch Butterflies of Illinois for EcoWatch were much easier to deal with since we have full editorial control. So over the next two weeks we'll be modifying the DTD for that project and working up 20 or so example treatments (all at the species level.) We had some immediate questions and comments even before we convert one treatment. Treatment not equal to file: The second paragraph of the General Requirements of the DDST v1.1 states that "One file will comprise one treatment...". There is no reason to require in the specification that a treatment = a file. This is because there is good reason to synthesize XML treatments on the fly from databases and allow users to query the XML structure to extract just the elements that they need. Most of the proposed spec works fine it we say that "A treatment is a structured presentation of information. The DDST provides a uniform view of descriptive data for taxonomy but it does not specify the underlying storage and representation of the information." We'll examples of why I say this below. Taxon Name: We start right off with the Treatment Name. For the butterfly project at least, the treatments are all about species. Is it reasonable to give each treatment a name that equals the Taxon Name | Taxon ID? That is what we're doing for now. In general, should the DDST include treatment naming conventions linked to the Taxon ID? Of course you get into trouble if the Taxon Name changes but you would be OK if the ID were kept constant. So for now our treatments will have names like <Treatment Name>Papilio polyxene</Treatment Name> but I am not happy with the decision. Any better ideas out there? If so it would be helpful if those ideas were in the spec? In our case the treatment name will also be a file name for now but eventually parts will be in a database and we'll need a unique <Treatment ID>. Description: I think it is fine that there is a description element to the treatment but we were confused about what should be being described; the taxon or really the treatment. We decided that decided to take the spec at face value and really include the treatment which is fairly boring. All our treatment descriptions will be pretty much the same. <DESCRIPTION>This treatment was created as part of the Biological Information Browsing Environment project funded by NSF, UIUC... Information selected for the treatment is intended to support tasks associated with the PrairieWatch project....</DESCRIPTION> This uniformity makes me believe that many aspects in this description should be removed from the treatment and moved "up" one level to a higher order entity such as a collection of treatments or project. Several other elements of the treatment format support that notion. It means that DDST v1.1 is really a Treatment Spec. We also need a Collection Spec. This means that the treatments should all have an element called the Project ID. Treatment build/revision number Treatment build/revision number is a really good idea and we all agreed that we should have had it in our treatments before. <TREATMENT REVISION>0.1</TREATMENT REVISION> We use the 0 when the treatment is under development and not released yet. I do not know how to deal with dynamic treatment build revision numbers. Suggestions are welcome. Contributors List: Notice that if the information about collectors in the list such as contact information is repeated in each treatment there is a data integrity problem. People move. It might be possible that some treatments would have one set of old contact information and not be updated while some treatments might get updated information. It should be that when a person requests a treatment in DDST v1.1 format that they will get back a contributor list with ID, Name and Contact Details but in most cases the system providing the information would insert the contact information "on the fly". There does not seem to be a reason to force the contributor IDs to be numeric as stated in the spec. Alpha numeric can carry at least a little information to make the codes readable. The current standard makes a relatively weak standard that the contributor codes are unique to the treatment. I think we need to use a broader definition. The should be unique to a collection at least. That way it would be easy to find all treatments in a collection that a particular person was a contributor for. Better if they were unique for the world but that is more than this group can bite off I think. DDST should at least say that the Contributor ID is unique to a Collection. For us we have <CONTRIBUTOR LIST> <CONTRIBUTOR> <CONTRIBUTOR ID>ML1</CONTRIBUTOR ID> <CONTRIBUTOR NAME>Mary L.</CONTRIBUTOR NAME> <CONTACT DETAILS> <EMAIL>maryl(a)uiuc.edu</EMAIL> <MAILING ADDRESS> Address here </MAILING ADDRESS> </CONTACT DETAILS> </CONTRIBUTOR><CONTRIBUTOR> <CONTRIBUTOR ID>PBH1</CONTRIBUTOR ID> <CONTRIBUTOR NAME>P. Bryan Heidorn</CONTRIBUTOR NAME> <CONTACT DETAILS> <EMAIL>PHEIDORN(a)UIUC.EDU</EMAIL> <MAILING ADDRESS> Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign, 501 East Daniel St., Champaign, IL 61820-6212 </MAILING ADDRESS> </CONTACT DETAILS> </CONTRIBUTOR> <CONTRIBUTOR LIST> Attribution: Must this be a contributor? If so this information should be handles as a property of <CONTRIBUTOR ROLE=PRINCIPLE|COPRINCIPLE> or as another tag of <CONTRIBUTOR> .... <ROLE>PRINCIPLE</ROLE>.... Suggestions? List of Sources: Again, usually the underlying information might be kept in different files or databases but the view of a treatment would include the fields of the spec. For us the spec will include bibliographic references and "personal observations" Again is seems that source ID should at least be unique to an entire collection, not just to the treatment. <LIST OF SOURCES> <SOURCE> <SOURCE ID>KL2000</SOURCE ID> <DESCRIPTION>Koller and Smith (2000). Butterflies of the Great West. Biology Press: New York. </DESCRIPTION> </SOURCE> <SOURCE> <SOURCE ID>MJ2000</SOURCE ID> <DESCRIPTION>Mike Jacobs, personal communication August 10, 2000. </DESCRIPTION> </SOURCE> </SOURCE LIST> Actually in XML we don't need the <SOURCE LIST> but if is OK for now. Sorry, I'm way out of time. I'll have to get back to our other comments later. This is enough for one message in any case. Cheers, Bryan -- -------------------------------------------------------------------- P. Bryan Heidorn Graduate School of Library and Information Science pheidorn(a)uiuc.edu University of Illinois at Urbana-Champaign (V)217/ 244-7792 501 East Daniel St., Champaign, IL 61820-6212 (F)217/ 244-3302 http://alexia.lis.uiuc.edu/~heidorn

1 0

Re: characters/states and measurements and other hoary problems
by Stuart G. Poss 07 Aug '00

07 Aug '00

Kevin Thiele wrote: An item in the DDS is like an old OTU - it may be anything from a specimen > to a species to a Kingdom. ... > > The possibility of specimens being items comes from the ability to nest > treatments. > > e.g. Treatment A comprises measurements of a series of specimens of > Imaginaria magica for some characters (e.g. hypopode length). Treatment B > comprises as its items species of Imaginaria. Some characters (e.g. CAM > metabolism) are scored directly in B while others (e.g. the range of > hypopode lengths) are collated from A. > > Data for the character hypopode length in B would then be attributed to > treatment A, where each data element is attributed to measurements taken by > me on a specimen. Redetermination of a specimen followed by rebuilding of > the treatment tree would allow the redetermination to be incorporated all > the way up. > > Would this work, and does it address your query? This seem plausible to me. The only complication that comes to mind is the situation where one might have two inconsistent identifications of particular specimens. How would the redetermination rule work with respect to A if one has a Treatment C, say for hypopode width, for which the ID for (some?) of the same specimens in A is inconsistent with that used in treatement A? If the inconsistency is based on soley a difference of combination (perhaps old synonym incorrectly used) one could make the redetermination automatically (using the nomenclaturally appropriate combination). Otherwise, I'm not sure what flags get raised since it would depend on the nature of the inconsistency. Perhaps just throw the data back to the user with an error message and let them make a decision might be the appropriate response? This could certainly aid in identifying mistaken identifications, but in any event there probably needs to be some kind of default logic once inconsistency in the data is located. It will be difficult to eliminate entirely. > > > >Statement: > > > >One file will comprise one treatment, the basic unit of which is one or more > >characters describing one or more taxa or individuals. > > > Surely the data level to which a character is attributed is determined by > the context of the treatment: I would agree. I think its a question of to what extent we want the system to be able to make (part of) the inference for us so that we can be aware of the "universe" of features out there for which there are relevant data. When the data are initially recorded they are most likely defined within a much narrower context. I'm still struggling with how we can determine the extent to which the rules might be constructed to determine (identify/establish) context. > > Another collation rule may convert low-order numeric data into a > higher-order multistate: > Could be lots of options here. >

1 0

Re: characters/states and measurements and other hoary problems
by Kevin Thiele 07 Aug '00

07 Aug '00

At 12:52 PM 3/8/00 -0600, Stuart Poss wrote: >1 Attribution and sources for an item datum overides that for a character or >taxon, which override that for the treatment as a whole. Attribution for >characters and taxa are equivalent and additive. > >Don't we also need to say that item data attribution at the specimen level >may override that for a character or taxon as a whole? Otherwise, the system >has no way of dealing with misidentifications, particularly if some but not >all parts have been associated with the wrong ID, as might be frequently >encountered in fossils, or when dealing with taxa whose character state >definitions are later found applicable for only a specific size range (ie >differentially break down at small sizes). Specimen or parts level >data/attributions also would presumably be additive, even if potentially >contradictory (subject to differences of opinion), would they not? An item in the DDS is like an old OTU - it may be anything from a specimen to a species to a Kingdom. In footnote 1 I'm trying to address the relationships between attribution of the different possible sources of data, not between different levels of the hierarchy. That is, there may be an attribution for the treatment as a whole: e.g. Data for this treatment, unless otherwise attributed, are derived from the literature source Bloggs & Hogwash (1899) But data for a particular character (for all OTUs) may come from a different source: e.g. Data for character 9 (presence or absence of the CAM photosynthetic pathway) comes from the literature source Meg & Mog (1999) and the attribution Meg & Mog 1999 will override the default attribution. But data for a particular item (in this case a taxon that was not included in Meg & Mog 1999) for character 9 comes from pers. obs. The possibility of specimens being items comes from the ability to nest treatments. e.g. Treatment A comprises measurements of a series of specimens of Imaginaria magica for some characters (e.g. hypopode length). Treatment B comprises as its items species of Imaginaria. Some characters (e.g. CAM metabolism) are scored directly in B while others (e.g. the range of hypopode lengths) are collated from A. Data for the character hypopode length in B would then be attributed to treatment A, where each data element is attributed to measurements taken by me on a specimen. Redetermination of a specimen followed by rebuilding of the treatment tree would allow the redetermination to be incorporated all the way up. Would this work, and does it address your query? >Statement: > >One file will comprise one treatment, the basic unit of which is one or more >characters describing one or more taxa or individuals. > >If we are to presume nested levels of groupings of descriptor elements, then >we need to be able to clearly distinguish data values that are regarded as >referring to "individual units" (specimen, species, higher level taxon) at >one level, but are "collective" when evaluated at a different level. Not >only with the "collation rules" be different for different levels, but may be >different depending on whether a given feature (possibly "same feature but >defined differently") is regarded as a data item refering to a specific >"individual unit", or as a "collective unit". That is, the data item is a >representation of data that might apply to a collection, rather than a >measurable value that may have a scope no larger than a specific measure of a >specific specimen. Perhaps some treatments might include "collections of >collections" that would imply a mixing of both situations. > >Seems to me we need a means of distinguishing between collective >representations and "unit values" (for lack of a better word), if for no >other reason than to be able to track which data items apply to particular >"features of general interest" (eg. leaves, roots, head, foot, etc) and which >taxa are involved (higher-level taxon, species, individuals, parts of >individuals). Its not clear to me how the current DDST can be used to >associate "basic units" that might be reasonably differentially defined at >different "levels of composition" by different investigators. Perhaps we >need some general means to assign the "scope" over which the definitions of >"unit" apply. Surely the data level to which a character is attributed is determined by the context of the treatment: >we need to be able to clearly distinguish data values that are regarded as >referring to "individual units" (specimen, species, higher level taxon) at >one level, but are "collective" when evaluated at a different level. Not The attribution of data for a collated character in B to "raw" data in A provides the context: we know that the character in B is collated and hence higher-level data than in A. Same goes for "collations of collations" - the attribution allows you to track the data down to its source. Or am I misinterpreting you? Peter Stevens added: >the measurement(s) of either the specimen or the species could be assigned >to states; those states might encompass a range of variation such that the >attributuion of a specimen/species to that state would mean that, when >refering to the state, one loses sight of the original measurements. A collation rule may derive a higher-order numeric from numeric data: e.g. Treatment A stores hypopode lengths for specimens, Treatment B collates a range (and/or mean, median etc) of hypopode lengths for a species, Treatment C collates a higher-order range of lengths for a genus etc. Another collation rule may convert low-order numeric data into a higher-order multistate: e.g. Treatment A stores leaf length, leaf width and distance to widest point for specimens, Treatment B collates these into shapes for species, Treatment C collates the shapes again, this time direct from treatment A rather than bringing in the problems of collating a collation. (Note that sometimes collating a collation may be acceptable e.g. determining the full range of petal colours in a higher-level taxon, where the rules at all levels are simple additive) Thus, with both the collation rules and attribution at all levels fully explicit, one should be able to track any data element to its source. Any good, Peter? Cheers - k

1 0

Re: Where are we?
by Jim Croft 05 Aug '00

05 Aug '00

>We're discussing the need for a standard rather than actually creating one. Very deftly put Kevin... >I put up a while ago an initial attempt at a proposal (the document DDST >Specifications.doc, you all will have this buried in your attachments >folder somewhere). Some people have looked at this, I know, but I think we >need to decide: >1. Is this a start, or is it the wrong way to proceed and should be ditched >in favour of something better I think it is a good start... if you ditched it, you would only have to reinvent it, or something very like it... >2. If it's an adequate start, what concrete changes are needed to >incorporate the ideas people are discussing? It would be useful to migrate to definition to the XML arena so we can throw some data at it, dump into some XML parsers and see what the resulting mess looks like... >3. If it should be ditched, can anyone produce a better draft specification >that we can build on. ??? Is anyone out there going to XMLify it? >I think at this stage we need a draft document around which the discussion >can focus. Contributions that point out complications and problems can then >make concrete suggestions as to changes to the draft proposal, rather than >be mere discussion points. It seems to me we should be at this stage around >now - we've discussed the problems for close on 12 months now. yep... time to stop slicing bait and go after the fish... >Now it may be that the structure of this group is poorly suited for the >task, and a recommendation should go to TDWG that this isn't the way to >proceed. As it stands, it seems to me that we're functioning well as a >discussion group but not well as a working group (actually working on the >problem). This is undoubtedly partly because we're all busy - I know that >I'm doing all this in my "spare" time and I'm sure most of us are in the >same boat. Perhaps the only way that this can happen properly is for funds >to be sought to employ someone on a contract basis to turn the ideas raised >here into concrete output, using the list as an expert group/listening >post/scratch-pad resource or whatever. We need to be honest as to whether >we're going to achieve something. Experience with contractors/consultants is that they require a fairly tight specification of what they are being asked to build or you will end up with a pedal-powered space shuttle, or worse. We need to come up with the specs, and having done that there will probably not be much for the contractor to do. Nailing the specs to the wall should be something that this group can be able to do. We have all developed specs, schemas, models, etc. at some stage of our lives for our own enterprises. All we have to do is reach consensus on what this one might look like. Hey - shouldn't be much more difficult than herding cats or nailing jelly to a wall... :) And once we have the specs there may even be one or several among us who will actually build something with it. jim

1 0

Re: Structure vs content
by Jim Croft 05 Aug '00

05 Aug '00

>To some people, character definitions are part of the *structure* of >a data format, while to others these definitions are *content*. E.g., >character definitions as structure (attribution lost): Quite! I have always regarded this group as one dealing with structure and until we are able to nail down this distinction and put the content to one side for a short while, it is going to continue to be difficult to make clear progress on the underlying structural issues... jim

1 0

Re: Expression of continuing interest
by Bryan Heidorn 04 Aug '00

04 Aug '00

In my humble opinion, as many people have stated in this group, we need to put the actual content issues aside to establish the standard. In the end we only really care about sharing the content. The structure is just a means to enable sharing of content. The point that I was trying to make about "ovate" is not that the actual word "ovate" needs to be in the standard but that the descriptive mechanisms of the standard should allow data types such as "ovate" to exist as content and that the structure of the standard should be able to encode information about the content to allow people or computer programs to make decisions about the use of the content. The particular XML-like syntax is only as means of how that kind of information could be encoded in the structure. Maybe if I ground this in Kevin's document I'll have a little better luck trying to communicate. ------------------------------------- Character List {required unless an external lexicon resource has been specified above} [ Character Name3 Character ID Set membership {list of sets to which the character belongs; a character must be able to belong to more than one set} Attribution1 {reference to a contributor�s ID from the Contributors list} Source1 (reference to a source�s ID from the Sources list) Collated Character source {path name for another treatment that contains lower-level data for this character} Collation rule name {Name of a collation rule as defined in the Collation Rules list}2 Character type {ordered multistate, unordered multistate etc} Character dependencies (up) 4 Applies To list (or global/restricted type definition, then leave it to program to extract) 5 Character attachments [ attachment name attachment type attachment path/URL Public notes7 Private notes7 ] Private notes {internal notes for character}7 Character State List [ Character state name | Character state ID Character dependencies (down) Character state attachments . . . --------------- The authors of an Angiosperm treatment standard might want to use the document created by this group (BioSDD?) as a guideline for making their treatment definition. As part of their definition may wish to create a Character Name = "Leaf Shape", with a Character dependency of "Leaves". They will need to provide a Character State list for this Character. These states could take many forms. The BioSDD should provide guidance on this form. There should be a range of options of data types and processing to choose from (along of course with the options of using something not included in the BioSDD). I am arguing that the BioSDD should include data types and type qualifiers that are different from and addition to the "Character type" in Kevin's document "Character type {ordered multistate, unordered multistate etc} " So you might say that a particular character like "leaf shape" in the Angiosperm might have Character States that are assigned by "text", "collation rule" or "Fourier descriptor". The Angiosperm standard authors might further specify that vocabulary for character states of the text based state descriptions. They might include "ovate", "oblong" and "round" in that list. (This list would not be included in the BioSDD. The BioSDD would just say that if you have a unordered unistate text field that you can provide a list to the authors or leave the state values open to any text.) An author of a treatment then might decide by ungrounded opinion that the leaves of the Waga oni plant are "ovate". The author would record "ovate" and that it is of type "text" so that not computer program will read it and try to treat it like a Fourier descriptor. I am sorry that I am having difficulty making this clear. I might be way off base. There seem to be two issues related to this problem. Should we allow a standard that flags these kinds of data and how can we represent this type of functionality in the structure of the BioSDD guidelines so that Angiosperm standard authors can easily use them. now I need to get some work done. Bryan

1 0

Structure vs content
by Una Smith 04 Aug '00

04 Aug '00

I have just finished reading all the traffic on TDWG-SSD after 2 weeks away, and I noticed something interesting. There is an important difference of perspective here that has not so far been made explicit: To some people, character definitions are part of the *structure* of a data format, while to others these definitions are *content*. E.g., character definitions as structure (attribution lost): >> [...] So we need to support the word "ovate" as a leaf shape >>descriptor <<leaf_shape>ovate<</leaf_shape> or we will not be able to >>support legacy data. ...and as content (Peter Stevens): >Of course we have to be able to <underline>handle</underline> such >data, it will simply be a case of caveat emptor. Count me among those who regard character (and taxon) definitions as content. Una Smith una.smith(a)yale.edu Department of Ecology and Evolutionary Biology Yale University New Haven, CT 06520-8106

1 0

Re: Expression of continuing interest
by P. F. Stevens 04 Aug '00

04 Aug '00

>I also mean that not all character states can be grounded in reality. I agree >with Peter that they _should_ but there is a lot of legacy data out there and >we don;t have the collation rules that were used by the authors to come up >with >the generalities. So we need to support the word "ovate" as a leaf shape >descriptor <<leaf_shape>ovate<</leaf_shape> or we will not be able to support >legacy data. Even if we do not really know what they meant by "ovate." > Of course we have to be able to <underline>handle</underline> such data, it will simply be a case of caveat emptor. P.

1 0

Re: Expression of continuing interest
by Bryan Heidorn 03 Aug '00

03 Aug '00

Yes. Actually I mean two things. The simple is that there is some data that just is text and not processable but should still be in a treatment. For example, Natural History comments or social factors like, "Native Americans used this tree to construct canoes because they could." or some other more useful comment. I also mean that not all character states can be grounded in reality. I agree with Peter that they _should_ but there is a lot of legacy data out there and we don;t have the collation rules that were used by the authors to come up with the generalities. So we need to support the word "ovate" as a leaf shape descriptor <leaf_shape>ovate</leaf_shape> or we will not be able to support legacy data. Even if we do not really know what they meant by "ovate." I think this can be covered in your standard document and in XML since we can define a property for the tag to say how we got it. <leaf_shape type=text> or for "grounded data" <leaf_shape type=collationRule|where-to-find-rule> or if the leaf shape was for a specimen instance and described with an ellipsoid Fourier descriptor <leaf_shape type=EFD>e1:0001;e2:0212;e3:0981</leaf_shape> An application can define the semantics based on the type property (meta-data) conversely we could make different leaf_shape tags for the different types but I think that is a bad idea. cheers, bryan At 08:28 AM 7/20/00 +1000, you wrote: >At 06:58 PM 18/7/00 -0500, Bryan Heidorn wrote: > >>I do not see how we can avoid mixing data in a treatment. I'll need to go back >>and reread Bob's comments on that. Generally I don't see how we can have a >>standard that requires all fields to be reduced to finite states. Some things >>will just need to be unprocessed (by the computer) text. > >You mean like the "Text" character type of DELTA?, or unparsed textual >notes attached to characters/states/taxa/data items? > >Cheers - k > -- -------------------------------------------------------------------- P. Bryan Heidorn Graduate School of Library and Information Science pheidorn(a)uiuc.edu University of Illinois at Urbana-Champaign (V)217/ 244-7792 501 East Daniel St., Champaign, IL 61820-6212 (F)217/ 244-3302 http://alexia.lis.uiuc.edu/~heidorn

1 0

Re: characters/states and measurements and other hoary problems
by Kevin Thiele 03 Aug '00

03 Aug '00

Dear List'eners attached find DDST Specifications.htm. See if this works. Cheers - k

1 0