December 2001 - tdwg-content

Re: Questions on Dependencies and identifiers
by Jim Croft 06 Dec '01

06 Dec '01

Kerry wrote: > First, I am stuck trying to adequately model the structure and dependencies > of the leaf apex characters > "apex obtuse or minutely mucronate within an apical notch". I am parsing > this out as: > Apex shape > obtuse > notched > Apex projection > immucronate > mucronate within notch (dependent on 'Apex shape - notched') > The model has an attribute for 'dependent on' under the character element. > Does the proper solution of this problem require a 'dependent on' attribute > as part of the //character/state element as well? Am I parsing the data > correctly? Maybe... but is there a correct answer to this in the absence of context? Without knowing what the other taxa in the application are, you could get away with: Apex shape obtuse minutely mucronate in an apical notch Or even, trivially: Apex shape obtuse or minutely mucronate in an apical notch It maybe that there are only two character states in this group of taxa - the bluntish ones with obtuse/mucronate tips and the sharp ones with acute/attenuate tips. In this case building a line of dependency might not only be unnecessary, but might it also be misleading? But I do not this that was the real issue, was it? The challenge was, if you had a dependency, how would you deal with it... > The first problem led me to the second problem, which is: how do we > identify the character or state another character depends on? This is not a > problem when working with DELTA format data which has unique identifiers > already, but text descriptions do not. Is an Xpath description adequate or > do we need to be able to have a structure that allows us to uniquely > identify what a character state describes? Is there any harm in requiring a unique id for each node? Most descritpive applications probably do this already to manage their data matrices. DELTA lets you see them which is considered mabye useful, Lucid doesn't which is considered healthy... :) > I originally thought a unique identifier was needed to identify > missing data not explicitly encoded in the description and to provide a > pointer to characters or states that are other characters of states depend > on. Now that I see how complex this may be to implement, I'm wondering if > another way would be better. Is our structure/Schema/DTD is right, it would be nice to think that it was self referenceing and that we did not have to worry about internal ids and pointers, wouldn't it... Thinking further about Kevin's two types of dependency, context and content (or whatever the terminology was), isn't the former an artifact of the character set we have decided to use and the latter an artifact of the taxa we are covering? The former is just not possible, the latter might be possible but is just not there. If this really is the case and one sort of dependency arises from the character state and another arises from the taxa being considered, then it is very likely that they should be modelled in slightly or completely different ways. > I originally included an attribute called 'describes' under the > //character/state element. I thought that this could be used for two > purposes: (1) to add descriptive text to the character state and (2) to help > build a unique identifier for a character state. The first use is no > problem; this structure allows an author to easily add "high' to the value > for the character state describing height for example. The structure is > less useful for building an identifier for what the character state > describes. //character/state is not always going to be unique is it? The tip mucronate, obtuse, acute condition could exist in scales, bracts, leaves, petals, sepals, anthers, etc. A more complete path is going to have to be specified which could get pretty nasty... but hey, computers are good at nasty... > I thought I could build a unique identifier by combining the > character name and the 'describes' attribute. So for > > <Character> > <CharName>Leaf</Charname> > <State describes="PlaneShape" modifier="+/-">oblong</State> > </Character> > > The identifier 'LeafPlaneShape' identifies what the state describes. This > allows you to recognize that even when there are multiple states (ie. > 'ovate', 'round', etc.) they are all describing one aspect of the leaf. If > nested characters are recognized, the identifier must get longer. > 'ApexShape' or 'ApexProjection' wouldn't work, it would have to be > 'LeafApexShape' or 'LeafApexProjection'. If the nesting gets deep, this > gets awkward pretty fast. It would work, though. but not very elegantly... Dependency is the act of a state/value controlling the applicability or otherwise of a character/feature or as Kevin has pointed out a taxon controlling a character feature (in the identification process this get s bit circular as you have to hav an idea what a taxon is before you can decide if a character is inapplicable, but such knowledge can be built into the data in various ways). Thus the model will have to accommodate that a certain character/feature is unavailable if a certain state/value exists, perhaps in a totally different branch of the character/feature tree - and the obligate reciprocal relationship between controlling state/values and controlled characters/feataures. Unique ids, even arbitrary sequential numeric ones a la DELTA, might be the best (or an adequate) way to do this. How about: <feature id="123"> <feateName>leaf</featureName> <character id="6" nullcharacter="3" nullstate="4"> <characterName>leaf shape</characterName> <state modifier="+/-" value="present">oblong</State> <state value="rarely">ovate</State> <state value="rarely">obovate</State> </character> </feature> ... just doodling... maybe this needs to be specified in a separate characterlist up front, rather than in the guts of the descriptive data. Actually we have not even decided that yet have we? Should the list of characters/states and the lists of taxa be described as separate blocks, or should they be implied from the content and structure of the standard data file. I prefer the former and both DELTA and Lucid do this... but have we decided that this is the way to go? > Do others think that having an identifier is necessary and, if so, > has anyone been able to come up with a better way to handle it? Is Xpath > adequate to our needs? I always have problems with Xpath - it is supposed to work, but I always manage to get lost in the hierarchy... need more practice... :) jim

1 0

Questions on Dependencies and identifiers
by Barringer, Kerry 05 Dec '01

05 Dec '01

I am still working through Challenge 1, modifiying my data model so that it can completely model the text descriptions. I am now stuck on two problems that arose when I was trying to work through the practical implications of the model. First, I am stuck trying to adequately model the structure and dependencies of the leaf apex characters "apex obtuse or minutely mucronate within an apical notch". I am parsing this out as: Apex shape obtuse notched Apex projection immucronate mucronate within notch (dependent on 'Apex shape - notched') The model has an attribute for 'dependent on' under the character element. Does the proper solution of this problem require a 'dependent on' attribute as part of the //character/state element as well? Am I parsing the data correctly? The first problem led me to the second problem, which is: how do we identify the character or state another character depends on? This is not a problem when working with DELTA format data which has unique identifiers already, but text descriptions do not. Is an Xpath description adequate or do we need to be able to have a structure that allows us to uniquely identify what a character state describes? I originally thought a unique identifier was needed to identify missing data not explicitly encoded in the description and to provide a pointer to characters or states that are other characters of states depend on. Now that I see how complex this may be to implement, I'm wondering if another way would be better. I originally included an attribute called 'describes' under the //character/state element. I thought that this could be used for two purposes: (1) to add descriptive text to the character state and (2) to help build a unique identifier for a character state. The first use is no problem; this structure allows an author to easily add "high' to the value for the character state describing height for example. The structure is less useful for building an identifier for what the character state describes. I thought I could build a unique identifier by combining the character name and the 'describes' attribute. So for <Character> <CharName>Leaf</Charname> <State describes="PlaneShape" modifier="+/-">oblong</State> </Character> The identifier 'LeafPlaneShape' identifies what the state describes. This allows you to recognize that even when there are multiple states (ie. 'ovate', 'round', etc.) they are all describing one aspect of the leaf. If nested characters are recognized, the identifier must get longer. 'ApexShape' or 'ApexProjection' wouldn't work, it would have to be 'LeafApexShape' or 'LeafApexProjection'. If the nesting gets deep, this gets awkward pretty fast. It would work, though. Do others think that having an identifier is necessary and, if so, has anyone been able to come up with a better way to handle it? Is Xpath adequate to our needs? Thanks, Kerry ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Kerry Barringer (Curator of the Herbarium) Herbarium 718-623-7318 (office) Brooklyn Botanic Garden 718-941-4774 (fax) 1000 Washington Avenue 718-623-7312 (herbarium) Brooklyn, NY 11225-1099 U.S.A. kbarringer(a)bbg.org http://www.bbg.org/ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

Una rote: > These <freeform comments> are natural-language paragraphs embedded > in otherwise fully parsable data, which is the opposite of Kevin's > case above. And much easier to handle, I think. That is an intersting perspecitve, and probably right... but although coming from different directions, is there a real difference in the final result? Both end up as mixed content of structured (and parsable character data interspersed with free form text... > I put interesting data in the comments for an extremely important > reason: I know I don't have enough data to represent the probable > diversity in the taxon (the item), and I do not want to understate > the diversity. Better to leave the character coded as unknown, > but record the data. At some point in the future, the data can be > moved from the comments to where it belongs. Absolutely... and isn't this the holy grail we are seeking in this descriptive data standard? I am quite sure that the model we come up with with have a structured place for this freeform and comment stuff... to do anything less will be to admit failure... jim

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

In respnse to Steve Kevin wrote: > The terminology is +/- trivial at this stage, but I'll explain that I chose > something different from character/state simply to break with tradition for > a while. Traditionally, a character has states and that's it - a 2-level > tree. In the example above one character (leaf) has as child another > character (margin). This seems odd to many people thinking traditionally > about characters/states. Let's agree that we'll use them interchangeably for > now. And I think we have all become bilingual in this regard... But sooner rather than later I would like us to nail this terminology down, to free up the synonyms forus elsewhere in our model as much as anything else... Also, I am not yet convinced that unbounded nesting of characters is necessary the best way to go in terms of representing a hioerarch of character data... but maybe it is... At the end of the day, a state of a particular character (or a value of a feature) is used in a key decision or choice and the hierachy is not all that important other than the order of presentation orlogical grouping of characters. In the description the hierarch similar provides the logical order of the characters and their states/values. Perhaps we could use features for the hierachy and characters for the ultimate branch. For example features could contain features or characters (but not both) and characters would have states which would have values (present, absent, doubtful, rarely, in error, or whatever, or a measurement/count). Reaching agreement of this level of data description, and the terminology we are going to use would seem to be essential for clear and unambiguous communication within the group. But if we can't agree, I can wait... because we aregoing to have to doit sooner or later... jim

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

> No, this is not equivalent to dependencies, although at first sight it seems > similar. It's a tree-topology rule: the rule "<states>s can have > <characters>s as siblings" allows certain topologies for the > data-representation trees that would be preluded by the rule "<states>s > cannot have <characters>s as siblings". As I said, I don't know whether the > rule's a good or necessary one or not. But it is important that we sort it out... Dependency can be expressed in ways other than allowing the ultimate state to be a parent of a character, can't it? This would seem to be very messy and recursive and would probably break the clean hierarchical representation of data we seemed to be aiming towards. > Dependencies establish relationship rules between different parts of the > tree. Sometimes these rules affect parent-sibling relationships as in the > example, but sometimes not - that is, dependencies are more complex than can > be handled by a topology rule. I tend to agree with this... > In the key, if a user chooses a state marked * the character "Trap > Structures" will disappear because of the dependencies. Note that the > dependency on tree and shrub is not a logical one but a contextual one - it > just happens that no trees or shrubs are carnivorous, so the dependency's > useful but not logically necessary. Well, this is important, isn't it? I think we have just established that our model must accomodate two kinds of dependency one based on logic andaracter tree topology, an the other based on content/context. In the former a character can not be present because to parent feature/character/organ is absent; in the latter it could conceivalbly be present, but just isn't. Up till now applications have just allowed scoring dependencies without making this distinction. > I don't think these dependency relationships can be expressed merely by the > topology of the tree, as you suggest. We will need to deal with dependency > rules as a separate issue. I agree with this. We need to bear it in mind, but let's try and nail down a basic descriptive framework first... jim

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 04 Dec '01

04 Dec '01

Kevin wrote: > I seem to be in a minority of one here again, but I'll continue to argue my > case for a bit longer. why not... whe have be doing this for several years now, a few more email iterations won't hurt... :) > 1. Exactly, it's easier to go from data to +/- natural language, which is > precisely why we need to try hard to facilitate the reverse. Which is not all that difficult... to mark up a single description manually is a relatively trivial task... but to process a lot automatically is another matter... and to do it in a way that the whole community accepts is another matter entirely... :) > 2. If we can effectively embed fully parsable data in a natural-language > paragraph, why not? becasue it creates mixed content (legal, but evil) where the data is parsable, but not structured. If we can not represent our data universe in a database in an elegant and easily understood way we may not necessarily have failed, but we will have fallen short of the target by a large margin. In fact, DELTA already does this sort of thing by allowing liberal appending and prepending <freeform comments> all over the place. While this makes for quasireadable descriptions, authors often embed interesting character data in the comments making it unusable by other applications, even within the DELTA suite. The 'rarely' and 'misinterpreted' scoring options in Lucid might be seen as an attempt to capture some of this information. > 3. If a structured data document based on our standard is a subset of a > marked-up description based on the standard, then creating a standard that > can support the latter gives us the best of both worlds. If it can be done, > why not? I agree... this is a laudable aim... but I must admit to never thinking of a structured document as a subset of a marked up description - oftentimes the reverse may be the case... Isn't it better to think of both structured documents and marked up descriptions as being subsets of the standard we are trying to create? > Personally I think that creating an XML representation of structured data > would be a doddle. But as we have seen, getting everyone to agree that one person's way of doing it is the one true and proper path to descriptive enlightenment is no easy task... > Creating a fully parsable but lossless XML representation > of a natural language description (which hence can also handle the degraded > case of structured data) - now that would really be something to write home > about! Well, dreaming about it at least... I think we are dealing with two different, and I fear irreconsilable things here... descriptions by their very nature are lossy - they are abstract representations of the gestalt of a sample of a taxon, often with an arbitrary word limit, attempting to portray in a familiar format what an author thinks a taxon looks like. Structured documents such as DELTA and Lucid at least have to potential to store everything that is remotely interesting about every taxon in the set and often come close to achieving it in reality. So what if the resulting descriptions do not have the poetic beauty of a Shakespearian sonnet; at least the information will be there and retrievable... In the case of biological description, beauty is not necessarily truth, or at least the whloe truth... > Anyone else out there +/- agree with me, or should I give up now? Don't do that... if you do, you will never have anything to write home about... jim

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

I seem to be in a minority of one here again, but I'll continue to argue my case for a bit longer. ----- Original Message ----- From: "Robert A. (Bob) Morris" <ram(a)CS.UMB.EDU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 2:11 PM Subject: It's How the Data will be Used that Counts | I think natural language parsing understanding is harder than natural | language production from structure. So I think there is less work to | go from data to description than the other way around. 1. Exactly, it's easier to go from data to +/- natural language, which is precisely why we need to try hard to facilitate the reverse. 2. If we can effectively embed fully parsable data in a natural-language paragraph, why not? 3. If a structured data document based on our standard is a subset of a marked-up description based on the standard, then creating a standard that can support the latter gives us the best of both worlds. If it can be done, why not? Personally I think that creating an XML representation of structured data would be a doddle. Creating a fully parsable but lossless XML representation of a natural language description (which hence can also handle the degraded case of structured data) - now that would really be something to write home about! Anyone else out there +/- agree with me, or should I give up now? Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

----- Original Message ----- From: "Steve Shattuck" <Steve.Shattuck(a)CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 11:31 AM Subject: It's How the Data will be Used that Counts | Kevin's representation is too focused on text descriptions. A more complete | representation might be: | | <character name="leaf"> | <state>present</state> | <character name="leaf margin"> | <state>serrate</state> | <character name = "tooth orientation"> | <state>forward-pointing</state> | </character> | </character> | </character> I would argue that this is a less complete representation, because you've abstracted the original data further than I have. I won't use the "markup" from my last email for "Leaf margins serrate with forward-pointing teeth", as this was designed to exemplify one limited problem (how to expand a description if characters are nested) and wasn't actually marked up in the way I proposed for challenge case 1. Two possible ways I'd do this example are: "Leaf margins serrate with forward-pointing teeth" {Using the rule "<states>s cannot have <characters>s as siblings"} <Feature><Name>Leaf</Name> <Feature name = "Presence" value = "Present"> <Feature Name="marginal toothing">margins <Value>serrate</Value> </Feature> with <Feature Name = "tooth orientation"> <State>forward-pointing</State> teeth </Feature> </Feature> {Relaxing the rule so that <states>s can have <characters>s as siblings} <Feature><Name>Leaf</Name> <Feature name = "Presence" value = "Present"> <Feature Name="marginal toothing">margins <Value>serrate</Value> with <Feature Name = "tooth orientation"> <State>forward-pointing</State> teeth </Feature> </Feature> </Feature> Your proposal is: <character name="leaf"> <state>present</state> <character name="leaf margin"> <state>serrate</state> <character name = "tooth orientation"> <state>forward-pointing</state> </character> </character> </character> The score: 1. Can we parse from these the data atoms "leaf = present "leaf margin = serrate" and "tooth orientation = forward-pointing"? Kevin's = Yes Steve's = Yes Can we easily retrieve from these the original natural language string? Kevin's = Yes Steve's = No On this scoring I'm one up. Then again, yours would be slightly easier to parse than mine, so we're probably equal. What's most important here? Dunno. Further, it seems to me that yours is a subset of mine: a Schema that allowed mine would also allow yours, but not vice versa. | In my original, DELTA-centric model I used a <description> tag to try and | capture the text description information separate from the <state> | information. My thinking was that these two | requirements/approaches/viewpoints are too distinct to cram together without | falling into the same trap as the current DELTA Standard (which is a | least-common denominator approach). Yes we could tag the bits of free-form text. But is there any need? They will (by definition) be ignored by all processors except for natural-language - since the NL is fully retrievable from my model, why not leave them untagged? In your model, they would need to be tagged since the model does not represent a natural description - it represents abstracted data from which a description can be +/- created. | The problem here is that the phrase "Leaf margins serrate with | forward-pointing teeth" concerns 3 characters (leaf, margin and teeth) and 1 | implied and 2 expressed states (present, serrate and forward pointing) with | the characters being dependent (and therefore the context containing | significant information - we know that 'teeth' have something to do with | 'serrate' which has something to do with 'leaf margins' - the leaves being | present because we're describing them). There's a lot of logic involved in | parsing this. I can't think of a simple way of representing all this | complex information without separating it at some level. Kevin's suggestion | represents the text description and mine the underlying data, but neither | works well for the other. I agree - there's still too much complex logic even in the very simple types of examples we're using so far. We need somehow to step back further to even more basic examples to tease these issues out. Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

----- Original Message ----- From: "Steve Shattuck" <Steve.Shattuck(a)CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 11:31 AM Subject: It's How the Data will be Used that Counts | A couple of points: | | Kevin suggests a rule "<states>s cannot have <characters>s as siblings". | But this is what DELTA calls Dependencies, it represents the state that | controls a character. This would seem to be a good thing (and may be very | important). No, this is not equivalent to dependencies, although at first sight it seems similar. It's a tree-topology rule: the rule "<states>s can have <characters>s as siblings" allows certain topologies for the data-representation trees that would be preluded by the rule "<states>s cannot have <characters>s as siblings". As I said, I don't know whether the rule's a good or necessary one or not. Dependencies establish relationship rules between different parts of the tree. Sometimes these rules affect parent-sibling relationships as in the example, but sometimes not - that is, dependencies are more complex than can be handled by a topology rule. In dependencies, there may be several controlling states for a character, and these may or may not occur on nodes antecedent to the node in question. An example (from a key of mine): the character: Trap structures (on carnivorous plants) submerged or underground bladder-traps pitcher-traps sticky hairs irritable leaf-blade segments has controlling states in several other characters, viz: Nutritional strategy carnivorous parasitic* neither carnivorous nor parasitic* Habit tree* shrub* terrestrial herb aquatic herb In the key, if a user chooses a state marked * the character "Trap Structures" will disappear because of the dependencies. Note that the dependency on tree and shrub is not a logical one but a contextual one - it just happens that no trees or shrubs are carnivorous, so the dependency's useful but not logically necessary. I don't think these dependency relationships can be expressed merely by the topology of the tree, as you suggest. We will need to deal with dependency rules as a separate issue. Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

Thanks Steve - at last we have some alternatives we can sink our teeth into. Comments below. ----- Original Message ----- From: "Steve Shattuck" <Steve.Shattuck(a)CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 11:31 AM Subject: It's How the Data will be Used that Counts | I've been giving Kevin's approach some thought and have the following | comments: | | Kevin's original information flow model is too simplistic. A more realistic | model would be something like: | | Text Descriptions | | Text Descriptions | Phylogenetics |---> Structured Data --->| Phylogenetics | Specimens | | Identification Tools | (many) Yes, of course. There are varied sources for the structured data. It still seems to me that capturing the non-text sources will probably be a subset of what's needed to capture the text sources. This is because textual descriptions are probably the least formally structured data we need to deal with as input (with the exception of original observations which, in some taxonomists' minds at least, are highly structureless but are readily structurable) | The sources are much more varied and are often group-specific. For example, | invertebrates have very few good quality text descriptions (most are old, | are in a range of languages (English, French, German, etc), vary greatly in | style, quality, etc. etc) and the majority of invertebrates are currently | undescribed (having 80% new taxa during a revision is common). Yes I agree, a botany bias is showing through here. | Similarly, the outputs required vary greatly and in ways hard to predict. | While text descriptions would seem to be a common requirement, they are in | some ways "legacy" and may become less important in the future as | applications (and users) become more sophisticated. We need to make sure we | keep this range of uses in mind at all times. Yes, but see comment above. | Because of this I don't really think the details of the model | matter too much, more that it is rich enough to represent all data of | interest. Exactly my point - it needs to be rich enough to capture and express a textual description, hopefully losslessly! | ******** | | I've also been thinking about Kevin's latest example: | | "Leaf margins serrate with forward-pointing teeth" | <feature name="leaf"> | <feature name="margin"> | <feature name = "teething shape"> | <value>serrate</value> | </feature> | <feature name = "teeth orientation">with | <value>forward-pointing</value>teeth | </feature> | </feature> | </feature> | | First, it seems to me that "feature" is what taxonomists call "character" | and "value" is "state". Being a traditionalist I'll switch back to this | common terminology: The terminology is +/- trivial at this stage, but I'll explain that I chose something different from character/state simply to break with tradition for a while. Traditionally, a character has states and that's it - a 2-level tree. In the example above one character (leaf) has as child another character (margin). This seems odd to many people thinking traditionally about characters/states. Let's agree that we'll use them interchangeably for now. (Other points have been split into separate emails) Cheers - k

1 0