December 2001 - tdwg-content

Workshops in Spring
by Gregor Hagedorn 21 Dec '01

21 Dec '01

Dear collegues as has been announced at the TDWG meeting in Sydney we are planning 2 workshops in March 2002 to discuss the topic that remain open during the email discussions. Karen Wilson and Nicolas Bailly have informed me that the dates of the other meetings are fixed. For the Workshop in Sydney the best dates seem to be the 13-14 March 2002. Please inform me if you think that we should try to find another date that fits into the BioForum 2 schedule (see http://plantnet.rbgsyd.gov.au/bioforum/index.html) For Paris, we may have a choice between the 6. and 10. of March. Please send me a direct mail (g.hagedorn(a)bba.de) if you are considering attending this extra 1 day workshop in Paris, and write which day you would prefer. I wish all of you a very nice Christmas celebration and holiday, Best wishes Gregor Hagedorn ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: HTML in XML
by Jim Croft 21 Dec '01

21 Dec '01

Leigh wrote: > Except for this kind of usage. Mixed content is perfect for marking up > free-form text. Bad for mapping data into data structures (in most > cases a database). > If I had XML data containing this structure, I'd simply store the contents > as the raw XML. This is how I would like to do business as well, but unfortunately once you do this in common databases, they lose access to all the useful structural stuff in there... One day 'xml' might be a common database content type, up there with text, integers, real numbers, dates and the like, with real database facilities to manage and use this content... maybe it is already there... somewhere... jim

1 0

Re: HTML in XML
by Jim Croft 20 Dec '01

20 Dec '01

Tim wrote: ><species_comments>The nature of Carcinus maenas is blah blah >blah.</species_comments> This is horrible mixed content stuff, and I am told that this is ipso facto, evil, and we will burn in hell for eternity for considering such a construct... >What I would like to do is have this attribute parsed using XSL and retain >the formatting over species names. Wouldn't we all... >When the text is processed normally the >HTML tags are dropped. If it is kept as CDATA the tags are not dropped but >also not processed. Any clues how to handle this? In one test application where plant names embedded in text were surrounded by <taxon></taxon> (more meaningful than ), I fudged it using fragments like the following xsl: <xsl:for-each select="note"> <xsl:apply-templates/> </xsl:for-each> .... <xsl:template match="taxon"> <xsl:apply-templates/> </xsl:template> which spewed the output to browser compatible html. The result was a series of note paragraphs with italicised species names... Not very elegant, pretty trivial, and I am not particularly proud of it. But hey, it worked and current IE and NS browsers knew what to do with it... jim ~ Jim Croft ~ jrc(a)anbg.gov.au ~ 02-62465500 ~ www.anbg.gov.au/jrc/ ~

1 0

HTML in XML
by Tim Jones 20 Dec '01

20 Dec '01

Hi all you XML gurus, Has anyone had any experience storing HTML markup within an XML document. I would like to have a document that describes a species and one of the attributes is a descriptive piece of text eg. <species_comments>The nature of Carcinus maenas is blah blah blah.</species_comments> What I would like to do is have this attribute parsed using XSL and retain the formatting over species names. When the text is processed normally the HTML tags are dropped. If it is kept as CDATA the tags are not dropped but also not processed. Any clues how to handle this? Cheers Tim --------------------------------------------------------- Database Manager Centre for Research on Introduced Marine Pests GPO Box 1538, Hobart Tas 7000, Australia. Phone : (03) 62325222 (switch), (03) 62325213 (direct) Mobile: 0411 560057 Fax : (03) 62325485 E-mail: tim.jones(a)marine.csiro.au ---------------------------------------------------------

1 0

Re: HTML in XML
by Leigh Dodds 20 Dec '01

20 Dec '01

> Tim wrote: > ><species_comments>The nature of Carcinus maenas is blah blah > >blah.</species_comments> > > This is horrible mixed content stuff, and I am told that this is ipso > facto, evil, and we will burn in hell for eternity for considering such a > construct... Except for this kind of usage. Mixed content is perfect for marking up free-form text. Bad for mapping data into data structures (in most cases a database). If I had XML data containing this structure, I'd simply store the contents as the raw XML. L.

1 0

Re: HTML in XML
by Leigh Dodds 20 Dec '01

20 Dec '01

> <species_comments>The nature of Carcinus maenas is blah blah > blah.</species_comments> > > What I would like to do is have this attribute parsed using XSL and retain > the formatting over species names. When the text is processed normally the > HTML tags are dropped. That's because you're hitting the default template rules, which strips elements but retains their content. > If it is kept as CDATA the tags are not dropped but > also not processed. If you mean you've got Carcinus maenas or something similar, then the XSLT processor won't treat them as tags, but text. >Any clues how to handle this? You need a template rule that copies over the contents of this element into the result tree unchanged. something like: <xsl:template match="species_comments"> <xsl:copy> <xsl:copy-of select="@*"/> <xsl:apply-templates/> </xsl:copy> </xsl:template> This should copy the contents unchanged (including attributes) into the output. HtH, Cheers, L.

1 0

Re: Character hierarchies
by Robert A. (Bob) Morris 12 Dec '01

12 Dec '01

Jim Croft writes: > Date: Sat, 1 Dec 2001 07:28:21 +1100 > From: Jim Croft <jrc(a)anbg.gov.au> > To: TDWG-SDD(a)usobi.org > Subject: Re: Character hierarchies > > >The other qualifiers (misinterps etc) aren't in there yet, but will come. > >We'll have to be much more sophisticated that in my example, of course. > > > >Now how about "definately known to be unknown" > > Very droll... However, Greg and I have discussed this in depth over a ^^^^^^^ > bottle of red and have come to the firm conclusion that without the ^^^^^^^^^^^^^ over "a" bottle of wine, or over "a quantity of bottles of red wine wine more than zero and of indeterminate number" ? > attribute of "often misinterpreted as definitely known to be unknown", and > the corresponding "rarely misinterpreted as definitely known to be > unknown", accurate representation of author intent in biological > descriptions will never be possible... > > So get to it - we expect to see this in the next software release, real > soon now... > > jim > > ~ Jim Croft ~ jrc(a)anbg.gov.au ~ 02-62465500 ~ www.anbg.gov.au/jrc/ ~ >

1 0

Working Document
by Georgina MacKenzie 11 Dec '01

11 Dec '01

Kevin Thiele's working document for the SDD group is now available from the TDWG web site at: http://www.tdwg.org/tdwg2001/SDD_TDWG_2001.htm Georgina MacKenzie, Secretary, Newsletter Editor and Webmaster, International Working Group on Taxonomic Databases (TDWG) gmackenzie(a)york.biosis.org Telephone: +44 1904 642816 Fax: +44 1904 612793 54 Micklegate, York, YO1 6WF, UK ********************************************************************* This email is intended for the addressee(s) only. If you are not an intended recipient please inform the sender. BIOSIS UK does not accept responsibility for the contents of this message. *********************************************************************

1 0

Re: Questions on Dependencies and identifiers
by Kevin Thiele 06 Dec '01

06 Dec '01

Jim wrote | Thinking further about Kevin's two types of dependency, context and | content (or whatever the terminology was), isn't the former an artifact | of the character set we have decided to use and the latter an artifact | of the taxa we are covering? The former is just not possible, the latter | might be possible but is just not there. If this really is the case and | one sort of dependency arises from the character state and another | arises from the taxa being considered, then it is very likely that they | should be modelled in slightly or completely different ways. This resulted from c.10 minutes thinbking in response to Steve's point. It does seem to be true that there are these 2 types of dependencies, but we need to think much more about (a) whether they need to be treated differently and (b) whether there aren't even more types. Again, shall we keep this thread going or set it aside to pick up later. I'm inclined to the latter. Cheers - k

1 0

Re: Questions on Dependencies and identifiers
by Kevin Thiele 06 Dec '01

06 Dec '01

Barry - these are just the sort of problems that Challenge 1 was designed to throw up. In retrospect, though, it's perhaps too complex already. Maybe as we consider it we can break it up into sub-challenges? Dependencies are indeed a large issue that need to be dealt with carefully. Should we set them aside for the time being or do you want to deal with them now? When I put in that bit about mucronate within a notch my thought was that there would be two states "obtuse" and "mucronate within an apical notch" and obviously Jim's right that it's dependent on the context - the other taxa in the treatment - whether to treat it like this or otherwise. Your interpretation: | Apex shape | obtuse | notched | | Apex projection | immucronate | mucronate within notch (dependent on 'Apex shape - notched') seems to have a dependency between a state of one character and a state of another character (I'll use the character/state terminology for the time being here for simplicity). This is not the way it's usually done - dependencies usually set up a relationship between an entire character and a state of another character. e.g. the character Leaf shape ovate/elliptic will be dependent on the state "present" of the character "Leaves present/absent". In an interactive key the behaviour will be that if someone chooses Leaves absent then the character Leaf Shape will disappear. Likewise when building natural language descriptions, if a taxon has no leaves then the leaf descriptive characters will all be skipped. You seem to be doing something else here, and I'm not sure what's the behaviour you were trying to get. Can you give us a fragment of your model for the simplest case extractable from Challenge 1, without the complications of numeric characters, dependencies etc. Yes, there's a real need for character lists and identifiers. Do we go with e.g. <Feature Name = "Leaves"> <Value>ovate</Value> </Feature> or <Feature FeatureID="23" valueID = "6"> The former certainly makes for better human-readability, but is this important? I would have thought that a computer can make sense of either, and we can support both. The parser would simply need to know that if the attribute FeatureID is specified then use that and look up the idref to get the name, else if the attribute Name is specified then use that. Presumably our document structure will be the same whether we use Names or Idrefs? Cheers - k

1 0

Re: Questions on Dependencies and identifiers
by Jim Croft 06 Dec '01

06 Dec '01

Kerry wrote: > First, I am stuck trying to adequately model the structure and dependencies > of the leaf apex characters > "apex obtuse or minutely mucronate within an apical notch". I am parsing > this out as: > Apex shape > obtuse > notched > Apex projection > immucronate > mucronate within notch (dependent on 'Apex shape - notched') > The model has an attribute for 'dependent on' under the character element. > Does the proper solution of this problem require a 'dependent on' attribute > as part of the //character/state element as well? Am I parsing the data > correctly? Maybe... but is there a correct answer to this in the absence of context? Without knowing what the other taxa in the application are, you could get away with: Apex shape obtuse minutely mucronate in an apical notch Or even, trivially: Apex shape obtuse or minutely mucronate in an apical notch It maybe that there are only two character states in this group of taxa - the bluntish ones with obtuse/mucronate tips and the sharp ones with acute/attenuate tips. In this case building a line of dependency might not only be unnecessary, but might it also be misleading? But I do not this that was the real issue, was it? The challenge was, if you had a dependency, how would you deal with it... > The first problem led me to the second problem, which is: how do we > identify the character or state another character depends on? This is not a > problem when working with DELTA format data which has unique identifiers > already, but text descriptions do not. Is an Xpath description adequate or > do we need to be able to have a structure that allows us to uniquely > identify what a character state describes? Is there any harm in requiring a unique id for each node? Most descritpive applications probably do this already to manage their data matrices. DELTA lets you see them which is considered mabye useful, Lucid doesn't which is considered healthy... :) > I originally thought a unique identifier was needed to identify > missing data not explicitly encoded in the description and to provide a > pointer to characters or states that are other characters of states depend > on. Now that I see how complex this may be to implement, I'm wondering if > another way would be better. Is our structure/Schema/DTD is right, it would be nice to think that it was self referenceing and that we did not have to worry about internal ids and pointers, wouldn't it... Thinking further about Kevin's two types of dependency, context and content (or whatever the terminology was), isn't the former an artifact of the character set we have decided to use and the latter an artifact of the taxa we are covering? The former is just not possible, the latter might be possible but is just not there. If this really is the case and one sort of dependency arises from the character state and another arises from the taxa being considered, then it is very likely that they should be modelled in slightly or completely different ways. > I originally included an attribute called 'describes' under the > //character/state element. I thought that this could be used for two > purposes: (1) to add descriptive text to the character state and (2) to help > build a unique identifier for a character state. The first use is no > problem; this structure allows an author to easily add "high' to the value > for the character state describing height for example. The structure is > less useful for building an identifier for what the character state > describes. //character/state is not always going to be unique is it? The tip mucronate, obtuse, acute condition could exist in scales, bracts, leaves, petals, sepals, anthers, etc. A more complete path is going to have to be specified which could get pretty nasty... but hey, computers are good at nasty... > I thought I could build a unique identifier by combining the > character name and the 'describes' attribute. So for > > <Character> > <CharName>Leaf</Charname> > <State describes="PlaneShape" modifier="+/-">oblong</State> > </Character> > > The identifier 'LeafPlaneShape' identifies what the state describes. This > allows you to recognize that even when there are multiple states (ie. > 'ovate', 'round', etc.) they are all describing one aspect of the leaf. If > nested characters are recognized, the identifier must get longer. > 'ApexShape' or 'ApexProjection' wouldn't work, it would have to be > 'LeafApexShape' or 'LeafApexProjection'. If the nesting gets deep, this > gets awkward pretty fast. It would work, though. but not very elegantly... Dependency is the act of a state/value controlling the applicability or otherwise of a character/feature or as Kevin has pointed out a taxon controlling a character feature (in the identification process this get s bit circular as you have to hav an idea what a taxon is before you can decide if a character is inapplicable, but such knowledge can be built into the data in various ways). Thus the model will have to accommodate that a certain character/feature is unavailable if a certain state/value exists, perhaps in a totally different branch of the character/feature tree - and the obligate reciprocal relationship between controlling state/values and controlled characters/feataures. Unique ids, even arbitrary sequential numeric ones a la DELTA, might be the best (or an adequate) way to do this. How about: <feature id="123"> <feateName>leaf</featureName> <character id="6" nullcharacter="3" nullstate="4"> <characterName>leaf shape</characterName> <state modifier="+/-" value="present">oblong</State> <state value="rarely">ovate</State> <state value="rarely">obovate</State> </character> </feature> ... just doodling... maybe this needs to be specified in a separate characterlist up front, rather than in the guts of the descriptive data. Actually we have not even decided that yet have we? Should the list of characters/states and the lists of taxa be described as separate blocks, or should they be implied from the content and structure of the standard data file. I prefer the former and both DELTA and Lucid do this... but have we decided that this is the way to go? > Do others think that having an identifier is necessary and, if so, > has anyone been able to come up with a better way to handle it? Is Xpath > adequate to our needs? I always have problems with Xpath - it is supposed to work, but I always manage to get lost in the hierarchy... need more practice... :) jim

1 0

Questions on Dependencies and identifiers
by Barringer, Kerry 05 Dec '01

05 Dec '01

I am still working through Challenge 1, modifiying my data model so that it can completely model the text descriptions. I am now stuck on two problems that arose when I was trying to work through the practical implications of the model. First, I am stuck trying to adequately model the structure and dependencies of the leaf apex characters "apex obtuse or minutely mucronate within an apical notch". I am parsing this out as: Apex shape obtuse notched Apex projection immucronate mucronate within notch (dependent on 'Apex shape - notched') The model has an attribute for 'dependent on' under the character element. Does the proper solution of this problem require a 'dependent on' attribute as part of the //character/state element as well? Am I parsing the data correctly? The first problem led me to the second problem, which is: how do we identify the character or state another character depends on? This is not a problem when working with DELTA format data which has unique identifiers already, but text descriptions do not. Is an Xpath description adequate or do we need to be able to have a structure that allows us to uniquely identify what a character state describes? I originally thought a unique identifier was needed to identify missing data not explicitly encoded in the description and to provide a pointer to characters or states that are other characters of states depend on. Now that I see how complex this may be to implement, I'm wondering if another way would be better. I originally included an attribute called 'describes' under the //character/state element. I thought that this could be used for two purposes: (1) to add descriptive text to the character state and (2) to help build a unique identifier for a character state. The first use is no problem; this structure allows an author to easily add "high' to the value for the character state describing height for example. The structure is less useful for building an identifier for what the character state describes. I thought I could build a unique identifier by combining the character name and the 'describes' attribute. So for <Character> <CharName>Leaf</Charname> <State describes="PlaneShape" modifier="+/-">oblong</State> </Character> The identifier 'LeafPlaneShape' identifies what the state describes. This allows you to recognize that even when there are multiple states (ie. 'ovate', 'round', etc.) they are all describing one aspect of the leaf. If nested characters are recognized, the identifier must get longer. 'ApexShape' or 'ApexProjection' wouldn't work, it would have to be 'LeafApexShape' or 'LeafApexProjection'. If the nesting gets deep, this gets awkward pretty fast. It would work, though. Do others think that having an identifier is necessary and, if so, has anyone been able to come up with a better way to handle it? Is Xpath adequate to our needs? Thanks, Kerry ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Kerry Barringer (Curator of the Herbarium) Herbarium 718-623-7318 (office) Brooklyn Botanic Garden 718-941-4774 (fax) 1000 Washington Avenue 718-623-7312 (herbarium) Brooklyn, NY 11225-1099 U.S.A. kbarringer(a)bbg.org http://www.bbg.org/ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

Una rote: > These <freeform comments> are natural-language paragraphs embedded > in otherwise fully parsable data, which is the opposite of Kevin's > case above. And much easier to handle, I think. That is an intersting perspecitve, and probably right... but although coming from different directions, is there a real difference in the final result? Both end up as mixed content of structured (and parsable character data interspersed with free form text... > I put interesting data in the comments for an extremely important > reason: I know I don't have enough data to represent the probable > diversity in the taxon (the item), and I do not want to understate > the diversity. Better to leave the character coded as unknown, > but record the data. At some point in the future, the data can be > moved from the comments to where it belongs. Absolutely... and isn't this the holy grail we are seeking in this descriptive data standard? I am quite sure that the model we come up with with have a structured place for this freeform and comment stuff... to do anything less will be to admit failure... jim

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

In respnse to Steve Kevin wrote: > The terminology is +/- trivial at this stage, but I'll explain that I chose > something different from character/state simply to break with tradition for > a while. Traditionally, a character has states and that's it - a 2-level > tree. In the example above one character (leaf) has as child another > character (margin). This seems odd to many people thinking traditionally > about characters/states. Let's agree that we'll use them interchangeably for > now. And I think we have all become bilingual in this regard... But sooner rather than later I would like us to nail this terminology down, to free up the synonyms forus elsewhere in our model as much as anything else... Also, I am not yet convinced that unbounded nesting of characters is necessary the best way to go in terms of representing a hioerarch of character data... but maybe it is... At the end of the day, a state of a particular character (or a value of a feature) is used in a key decision or choice and the hierachy is not all that important other than the order of presentation orlogical grouping of characters. In the description the hierarch similar provides the logical order of the characters and their states/values. Perhaps we could use features for the hierachy and characters for the ultimate branch. For example features could contain features or characters (but not both) and characters would have states which would have values (present, absent, doubtful, rarely, in error, or whatever, or a measurement/count). Reaching agreement of this level of data description, and the terminology we are going to use would seem to be essential for clear and unambiguous communication within the group. But if we can't agree, I can wait... because we aregoing to have to doit sooner or later... jim

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

> No, this is not equivalent to dependencies, although at first sight it seems > similar. It's a tree-topology rule: the rule "<states>s can have > <characters>s as siblings" allows certain topologies for the > data-representation trees that would be preluded by the rule "<states>s > cannot have <characters>s as siblings". As I said, I don't know whether the > rule's a good or necessary one or not. But it is important that we sort it out... Dependency can be expressed in ways other than allowing the ultimate state to be a parent of a character, can't it? This would seem to be very messy and recursive and would probably break the clean hierarchical representation of data we seemed to be aiming towards. > Dependencies establish relationship rules between different parts of the > tree. Sometimes these rules affect parent-sibling relationships as in the > example, but sometimes not - that is, dependencies are more complex than can > be handled by a topology rule. I tend to agree with this... > In the key, if a user chooses a state marked * the character "Trap > Structures" will disappear because of the dependencies. Note that the > dependency on tree and shrub is not a logical one but a contextual one - it > just happens that no trees or shrubs are carnivorous, so the dependency's > useful but not logically necessary. Well, this is important, isn't it? I think we have just established that our model must accomodate two kinds of dependency one based on logic andaracter tree topology, an the other based on content/context. In the former a character can not be present because to parent feature/character/organ is absent; in the latter it could conceivalbly be present, but just isn't. Up till now applications have just allowed scoring dependencies without making this distinction. > I don't think these dependency relationships can be expressed merely by the > topology of the tree, as you suggest. We will need to deal with dependency > rules as a separate issue. I agree with this. We need to bear it in mind, but let's try and nail down a basic descriptive framework first... jim

1 0

Re: It's How the Data will be Used that Counts
by Jim Croft 05 Dec '01

05 Dec '01

Kevin wrote: > I seem to be in a minority of one here again, but I'll continue to argue my > case for a bit longer. why not... whe have be doing this for several years now, a few more email iterations won't hurt... :) > 1. Exactly, it's easier to go from data to +/- natural language, which is > precisely why we need to try hard to facilitate the reverse. Which is not all that difficult... to mark up a single description manually is a relatively trivial task... but to process a lot automatically is another matter... and to do it in a way that the whole community accepts is another matter entirely... :) > 2. If we can effectively embed fully parsable data in a natural-language > paragraph, why not? becasue it creates mixed content (legal, but evil) where the data is parsable, but not structured. If we can not represent our data universe in a database in an elegant and easily understood way we may not necessarily have failed, but we will have fallen short of the target by a large margin. In fact, DELTA already does this sort of thing by allowing liberal appending and prepending <freeform comments> all over the place. While this makes for quasireadable descriptions, authors often embed interesting character data in the comments making it unusable by other applications, even within the DELTA suite. The 'rarely' and 'misinterpreted' scoring options in Lucid might be seen as an attempt to capture some of this information. > 3. If a structured data document based on our standard is a subset of a > marked-up description based on the standard, then creating a standard that > can support the latter gives us the best of both worlds. If it can be done, > why not? I agree... this is a laudable aim... but I must admit to never thinking of a structured document as a subset of a marked up description - oftentimes the reverse may be the case... Isn't it better to think of both structured documents and marked up descriptions as being subsets of the standard we are trying to create? > Personally I think that creating an XML representation of structured data > would be a doddle. But as we have seen, getting everyone to agree that one person's way of doing it is the one true and proper path to descriptive enlightenment is no easy task... > Creating a fully parsable but lossless XML representation > of a natural language description (which hence can also handle the degraded > case of structured data) - now that would really be something to write home > about! Well, dreaming about it at least... I think we are dealing with two different, and I fear irreconsilable things here... descriptions by their very nature are lossy - they are abstract representations of the gestalt of a sample of a taxon, often with an arbitrary word limit, attempting to portray in a familiar format what an author thinks a taxon looks like. Structured documents such as DELTA and Lucid at least have to potential to store everything that is remotely interesting about every taxon in the set and often come close to achieving it in reality. So what if the resulting descriptions do not have the poetic beauty of a Shakespearian sonnet; at least the information will be there and retrievable... In the case of biological description, beauty is not necessarily truth, or at least the whloe truth... > Anyone else out there +/- agree with me, or should I give up now? Don't do that... if you do, you will never have anything to write home about... jim

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

I seem to be in a minority of one here again, but I'll continue to argue my case for a bit longer. ----- Original Message ----- From: "Robert A. (Bob) Morris" <ram(a)CS.UMB.EDU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 2:11 PM Subject: It's How the Data will be Used that Counts | I think natural language parsing understanding is harder than natural | language production from structure. So I think there is less work to | go from data to description than the other way around. 1. Exactly, it's easier to go from data to +/- natural language, which is precisely why we need to try hard to facilitate the reverse. 2. If we can effectively embed fully parsable data in a natural-language paragraph, why not? 3. If a structured data document based on our standard is a subset of a marked-up description based on the standard, then creating a standard that can support the latter gives us the best of both worlds. If it can be done, why not? Personally I think that creating an XML representation of structured data would be a doddle. Creating a fully parsable but lossless XML representation of a natural language description (which hence can also handle the degraded case of structured data) - now that would really be something to write home about! Anyone else out there +/- agree with me, or should I give up now? Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

----- Original Message ----- From: "Steve Shattuck" <Steve.Shattuck(a)CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 11:31 AM Subject: It's How the Data will be Used that Counts | Kevin's representation is too focused on text descriptions. A more complete | representation might be: | | <character name="leaf"> | <state>present</state> | <character name="leaf margin"> | <state>serrate</state> | <character name = "tooth orientation"> | <state>forward-pointing</state> | </character> | </character> | </character> I would argue that this is a less complete representation, because you've abstracted the original data further than I have. I won't use the "markup" from my last email for "Leaf margins serrate with forward-pointing teeth", as this was designed to exemplify one limited problem (how to expand a description if characters are nested) and wasn't actually marked up in the way I proposed for challenge case 1. Two possible ways I'd do this example are: "Leaf margins serrate with forward-pointing teeth" {Using the rule "<states>s cannot have <characters>s as siblings"} <Feature><Name>Leaf</Name> <Feature name = "Presence" value = "Present"> <Feature Name="marginal toothing">margins <Value>serrate</Value> </Feature> with <Feature Name = "tooth orientation"> <State>forward-pointing</State> teeth </Feature> </Feature> {Relaxing the rule so that <states>s can have <characters>s as siblings} <Feature><Name>Leaf</Name> <Feature name = "Presence" value = "Present"> <Feature Name="marginal toothing">margins <Value>serrate</Value> with <Feature Name = "tooth orientation"> <State>forward-pointing</State> teeth </Feature> </Feature> </Feature> Your proposal is: <character name="leaf"> <state>present</state> <character name="leaf margin"> <state>serrate</state> <character name = "tooth orientation"> <state>forward-pointing</state> </character> </character> </character> The score: 1. Can we parse from these the data atoms "leaf = present "leaf margin = serrate" and "tooth orientation = forward-pointing"? Kevin's = Yes Steve's = Yes Can we easily retrieve from these the original natural language string? Kevin's = Yes Steve's = No On this scoring I'm one up. Then again, yours would be slightly easier to parse than mine, so we're probably equal. What's most important here? Dunno. Further, it seems to me that yours is a subset of mine: a Schema that allowed mine would also allow yours, but not vice versa. | In my original, DELTA-centric model I used a <description> tag to try and | capture the text description information separate from the <state> | information. My thinking was that these two | requirements/approaches/viewpoints are too distinct to cram together without | falling into the same trap as the current DELTA Standard (which is a | least-common denominator approach). Yes we could tag the bits of free-form text. But is there any need? They will (by definition) be ignored by all processors except for natural-language - since the NL is fully retrievable from my model, why not leave them untagged? In your model, they would need to be tagged since the model does not represent a natural description - it represents abstracted data from which a description can be +/- created. | The problem here is that the phrase "Leaf margins serrate with | forward-pointing teeth" concerns 3 characters (leaf, margin and teeth) and 1 | implied and 2 expressed states (present, serrate and forward pointing) with | the characters being dependent (and therefore the context containing | significant information - we know that 'teeth' have something to do with | 'serrate' which has something to do with 'leaf margins' - the leaves being | present because we're describing them). There's a lot of logic involved in | parsing this. I can't think of a simple way of representing all this | complex information without separating it at some level. Kevin's suggestion | represents the text description and mine the underlying data, but neither | works well for the other. I agree - there's still too much complex logic even in the very simple types of examples we're using so far. We need somehow to step back further to even more basic examples to tease these issues out. Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

----- Original Message ----- From: "Steve Shattuck" <Steve.Shattuck(a)CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 11:31 AM Subject: It's How the Data will be Used that Counts | A couple of points: | | Kevin suggests a rule "<states>s cannot have <characters>s as siblings". | But this is what DELTA calls Dependencies, it represents the state that | controls a character. This would seem to be a good thing (and may be very | important). No, this is not equivalent to dependencies, although at first sight it seems similar. It's a tree-topology rule: the rule "<states>s can have <characters>s as siblings" allows certain topologies for the data-representation trees that would be preluded by the rule "<states>s cannot have <characters>s as siblings". As I said, I don't know whether the rule's a good or necessary one or not. Dependencies establish relationship rules between different parts of the tree. Sometimes these rules affect parent-sibling relationships as in the example, but sometimes not - that is, dependencies are more complex than can be handled by a topology rule. In dependencies, there may be several controlling states for a character, and these may or may not occur on nodes antecedent to the node in question. An example (from a key of mine): the character: Trap structures (on carnivorous plants) submerged or underground bladder-traps pitcher-traps sticky hairs irritable leaf-blade segments has controlling states in several other characters, viz: Nutritional strategy carnivorous parasitic* neither carnivorous nor parasitic* Habit tree* shrub* terrestrial herb aquatic herb In the key, if a user chooses a state marked * the character "Trap Structures" will disappear because of the dependencies. Note that the dependency on tree and shrub is not a logical one but a contextual one - it just happens that no trees or shrubs are carnivorous, so the dependency's useful but not logically necessary. I don't think these dependency relationships can be expressed merely by the topology of the tree, as you suggest. We will need to deal with dependency rules as a separate issue. Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by Kevin Thiele 04 Dec '01

04 Dec '01

Thanks Steve - at last we have some alternatives we can sink our teeth into. Comments below. ----- Original Message ----- From: "Steve Shattuck" <Steve.Shattuck(a)CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Tuesday, December 04, 2001 11:31 AM Subject: It's How the Data will be Used that Counts | I've been giving Kevin's approach some thought and have the following | comments: | | Kevin's original information flow model is too simplistic. A more realistic | model would be something like: | | Text Descriptions | | Text Descriptions | Phylogenetics |---> Structured Data --->| Phylogenetics | Specimens | | Identification Tools | (many) Yes, of course. There are varied sources for the structured data. It still seems to me that capturing the non-text sources will probably be a subset of what's needed to capture the text sources. This is because textual descriptions are probably the least formally structured data we need to deal with as input (with the exception of original observations which, in some taxonomists' minds at least, are highly structureless but are readily structurable) | The sources are much more varied and are often group-specific. For example, | invertebrates have very few good quality text descriptions (most are old, | are in a range of languages (English, French, German, etc), vary greatly in | style, quality, etc. etc) and the majority of invertebrates are currently | undescribed (having 80% new taxa during a revision is common). Yes I agree, a botany bias is showing through here. | Similarly, the outputs required vary greatly and in ways hard to predict. | While text descriptions would seem to be a common requirement, they are in | some ways "legacy" and may become less important in the future as | applications (and users) become more sophisticated. We need to make sure we | keep this range of uses in mind at all times. Yes, but see comment above. | Because of this I don't really think the details of the model | matter too much, more that it is rich enough to represent all data of | interest. Exactly my point - it needs to be rich enough to capture and express a textual description, hopefully losslessly! | ******** | | I've also been thinking about Kevin's latest example: | | "Leaf margins serrate with forward-pointing teeth" | <feature name="leaf"> | <feature name="margin"> | <feature name = "teething shape"> | <value>serrate</value> | </feature> | <feature name = "teeth orientation">with | <value>forward-pointing</value>teeth | </feature> | </feature> | </feature> | | First, it seems to me that "feature" is what taxonomists call "character" | and "value" is "state". Being a traditionalist I'll switch back to this | common terminology: The terminology is +/- trivial at this stage, but I'll explain that I chose something different from character/state simply to break with tradition for a while. Traditionally, a character has states and that's it - a 2-level tree. In the example above one character (leaf) has as child another character (margin). This seems odd to many people thinking traditionally about characters/states. Let's agree that we'll use them interchangeably for now. (Other points have been split into separate emails) Cheers - k

1 0

Re: It's How the Data will be Used that Counts
by unknown＠example.com 04 Dec '01

04 Dec '01

Perfect agreement with Bob? Maybe he's starting to rub off on me! >For this reason, it is probably unnecessary to represent the state >`present' at all, provided the semantics could reasonably require that >a feature which is absent is never described. Is that a reasonable >requirement? I think this is probably true. I can only come up with two potential problems: (1) In this case the state "present" is implied but this may not always be the case when states are not specified. For example: <character name="distribution"> <character name="country"> <state>Australia</state> </character> </character> A default state of "present" doesn't make much sense for the character "distribution". This may not be a serious problem but we should give it some thought. (2) I think the bigger problem is the flip-side: if a character isn't in the description then it must be "absent". We can't do it this way. >To me, the main thing that this kind of model implies is the need, in >some cases, to provide a thesaurus, e.g. to provide advice that if a >character (here `margin') has a subcharacter `teeth' then it may be >described as `serrate'. Is that bad? Or would a purely textual >description which just said "Leaf margins with forward-pointing teeth" >be deemed wrong absent the word "serrate" ? Adding a thesaurus would add way too much work and removing "serrate" would be too restrictive. >I think natural language parsing understanding is harder than natural >language production from structure. So I think there is less work to >go from data to description than the other way around. If true (and I think it is) then this would suggest a more DELTA-like structure and less of a text-markup structure for the representation. Steve

1 0

Re: It's How the Data will be Used that Counts
by Una Smith 04 Dec '01

04 Dec '01

Kevin Thiele wrote: >> I seem to be in a minority of one here again, but I'll continue to argue my >> case for a bit longer. Go for it. Defensiveness is a waste of everyone's time. >> 2. If we can effectively embed fully parsable data in a natural-language >> paragraph, why not? Jim Croft wrote: >becasue it creates mixed content (legal, but evil) where the data is >parsable, but not structured. [...] >In fact, DELTA already does this sort of thing by allowing liberal >appending and prepending <freeform comments> all over the place. While >this makes for quasireadable descriptions, authors often embed >interesting character data in the comments making it unusable by other >applications, even within the DELTA suite. [...] These <freeform comments> are natural-language paragraphs embedded in otherwise fully parsable data, which is the opposite of Kevin's case above. And much easier to handle, I think. I put interesting data in the comments for an extremely important reason: I know I don't have enough data to represent the probable diversity in the taxon (the item), and I do not want to understate the diversity. Better to leave the character coded as unknown, but record the data. At some point in the future, the data can be moved from the comments to where it belongs. -- Una Smith Los Alamos National Laboratory, Mailstop K-710, Los Alamos, NM 87545

1 0

It's How the Data will be Used that Counts
by unknown＠example.com 04 Dec '01

04 Dec '01

I've been giving Kevin's approach some thought and have the following comments: Kevin's original information flow model is too simplistic. A more realistic model would be something like: Text Descriptions | | Text Descriptions Phylogenetics |---> Structured Data --->| Phylogenetics Specimens | | Identification Tools (many) The sources are much more varied and are often group-specific. For example, invertebrates have very few good quality text descriptions (most are old, are in a range of languages (English, French, German, etc), vary greatly in style, quality, etc. etc) and the majority of invertebrates are currently undescribed (having 80% new taxa during a revision is common). Similarly, the outputs required vary greatly and in ways hard to predict. While text descriptions would seem to be a common requirement, they are in some ways "legacy" and may become less important in the future as applications (and users) become more sophisticated. We need to make sure we keep this range of uses in mind at all times. The Structured Data is the format we're talking about here. Don't know what that will look like yet (but see below). I think it's important to realise that applications will be used to move among these in almost all cases and only very rarely will people manipulate the data directly. It's also important to remember that XSLT is nothing more than an application. Some comments seem to imply that XSLT is part of XML, but it isn't. For example, from a BioLink perspective (and DELTA and probably LucID) the above model will need to be extended by adding: Text Descriptions | | Text Descriptions Phylogenetics |---> Structured Data --->| Phylogenetics Specimens | | | Identification Tools (many) | |<-- | BioLink |--> | DELTA | LucID Builder That is, applications will import Structured Data, manipulate it and spit it out again. Because of this I don't really think the details of the model matter too much, more that it is rich enough to represent all data of interest. ******** I've also been thinking about Kevin's latest example: "Leaf margins serrate with forward-pointing teeth" <feature name="leaf"> <feature name="margin"> <feature name = "teething shape"> <value>serrate</value> </feature> <feature name = "teeth orientation">with <value>forward-pointing</value>teeth </feature> </feature> </feature> First, it seems to me that "feature" is what taxonomists call "character" and "value" is "state". Being a traditionalist I'll switch back to this common terminology: "Leaf margins serrate with forward-pointing teeth" <character name="leaf"> <character name="margin"> <character name = "teething shape"> <state>serrate</state> </character> <character name = "teeth orientation"> <state>forward-pointing</state> </character> </character> </character> A couple of points: Kevin suggests a rule "<states>s cannot have <characters>s as siblings". But this is what DELTA calls Dependencies, it represents the state that controls a character. This would seem to be a good thing (and may be very important). Kevin's representation is too focused on text descriptions. A more complete representation might be: <character name="leaf"> <state>present</state> <character name="leaf margin"> <state>serrate</state> <character name = "tooth orientation"> <state>forward-pointing</state> </character> </character> </character> This allows us to directly extract: leaf = present leaf margin = serrate tooth orientation = forward-pointing This will be important for both identification tools and phylogenetics. Trying to recover this information from Kevin's representation should be possible but will require a number of assumptions be made about the data. This representation also captures dependencies (although this is an advanced feature we shouldn't be talking about yet). In my original, DELTA-centric model I used a <description> tag to try and capture the text description information separate from the <state> information. My thinking was that these two requirements/approaches/viewpoints are too distinct to cram together without falling into the same trap as the current DELTA Standard (which is a least-common denominator approach). The problem here is that the phrase "Leaf margins serrate with forward-pointing teeth" concerns 3 characters (leaf, margin and teeth) and 1 implied and 2 expressed states (present, serrate and forward pointing) with the characters being dependent (and therefore the context containing significant information - we know that 'teeth' have something to do with 'serrate' which has something to do with 'leaf margins' - the leaves being present because we're describing them). There's a lot of logic involved in parsing this. I can't think of a simple way of representing all this complex information without separating it at some level. Kevin's suggestion represents the text description and mine the underlying data, but neither works well for the other. Two steps forward, one step back. Sorry about that. Thanks, Steve

1 0

It's How the Data will be Used that Counts
by Robert A. (Bob) Morris 03 Dec '01

03 Dec '01

Wow, Steve, we are virtually in perfect agreement! A first! I have one small quibble---you actually raised it yourself though---and some suggestions Steve Shattuck writes: > ... > Kevin's representation is too focused on text descriptions. A more complete > representation might be: > > <character name="leaf"> > <state>present</state> > <character name="leaf margin"> > <state>serrate</state> > <character name = "tooth orientation"> > <state>forward-pointing</state> > </character> > </character> > </character> > > > This allows us to directly extract: > > leaf = present > leaf margin = serrate > tooth orientation = forward-pointing > > The problem here is that the phrase "Leaf margins serrate with > forward-pointing teeth" concerns 3 characters (leaf, margin and teeth) and 1 > implied and 2 expressed states (present, serrate and forward pointing) with > the characters being dependent (and therefore the context containing > significant information - we know that 'teeth' have something to do with > 'serrate' which has something to do with 'leaf margins' - the leaves being > present because we're describing them). For this reason, it is probably unnecessary to represent the state `present' at all, provided the semantics could reasonably require that a feature which is absent is never described. Is that a reasonable requirement? For example, in such a case to extract from the XML all taxa which have a leaf character in their description is not any harder for lacking <state>present</state>. In fact it is triffling bit easier. The desire for a `present' state perhaps comes from table-based character-by-state organization where it could be hard to distinguish whether the character is absent from the taxon or absent from the data. That distinction can be made moot here, perhaps. `presence' may be the only such state though. >There's a lot of logic involved in > parsing this. I can't think of a simple way of representing all this > complex information without separating it at some level. If we assume that there is a rigorous semantics to the effect that syntactically nested characters are always logically nested (and here we may need to return to "feature" if "character" is too dear to overload), is there a problem with this: <character name="leaf"> <character name="margin"> <character name="teeth"> <character name = "orientation"> <state>forward-pointing</state> </character> </character> </character> </character> To me, the main thing that this kind of model implies is the need, in some cases, to provide a thesaurus, e.g. to provide advice that if a character (here `margin') has a subcharacter `teeth' then it may be described as `serrate'. Is that bad? Or would a purely textual description which just said "Leaf margins with forward-pointing teeth" be deemed wrong absent the word "serrate" ? >Kevin's suggestion > represents the text description and mine the underlying data, agreed >but neither works well for the other. I think natural language parsing understanding is harder than natural language production from structure. So I think there is less work to go from data to description than the other way around. > > Two steps forward, one step back. Sorry about that. Nah, at most 1/2 step back. Is your middle name Zeno? > > Thanks, Steve > Bob Morris

1 0

Re: Thought and a Question
by Leigh Dodds 03 Dec '01

03 Dec '01

> | I'd like to suggest, again, that the solutions for the challenges > | should be presented as simple data models. Syntax can come later > | (and may fall naturally out of the model). > > Leigh - can you give us an example of your solution to challenge 1 then? Certainly. I'm trying to free up some time to devote a bit more effort to this, so I can back up my comments with some useful (hopefully!) contributions. Hope to have something later this week. Cheers, L.

1 0