November 2001 - tdwg-content

Alternate Hierarchies
by Kevin Thiele 30 Nov '01

30 Nov '01

I agree. I've added an issue to challenge case 6 to address this. 6. Remote and local taxon lists a.. remote referencing of hierarchies from master lists e.g. ITIS b.. local definition of hierarchies c.. efficient rearrangement of hierarchies with knowledge changes d.. alternate hierarchies Cheers - k ----- Original Message ----- From: "Jim Croft" <jrc(a)ANBG.GOV.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Friday, November 30, 2001 1:15 AM Subject: Re: Document vs. database | > In DELTA, Lucid etc we express alternate hierarchies in alternate documents. | > Do we need to do better than this? | | Let's stop this line of thought right now... | | build one hierachy of characters that works first, THEN consider how to | manage alternatives... | | maybe... | | jim |

1 0

Character hierarchies
by Kevin Thiele 30 Nov '01

30 Nov '01

----- Original Message ----- From: "Jim Croft" <jrc(a)ANBG.GOV.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Friday, November 30, 2001 1:03 AM Subject: Re: Document vs. database | Also, in our new model we also want to show more than just presence | or absence of a state. Don't we want to show if it is present/rarely | present/present by misinterpretation/rarely present by | misisnterpretation, etc., and my favourites yet to be implemented as | a character stae attribute: definitely absent, absent by | misinterpretation, unknown, unscored Of course these will come in. In my example, see: <Feature Name="Longevity"><Value>persistent</Value>(rarely<Value Qualifier="rarely">deciduous</Value>)</Feature> The other qualifiers (misinterps etc) aren't in there yet, but will come. We'll have to be much more sophisticated that in my example, of course. Now how about "definately known to be unknown" | > I'm simply suggesting allowing n levels: | > <Leaves> | > <shape> | > <ovate> | > <elliptic> | > <Flowers> | > <colour> | > <blue> | > <red> | | That is where we want to get to do, but echoing Bob's words, what we | are after is the structure that allows us to get to that, giving | people all the appearance of freedom to do what they like but not | actually doing it and imposing a schematic straight jacket on the data. | A schema for the data structure rather than the descriptive data itself. | Like all freedoms, the descriptive data one too must be an illusion or | we wil miss out on on all the creative potential that it offers as the | free spirits among us do theri own thing. But that is another | discussion... :) Sorry, I made a blue with the example. It should have been: <Feature name = "Leaves"> <Feature name = "shape"> <Value>ovate</Value> <Value>elliptic</Value> </Feature> </Feature> <Feature name = "Flowers"> <Feature name = "colour"> <Value>blue</Value> <Value>red</Value> </Feature> </Feature> | There has been a lot of talk about the feature/value paradigm and how | this might be made to represent a biological description and even nested | features in such descriptions... | | At the recent TDWG meeting Richard Pankhurst described something like a | feature/character/value paradigm and at the time I made a note that this | was probalably worth considering in more detail, but so far have not had | the time. | | It probably would mean something like: | | <feature name="leaf" charecter="margin" value="serrate"/> or some But I can't see that "margin" is a thing different in kind from "leaf". Sure, the margin is part of a leaf but equally the leaf is part of the plant, so if you have feature|character|value then you will end up with "leaf" as a feature in one context and a character in another - messy, surely. My example above just has Feature|value. The difference is that a feature is a thing but a value is not (both "leaf" and "margin" are things that can be pointed at, "ovate" and "blue" are not). In the XML tree representation above features are nodes and values are twigs (or leaves if you like - of course, a "twig" would be a node, not a twig (or a leaf) - oh dear, can someone invent a language that isn't a language!) Ditto for Jim's second example: |<feature name="leaf"> | <character name="margin"> | <observation value="serrate" modification="obscurely" state="present" | basis="specimen" reference="Stevens in NGF67432"/> | <observation ..... /> | <observation ..... /> | </character> |</feature> Try interpolating another level in here (e.g. marginal teeth shape) and you get into strife. Whereas with <feature name="leaf"> <feature name="margin"> <value value="serrate" modification="obscurely" state="present" basis="specimen" reference="Stevens in NGF67432"/> <observation ..... /> <observation ..... /> </feature> </feature> We can surely add as many levels as we like. Or can we - here's a challenge - find the simplest way of expanding description 1: "Leaf margin serrate" into description 2: "Leaf margin serrate with forward-pointing teeth". Attempt 1. "Leaf margin serrate" <feature name="leaf"> <feature name="margin"> <value>serrate</value> </feature> </feature> "Leaf margins serrate with forward-pointing teeth" <feature name="leaf"> <feature name="margin"> <feature name = "teething shape"> <value>serrate</value> </feature> <feature name = "teeth orientation">with <value>forward-pointing</value>teeth </feature> </feature> </feature> This requires moving <value="serrate"/> from the node <feature name="margin"> to a new interpolated node <feature name = "teething shape">, so as to keep a rule that <value>s cannot have <feature>s as siblings. An alternative: Attempt 2. "Leaf margins serrate with forward-pointing teeth" <feature name="leaf"> <feature name="margin"> <value>serrate</value> <feature name = "teeth orientation"> <value>forward-pointing</value> </feature> </feature> </feature> breaks this rule. Attempt 2 seems preferable to me in terms of efficiency, but the rule seems somehow important. Of course, restricting ourselves to a 2-level heirarchy a la Lucid and DELTA makes these problems go away, but only because we artificially collapse the levels down and lose the structure: "Leaf margins serrate with forward-pointing teeth" <feature name="leaf margin teething"> <value>serrate</value> </feature> <feature name = "leaf margin teeth orientation"> <value>forward-pointing</value> </feature> I'm getting lost here - can someone help? We've +/- discussed this before, but we need more thinking around real examples such as these. Cheers - k

1 0

Re: Process: Forum software?
by Barringer, Kerry 30 Nov '01

30 Nov '01

Bob, If the group can keep up the current level of activity, a forum would be a great idea. They are really a much more flexible way to work together. Kerry

1 0

Re: Thought and a Question
by Barringer, Kerry 30 Nov '01

30 Nov '01

Thoughts on Leigh's Question At this stage, I am not worried about the differences in the XML vocabularies used by different people. For at least a little while we should explore the alternatives and see what the different models and different encodings have to offer. I have found that other peoples work has really helped me to see better solutions to some of the problems and sometimes helped me think of new ideas to try. I think that after a while, the alternatives will become set and the group will need to have a means of selecting among the alternatives. The challenge cases should provide a way to evaluate the effectiveness of the different solutions. Then the group can come up with a proposal, invite comments, and work out the first version. How to present the different solutions to the challenge cases is a problem that should be worked out now. If this was decided at the last meeting, let us know. Otherwise, it seems that many people have there own ideas of the proper way to present solutions. My ideas are: An XML data model is, for me, a DTD or a Schema. The model is an abstract representation of the data. This is what we should be producing. Personally, I prefer DTD's because I am more familiar with them. They are easier to understand because they are simple and relatively free of markup. As Bob Morris has pointed out though, they do lack some of the features found in XML Schemas. However, Schemas are more difficult to understand, especially for people who might know the systematics, but not the XML. A solution to a challenge case should have the model (DTD or Schema), markup of the challenge case that validates to the model, and a short program that solves the challenge using the markup. The last is important because it is too easy to develop a theoretically neat model that has practical problems. Also, if the group is to pick from the features of different alternatives it would be better to see how they work. These are just my thoughts and I would like to understand what others are thinking and what may have already been decided at the last meeting. Kerry

1 0

Thought and a Question
by Leigh Dodds 30 Nov '01

30 Nov '01

I'll start with the question: Am I right in assuming that the process is to propose and agree on a set of challenges, and then present different solutions for solving these? If so, how should those solutions be presented. Kevin and Kerry have both posted XML documents that meet Challenge 1, and and Steve Shattuck and I have also provided XML formats that can encode similar data. How will we judge these formats as better or worse? I can imagine producing a near infinite number of different XML vocabularies that can encode the same data (+/- attributes, different tag names, nesting, cross-referencing, etc, etc). I'd like to suggest, again, that the solutions for the challenges should be presented as simple data models. Syntax can come later (and may fall naturally out of the model). -- Now the thought: Kevin has pointed out that there is a large amount of taxonomic data which is simply free text, and it is a worthwhile goal to consider how to repurpose this information. However, I think that that is a separate work item to defining a format for capturing that data. I'd actually interpeted Challenge 1 to mean "represent the data contained in these natural language descriptions", and was ignoring (for the present) how that data would be extracted (manual markup of the text description or direct data entry into a suitable tool). Extracting information from free text is still far from being an automated process. If there is still going to be a manual element involved for some time, it seems better to have a standard and supporting tools to capture _new_ data in a rigorous format, while considering the markup and processing of 'legacy' data as a separate issue. To approach this slightly differently, the example markup that Kevin has suggested adding to free text descriptions would be interpreted according to a standard data model :) -- I hope this isn't interpreted as an attempt to derail the good progress being made (nice to see some discussion again! :) but in the 'XML world' we're seeing problems surface because of incompatibilities between specifications that have arisen due to slightly different interpretations of the XML data model. This is because the model came after the syntax. L. -- Leigh Dodds, Research Group, Ingenta | "Pluralitas non est ponenda http://weblogs.userland.com/eclectic | sine necessitate" http://www.xml.com/pub/xmldeviant | -- William of Ockham

1 0

Re: Document vs. database
by Jim Croft 30 Nov '01

30 Nov '01

> Am I missing something, but why do we keep on talking about states? This > is the last thing I think is interesting (double entendre here...) States > are frosting, observations - measurements, photos - make up the cake itself. Is this a real issue or one of semantics? When I write "leaf shape ovate", does it mean I have observed the leaf and its shape most closely matches that of ovate, or that I have observed the shape character leaf and consider it to be in a state of ovateness? The end result is probably the same... But I think the issue is worth pondering... Perhaps Richard did not have it quite right and perhaps we should be recording zero or more observations which are made of a feature that has a certain character one of the attributes (rather than the state) of which is present/ absent/misinterpreted, etc. Maybe something like: <observation sequence="3" basis="specimen" refereence="Stevens in NGF67432"> <feature name="leaf"> <character name="margin" value="serrate" state="present"/> </feature> </observation> <observation sequence="4" basis="literature" reference="Stevens 1998a"> etc... or maybe it would be better to invert it and define a feature up front and make one or more observations about it: <feature name="leaf"> <character name="margin"> <observation value="serrate" modification="obscurely" state="present" basis="specimen" reference="Stevens in NGF67432"/> <observation ..... /> <observation ..... /> </character> </feature> There are numberous ways to moddel this sort of stuff, and has been repeatedly pointed out there is no right way to do it, there are only ways. The challenge is to find one that is functional, comprehensive, flexible, extensible, pragmatic and elegant. And one that we can all agree on... > Probably getting old. as are we all... and we have to get this damn thing working before our use-by dates are up... :) jim

1 0

Process: Forum software?
by Robert A. (Bob) Morris 29 Nov '01

29 Nov '01

DON'T REPLY TO THE LIST: As I mentioned in Sydney, we have now established an instance of the vBulletin web-based forum software (see www.vbulletin.com) and are prepared to host a forum here for the current discussions if there is any desire to do so. Forums provide better threading than email (it is not based solely on the Subject field) and also permit participants to consider things somwhat more at their own schedule and according to their own interests. The board administrators can set how long postings remain at the forefront, so that it is easier to understand what is current. vBulletin allows you to be notified briefly by mail when something of interest appeared on the forum. There are a few disadvantages to moving this discussion now: (a)Quite a bit has transpired and people might feel they need to restate things in the forum [that could be an advantage....]; (b)we are new to this software and might not administer it as seamlessly as a mail-based system. So as not to clutter the list. I would urge people to /privately/ reply to me with their sympathy for or against the issue. I'll summarize and post the results and then people can discuss it, if warranted. Send mail to ram(a)cs.umb.edu. Your "Reply" facility will likely send to the list. For a glimpse of how vBulletin forums operate, see those at http://vbulletin.com/forum/index.php http://forums.progresstalk.com/index.php http://www.linuxquestions.org/questions/ http://www.ntcompatible.com/vb/index.php http://www.webdevforums.com/ As far as I can tell, vBulletin is the software under the forum parts of sourceforge.net Bob Morris p.s. Forum software is aimed at community building, so vBuilder has lots of flashy features like smiley face annotations, personal avatars, polls about the quality of a thread, "buddy lists", etc. I don't expect that SDD forums would operate with that stuff in your face, especially as we gain administrative experience.

1 0

Re: Document vs. database
by Jim Croft 29 Nov '01

29 Nov '01

> In DELTA, Lucid etc we express alternate hierarchies in alternate documents. > Do we need to do better than this? Let's stop this line of thought right now... build one hierachy of characters that works first, THEN consider how to manage alternatives... maybe... jim

1 0

Re: Document vs. database
by Jim Croft 29 Nov '01

29 Nov '01

I have been watching the animated discussion on the revived SDD list and it is tood to see... too busy to comment, but still interested, and useful things are being said by others... :) > Existing data structures allow 2-level hierarchies e.g. > #1. Leaf shape/ > 1. ovate/ > 2. elliptic/ > #2. Flower colour/ > 1. blue/ > 2. red/ Is that really two levels of character is is it only one with a state attribure? Also, in our new model we also want to show more than just presence or absence of a state. Don't we want to show if it is present/rarely present/present by misinterpretation/rarely present by misisnterpretation, etc., and my favourites yet to be implemented as a character stae attribute: definitely absent, absent by misinterpretation, unknown, unscored > I'm simply suggesting allowing n levels: > <Leaves> > <shape> > <ovate> > <elliptic> > <Flowers> > <colour> > <blue> > <red> That is where we want to get to do, but echoing Bob's words, what we are after is the structure that allows us to get to that, giving people all the appearance of freedom to do what they like but not actually doing it and imposing a schematic straight jacket on the data. A schema for the data structure rather than the descriptive data itself. Like all freedoms, the descriptive data one too must be an illusion or we wil miss out on on all the creative potential that it offers as the free spirits among us do theri own thing. But that is another discussion... :) There has been a lot of talk about the feature/value paradigm and how this might be made to represent a biological description and even nested features in such descriptions... At the recent TDWG meeting Richard Pankhurst described something like a feature/character/value paradigm and at the time I made a note that this was probalably worth considering in more detail, but so far have not had the time. It probably would mean something like: <feature name="leaf" charecter="margin" value="serrate"/> or some equivalent using XML entities rather than attributes. Is there any merit in this approach above using a feature that is say "leaf margin" and another that is say "leaf tip" and yet another that is say "leaf base"? It would seem to give some structure to the character set data. On the surface it would also allow easy generation of more readable descriptions without excessive text processing: Leaf outline ovate, margin serrate, tip acute, base attenuate, etc. as opposed to: Leaf outline ovate, leaf margin serrate, leaf tip acute, leaf base attenuate, leaf... etc. Has anyone else considered this approach? jim

1 0

Re: Document vs. database
by Kevin Thiele 29 Nov '01

29 Nov '01

----- Original Message ----- From: "Eric Zurcher" <Eric.Zurcher(a)PI.CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Thursday, November 29, 2001 3:32 PM Subject: Document vs. database | At 02:15 PM 28-11-01 +1100, Kevin Thiele wrote: | >Issues | > | >I've taken the route of marking up a textual description, using a minimum of | >tags. It seems to me that a description comprises a series of features with | >values. I've used mixed markup because I wanted to have the mimimum tagging | >and make maximum use of the text. That is, in cases like: | > | ><Feature Name="Life form"><Value>shrub</Value></Feature> | > | >I've made use of the fact that "shrub" occurs in the text by enclosing it in | ><Value> tags. In cases like: | > | ><Feature><Name>Leaves</Name> | > | >I'm using a <Name> tag rather than a Name attribute | > | >This may make processing considerably more difficult, but it seems to me | >that there are some advantages. An alternative would be to first process the | >description into a DELTA-like structure then mark up that. I chose not to go | >that route because I still have the dream that one day we'll have a method | >of semi-automatically marking up the content of natural language into | >something like the above. The other advantage of this structure I believe is | >that it maintains all the complexity and nuance of the original natural | >language (which can be retrieved merely by removing all the tags). I prefer | >this to the DELTA-process (take a natural-language description -> atomise | >it -> reconstruct semi-natural-language by putting the atoms back together | >again in some fashion). | | My position on this is rather different, and I would prefer to see | characters defined a bit more rigorously. | | As an example, suppose that the description of one taxon is given as: | | <Feature Name="Life form"><Value>shrub</Value></Feature> | | A second as: | | <Feature Name="Life form"><Value>bush</Value></Feature> | | And a third as: | | <Feature Name="Life form"><Value>small woody plant</Value></Feature> | | And perhaps a fourth as: | | <Feature Name="Growth form"><Value>shrub</Value></Feature> | | Are these equivalent? There's certainly no obvious way to associate them, | aside from knowing that "bush" and "shrub" mean pretty much the same thing | in some dialects of English. I think the goal of trying to extract meaning | by inserting a few tags into existing textual descriptions relies too | heavily on a level of artificial intelligence that we're still unlikely to | see for a couple of decades. Good point, and there will clearly be a need for some type of processing somewhere along the line to catch and allow validation of these situations. But this is a processing issue, not a structural one. There's nothing in the DELTA data structure to prevent #1. Life form/ 1. shrub/ 2. bush/ 3. small woody plant/ #2. Growth form/ 1. shrub/ 2. smallish twiggy plant/ and I argue there should be nothing in the structure of the SDD standard to prevent it either. People simply need to be aware of the issue. But the SDD does, I agree, need to be structured in such a way that validation of these situations is made possible. The first-cut structure that I proposed does not allow much eyeball validation of this type - is this a problem? | But pondering on this made me think about why Kevin and I often disagree on | some of these issues. If I may greatly over-simplify our viewpoints, his | view is (I think) that a formal description ought to be a document from | which data can be extracted; whereas I tend of view a formal description | more as a database from which text can be generated. | | So what ARE we after: the free-flowing flexibility of a document, or the | rigour and precision of a database? Fortunately, there is a fair bit of | middle ground. It's worth noting that XML was developed as a markup | language for documents, but that it's primary usage to date has been as a | sort of portable and relatively light-weight data container. Still I think | that finding the right balance between flexibility and rigour is going to | be a major challenge in this exercise. My view I suppose is that part of the power of XML (something that has never been possible in the past) is that it allows precise retrieval of atomised data from a fairly free-form document. XML blurs the distinction between database and text. Before XML a text string like: Rigid, spreading shrub to c. 1m high and wide; stems glabrous. Leaves soon deciduous..... was highly intractable to computer processing. This intractability required us to manually and substantially restructure into e.g. a DELTA file before we could really do much with it. But there are problems I think with the DELTA process vis-a-vis natural language. In the relatively common case that we begin with a natural language description, the process is: natural language ----(1)-----> DELTA -------(2)-------->natural language (etc) where (1) is human processing and (2) is largely computer (CONFOR) processing. Two related problems here. Firstly, I often don't much like the output of CONFOR (some people say we humans should put up with the limitations computers force on us, but call me old-fashioned , I rebel at that {see also Asimov's Second Law of Robotics}). The second problem is that step 2 is unidirectional - if I edit the output natural language I break the process (the edits are volatile, next time I invoke CONFOR they get overwritten). So I'm trying to explore the new possibilities opened up by XML to produce a minimally restructured but highly parsable document. I'm happy to admit that I may be off-beam here. I just reckon we shouldn't discount the possibility. | >Features are nested: | > | ><Feature><Name>Leaves</Name> | > <Feature><Name>Shape</Name></Feature> | ></Feature> | > | >Is this allowable? In XML-Spy I can create a Schema OK for this document. | >It's also well-formed. | | It's certainly allowable in XML. Is it desirable in a description? | Possibly, but I think it relates to my question from yesterday about | hierarchies of characters ought to be expressed. Nesting is a very good way | to express a single hierarchy, but not for handling alternative | hierarchies. Which do people want? Existing data structures allow 2-level hierarchies e.g. #1. Leaf shape/ 1. ovate/ 2. elliptic/ #2. Flower colour/ 1. blue/ 2. red/ I'm simply suggesting allowing n levels: <Leaves> <shape> <ovate> <elliptic> <Flowers> <colour> <blue> <red> I can't decide if this a trivial or fundamental difference. | Nesting is a very good way | to express a single hierarchy, but not for handling alternative | hierarchies. In DELTA, Lucid etc we express alternate hierarchies in alternate documents. Do we need to do better than this? Cheers - k

1 0