- tdwg-content - lists.tdwg.org

Re: Thought and a Question
by Barringer, Kerry 30 Nov '01

30 Nov '01

Thoughts on Leigh's Question At this stage, I am not worried about the differences in the XML vocabularies used by different people. For at least a little while we should explore the alternatives and see what the different models and different encodings have to offer. I have found that other peoples work has really helped me to see better solutions to some of the problems and sometimes helped me think of new ideas to try. I think that after a while, the alternatives will become set and the group will need to have a means of selecting among the alternatives. The challenge cases should provide a way to evaluate the effectiveness of the different solutions. Then the group can come up with a proposal, invite comments, and work out the first version. How to present the different solutions to the challenge cases is a problem that should be worked out now. If this was decided at the last meeting, let us know. Otherwise, it seems that many people have there own ideas of the proper way to present solutions. My ideas are: An XML data model is, for me, a DTD or a Schema. The model is an abstract representation of the data. This is what we should be producing. Personally, I prefer DTD's because I am more familiar with them. They are easier to understand because they are simple and relatively free of markup. As Bob Morris has pointed out though, they do lack some of the features found in XML Schemas. However, Schemas are more difficult to understand, especially for people who might know the systematics, but not the XML. A solution to a challenge case should have the model (DTD or Schema), markup of the challenge case that validates to the model, and a short program that solves the challenge using the markup. The last is important because it is too easy to develop a theoretically neat model that has practical problems. Also, if the group is to pick from the features of different alternatives it would be better to see how they work. These are just my thoughts and I would like to understand what others are thinking and what may have already been decided at the last meeting. Kerry

1 0

Thought and a Question
by Leigh Dodds 30 Nov '01

30 Nov '01

I'll start with the question: Am I right in assuming that the process is to propose and agree on a set of challenges, and then present different solutions for solving these? If so, how should those solutions be presented. Kevin and Kerry have both posted XML documents that meet Challenge 1, and and Steve Shattuck and I have also provided XML formats that can encode similar data. How will we judge these formats as better or worse? I can imagine producing a near infinite number of different XML vocabularies that can encode the same data (+/- attributes, different tag names, nesting, cross-referencing, etc, etc). I'd like to suggest, again, that the solutions for the challenges should be presented as simple data models. Syntax can come later (and may fall naturally out of the model). -- Now the thought: Kevin has pointed out that there is a large amount of taxonomic data which is simply free text, and it is a worthwhile goal to consider how to repurpose this information. However, I think that that is a separate work item to defining a format for capturing that data. I'd actually interpeted Challenge 1 to mean "represent the data contained in these natural language descriptions", and was ignoring (for the present) how that data would be extracted (manual markup of the text description or direct data entry into a suitable tool). Extracting information from free text is still far from being an automated process. If there is still going to be a manual element involved for some time, it seems better to have a standard and supporting tools to capture _new_ data in a rigorous format, while considering the markup and processing of 'legacy' data as a separate issue. To approach this slightly differently, the example markup that Kevin has suggested adding to free text descriptions would be interpreted according to a standard data model :) -- I hope this isn't interpreted as an attempt to derail the good progress being made (nice to see some discussion again! :) but in the 'XML world' we're seeing problems surface because of incompatibilities between specifications that have arisen due to slightly different interpretations of the XML data model. This is because the model came after the syntax. L. -- Leigh Dodds, Research Group, Ingenta | "Pluralitas non est ponenda http://weblogs.userland.com/eclectic | sine necessitate" http://www.xml.com/pub/xmldeviant | -- William of Ockham

1 0

Re: Document vs. database
by Jim Croft 30 Nov '01

30 Nov '01

> Am I missing something, but why do we keep on talking about states? This > is the last thing I think is interesting (double entendre here...) States > are frosting, observations - measurements, photos - make up the cake itself. Is this a real issue or one of semantics? When I write "leaf shape ovate", does it mean I have observed the leaf and its shape most closely matches that of ovate, or that I have observed the shape character leaf and consider it to be in a state of ovateness? The end result is probably the same... But I think the issue is worth pondering... Perhaps Richard did not have it quite right and perhaps we should be recording zero or more observations which are made of a feature that has a certain character one of the attributes (rather than the state) of which is present/ absent/misinterpreted, etc. Maybe something like: <observation sequence="3" basis="specimen" refereence="Stevens in NGF67432"> <feature name="leaf"> <character name="margin" value="serrate" state="present"/> </feature> </observation> <observation sequence="4" basis="literature" reference="Stevens 1998a"> etc... or maybe it would be better to invert it and define a feature up front and make one or more observations about it: <feature name="leaf"> <character name="margin"> <observation value="serrate" modification="obscurely" state="present" basis="specimen" reference="Stevens in NGF67432"/> <observation ..... /> <observation ..... /> </character> </feature> There are numberous ways to moddel this sort of stuff, and has been repeatedly pointed out there is no right way to do it, there are only ways. The challenge is to find one that is functional, comprehensive, flexible, extensible, pragmatic and elegant. And one that we can all agree on... > Probably getting old. as are we all... and we have to get this damn thing working before our use-by dates are up... :) jim

1 0

Process: Forum software?
by Robert A. (Bob) Morris 30 Nov '01

30 Nov '01

DON'T REPLY TO THE LIST: As I mentioned in Sydney, we have now established an instance of the vBulletin web-based forum software (see www.vbulletin.com) and are prepared to host a forum here for the current discussions if there is any desire to do so. Forums provide better threading than email (it is not based solely on the Subject field) and also permit participants to consider things somwhat more at their own schedule and according to their own interests. The board administrators can set how long postings remain at the forefront, so that it is easier to understand what is current. vBulletin allows you to be notified briefly by mail when something of interest appeared on the forum. There are a few disadvantages to moving this discussion now: (a)Quite a bit has transpired and people might feel they need to restate things in the forum [that could be an advantage....]; (b)we are new to this software and might not administer it as seamlessly as a mail-based system. So as not to clutter the list. I would urge people to /privately/ reply to me with their sympathy for or against the issue. I'll summarize and post the results and then people can discuss it, if warranted. Send mail to ram(a)cs.umb.edu. Your "Reply" facility will likely send to the list. For a glimpse of how vBulletin forums operate, see those at http://vbulletin.com/forum/index.php http://forums.progresstalk.com/index.php http://www.linuxquestions.org/questions/ http://www.ntcompatible.com/vb/index.php http://www.webdevforums.com/ As far as I can tell, vBulletin is the software under the forum parts of sourceforge.net Bob Morris p.s. Forum software is aimed at community building, so vBuilder has lots of flashy features like smiley face annotations, personal avatars, polls about the quality of a thread, "buddy lists", etc. I don't expect that SDD forums would operate with that stuff in your face, especially as we gain administrative experience.

1 0

Re: Document vs. database
by Jim Croft 30 Nov '01

30 Nov '01

> In DELTA, Lucid etc we express alternate hierarchies in alternate documents. > Do we need to do better than this? Let's stop this line of thought right now... build one hierachy of characters that works first, THEN consider how to manage alternatives... maybe... jim

1 0

Re: Document vs. database
by Jim Croft 30 Nov '01

30 Nov '01

I have been watching the animated discussion on the revived SDD list and it is tood to see... too busy to comment, but still interested, and useful things are being said by others... :) > Existing data structures allow 2-level hierarchies e.g. > #1. Leaf shape/ > 1. ovate/ > 2. elliptic/ > #2. Flower colour/ > 1. blue/ > 2. red/ Is that really two levels of character is is it only one with a state attribure? Also, in our new model we also want to show more than just presence or absence of a state. Don't we want to show if it is present/rarely present/present by misinterpretation/rarely present by misisnterpretation, etc., and my favourites yet to be implemented as a character stae attribute: definitely absent, absent by misinterpretation, unknown, unscored > I'm simply suggesting allowing n levels: > <Leaves> > <shape> > <ovate> > <elliptic> > <Flowers> > <colour> > <blue> > <red> That is where we want to get to do, but echoing Bob's words, what we are after is the structure that allows us to get to that, giving people all the appearance of freedom to do what they like but not actually doing it and imposing a schematic straight jacket on the data. A schema for the data structure rather than the descriptive data itself. Like all freedoms, the descriptive data one too must be an illusion or we wil miss out on on all the creative potential that it offers as the free spirits among us do theri own thing. But that is another discussion... :) There has been a lot of talk about the feature/value paradigm and how this might be made to represent a biological description and even nested features in such descriptions... At the recent TDWG meeting Richard Pankhurst described something like a feature/character/value paradigm and at the time I made a note that this was probalably worth considering in more detail, but so far have not had the time. It probably would mean something like: <feature name="leaf" charecter="margin" value="serrate"/> or some equivalent using XML entities rather than attributes. Is there any merit in this approach above using a feature that is say "leaf margin" and another that is say "leaf tip" and yet another that is say "leaf base"? It would seem to give some structure to the character set data. On the surface it would also allow easy generation of more readable descriptions without excessive text processing: Leaf outline ovate, margin serrate, tip acute, base attenuate, etc. as opposed to: Leaf outline ovate, leaf margin serrate, leaf tip acute, leaf base attenuate, leaf... etc. Has anyone else considered this approach? jim

1 0

Re: Document vs. database
by Kevin Thiele 29 Nov '01

29 Nov '01

----- Original Message ----- From: "Eric Zurcher" <Eric.Zurcher(a)PI.CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Thursday, November 29, 2001 3:32 PM Subject: Document vs. database | At 02:15 PM 28-11-01 +1100, Kevin Thiele wrote: | >Issues | > | >I've taken the route of marking up a textual description, using a minimum of | >tags. It seems to me that a description comprises a series of features with | >values. I've used mixed markup because I wanted to have the mimimum tagging | >and make maximum use of the text. That is, in cases like: | > | ><Feature Name="Life form"><Value>shrub</Value></Feature> | > | >I've made use of the fact that "shrub" occurs in the text by enclosing it in | ><Value> tags. In cases like: | > | ><Feature><Name>Leaves</Name> | > | >I'm using a <Name> tag rather than a Name attribute | > | >This may make processing considerably more difficult, but it seems to me | >that there are some advantages. An alternative would be to first process the | >description into a DELTA-like structure then mark up that. I chose not to go | >that route because I still have the dream that one day we'll have a method | >of semi-automatically marking up the content of natural language into | >something like the above. The other advantage of this structure I believe is | >that it maintains all the complexity and nuance of the original natural | >language (which can be retrieved merely by removing all the tags). I prefer | >this to the DELTA-process (take a natural-language description -> atomise | >it -> reconstruct semi-natural-language by putting the atoms back together | >again in some fashion). | | My position on this is rather different, and I would prefer to see | characters defined a bit more rigorously. | | As an example, suppose that the description of one taxon is given as: | | <Feature Name="Life form"><Value>shrub</Value></Feature> | | A second as: | | <Feature Name="Life form"><Value>bush</Value></Feature> | | And a third as: | | <Feature Name="Life form"><Value>small woody plant</Value></Feature> | | And perhaps a fourth as: | | <Feature Name="Growth form"><Value>shrub</Value></Feature> | | Are these equivalent? There's certainly no obvious way to associate them, | aside from knowing that "bush" and "shrub" mean pretty much the same thing | in some dialects of English. I think the goal of trying to extract meaning | by inserting a few tags into existing textual descriptions relies too | heavily on a level of artificial intelligence that we're still unlikely to | see for a couple of decades. Good point, and there will clearly be a need for some type of processing somewhere along the line to catch and allow validation of these situations. But this is a processing issue, not a structural one. There's nothing in the DELTA data structure to prevent #1. Life form/ 1. shrub/ 2. bush/ 3. small woody plant/ #2. Growth form/ 1. shrub/ 2. smallish twiggy plant/ and I argue there should be nothing in the structure of the SDD standard to prevent it either. People simply need to be aware of the issue. But the SDD does, I agree, need to be structured in such a way that validation of these situations is made possible. The first-cut structure that I proposed does not allow much eyeball validation of this type - is this a problem? | But pondering on this made me think about why Kevin and I often disagree on | some of these issues. If I may greatly over-simplify our viewpoints, his | view is (I think) that a formal description ought to be a document from | which data can be extracted; whereas I tend of view a formal description | more as a database from which text can be generated. | | So what ARE we after: the free-flowing flexibility of a document, or the | rigour and precision of a database? Fortunately, there is a fair bit of | middle ground. It's worth noting that XML was developed as a markup | language for documents, but that it's primary usage to date has been as a | sort of portable and relatively light-weight data container. Still I think | that finding the right balance between flexibility and rigour is going to | be a major challenge in this exercise. My view I suppose is that part of the power of XML (something that has never been possible in the past) is that it allows precise retrieval of atomised data from a fairly free-form document. XML blurs the distinction between database and text. Before XML a text string like: Rigid, spreading shrub to c. 1m high and wide; stems glabrous. Leaves soon deciduous..... was highly intractable to computer processing. This intractability required us to manually and substantially restructure into e.g. a DELTA file before we could really do much with it. But there are problems I think with the DELTA process vis-a-vis natural language. In the relatively common case that we begin with a natural language description, the process is: natural language ----(1)-----> DELTA -------(2)-------->natural language (etc) where (1) is human processing and (2) is largely computer (CONFOR) processing. Two related problems here. Firstly, I often don't much like the output of CONFOR (some people say we humans should put up with the limitations computers force on us, but call me old-fashioned , I rebel at that {see also Asimov's Second Law of Robotics}). The second problem is that step 2 is unidirectional - if I edit the output natural language I break the process (the edits are volatile, next time I invoke CONFOR they get overwritten). So I'm trying to explore the new possibilities opened up by XML to produce a minimally restructured but highly parsable document. I'm happy to admit that I may be off-beam here. I just reckon we shouldn't discount the possibility. | >Features are nested: | > | ><Feature><Name>Leaves</Name> | > <Feature><Name>Shape</Name></Feature> | ></Feature> | > | >Is this allowable? In XML-Spy I can create a Schema OK for this document. | >It's also well-formed. | | It's certainly allowable in XML. Is it desirable in a description? | Possibly, but I think it relates to my question from yesterday about | hierarchies of characters ought to be expressed. Nesting is a very good way | to express a single hierarchy, but not for handling alternative | hierarchies. Which do people want? Existing data structures allow 2-level hierarchies e.g. #1. Leaf shape/ 1. ovate/ 2. elliptic/ #2. Flower colour/ 1. blue/ 2. red/ I'm simply suggesting allowing n levels: <Leaves> <shape> <ovate> <elliptic> <Flowers> <colour> <blue> <red> I can't decide if this a trivial or fundamental difference. | Nesting is a very good way | to express a single hierarchy, but not for handling alternative | hierarchies. In DELTA, Lucid etc we express alternate hierarchies in alternate documents. Do we need to do better than this? Cheers - k

1 0

Re: Taxonomic hierarchy in SDD
by Kevin Thiele 29 Nov '01

29 Nov '01

I agree with all this Eric :-) | While canalization is a risk, there certainly is also merit in trying to | retain the best features of "prior art". The DELTA format has persisted for | over 20 years, which is nearly an eternity in the IT field. It must have | been doing one or two things right. There's no need to follow it too | closely, but certainly we can learn quite a bit from DELTA about what works | well and what doesn't. No-one has suggested ignoring prior art, be it DELTA or any of the other programs. I think we have agreed several times to start with +/- a blank slate, learning as you say quite a bit from DELTA, Lucid etc about what works and what doesn't along the way. | >2. Of all the descriptions in the world, 99.9999999% of them are not in | >DELTA. Probably 99.999% of them are textual (natural language) descriptions. | >Surely this should form the basis of our first challenge, methinks. | | I think that's actually a rather harsh assessment of DELTA's uptake. Let's | say there are about 4x10^6 known species. We'd like descriptions for them | all, along with descriptions of higher taxonomic levels (genera, families). | So the magnitude of total number of descriptions is about 10^7. I'd guess | that the number of taxa with DELTA descriptions is of the magnitude of | 10^5. So that means roughly 1% of all taxa already have a DELTA description. I didn't stop to consider the maths, for which my apologies. My point was simply that a vast minority of decriptions are in any type of standardised format at the present. If your estimations are correct (and I can't judge that) then there are still 99 non-DELTA descriptions for every DELTA one, or any other. | >3. Related to 2 above, many (though by no means all) DELTA datasets are | >already abstractions from the source (a set of natural language | >descriptions). We should start with the source. | | The source? Surely the ultimate source is observations made on individual | specimens. If we wish to start with the source, the first thing that needs | to be done is to make sure that we have a system which can adequately | describe a specimen in hand. Yes, you're right about the ultimate source, but there are chains of sources, of course. The reason why I'm interested in the vast legacy of textual descriptions as source material is that I think that effective semi-automated processing of these descriptions along the lines of Pankhurst and Taylor is not too far off (some processing is here already, if course). We should make this easier rather than harder, if at all possible. Capturing the truly ultimate direct observations is something that any Builder program supporting the new standard will have to be good at, of course. | (Sorry if I get too defensive about DELTA. I just can't help myself...) Please don't think that I'm too dismissive of DELTA. I just think we should be assuming that we can do a fair bit better. Cheers - k

1 0

Document vs. database
by Eric Zurcher 29 Nov '01

29 Nov '01

At 02:15 PM 28-11-01 +1100, Kevin Thiele wrote: >Issues > >I've taken the route of marking up a textual description, using a minimum of >tags. It seems to me that a description comprises a series of features with >values. I've used mixed markup because I wanted to have the mimimum tagging >and make maximum use of the text. That is, in cases like: > ><Feature Name="Life form"><Value>shrub</Value></Feature> > >I've made use of the fact that "shrub" occurs in the text by enclosing it in ><Value> tags. In cases like: > ><Feature><Name>Leaves</Name> > >I'm using a <Name> tag rather than a Name attribute > >This may make processing considerably more difficult, but it seems to me >that there are some advantages. An alternative would be to first process the >description into a DELTA-like structure then mark up that. I chose not to go >that route because I still have the dream that one day we'll have a method >of semi-automatically marking up the content of natural language into >something like the above. The other advantage of this structure I believe is >that it maintains all the complexity and nuance of the original natural >language (which can be retrieved merely by removing all the tags). I prefer >this to the DELTA-process (take a natural-language description -> atomise >it -> reconstruct semi-natural-language by putting the atoms back together >again in some fashion). My position on this is rather different, and I would prefer to see characters defined a bit more rigorously. As an example, suppose that the description of one taxon is given as: <Feature Name="Life form"><Value>shrub</Value></Feature> A second as: <Feature Name="Life form"><Value>bush</Value></Feature> And a third as: <Feature Name="Life form"><Value>small woody plant</Value></Feature> And perhaps a fourth as: <Feature Name="Growth form"><Value>shrub</Value></Feature> Are these equivalent? There's certainly no obvious way to associate them, aside from knowing that "bush" and "shrub" mean pretty much the same thing in some dialects of English. I think the goal of trying to extract meaning by inserting a few tags into existing textual descriptions relies too heavily on a level of artificial intelligence that we're still unlikely to see for a couple of decades. But pondering on this made me think about why Kevin and I often disagree on some of these issues. If I may greatly over-simplify our viewpoints, his view is (I think) that a formal description ought to be a document from which data can be extracted; whereas I tend of view a formal description more as a database from which text can be generated. So what ARE we after: the free-flowing flexibility of a document, or the rigour and precision of a database? Fortunately, there is a fair bit of middle ground. It's worth noting that XML was developed as a markup language for documents, but that it's primary usage to date has been as a sort of portable and relatively light-weight data container. Still I think that finding the right balance between flexibility and rigour is going to be a major challenge in this exercise. >Features are nested: > ><Feature><Name>Leaves</Name> > <Feature><Name>Shape</Name></Feature> ></Feature> > >Is this allowable? In XML-Spy I can create a Schema OK for this document. >It's also well-formed. It's certainly allowable in XML. Is it desirable in a description? Possibly, but I think it relates to my question from yesterday about hierarchies of characters ought to be expressed. Nesting is a very good way to express a single hierarchy, but not for handling alternative hierarchies. Which do people want? Eric Zurcher CSIRO Livestock Industries Canberra, ACT Australia E-mail: Eric.Zurcher(a)pi.csiro.au

1 0

Re: Taxonomic hierarchy in SDD
by Eric Zurcher 29 Nov '01

29 Nov '01

At 12:38 PM 28-11-01 +1100, Kevin Thiele wrote: >Concerning Steve's exemplar drawn from a DELTA butterfly treatment: I've >added a few challenges to the challenge cases (number 9 in the attached) >specifically designed to address representation of DELTA datasets in the new >standard. Personally, I don't think we should start with a DELTA data set as >our first challenge, for several reasons: > >1. We have agreed several times that it's important not to be canalized by >DELTA or any other existing representation, but to keep the existing >representations in mind while working. Starting with XDELTA seems to me to >increase the risk of canalization While canalization is a risk, there certainly is also merit in trying to retain the best features of "prior art". The DELTA format has persisted for over 20 years, which is nearly an eternity in the IT field. It must have been doing one or two things right. There's no need to follow it too closely, but certainly we can learn quite a bit from DELTA about what works well and what doesn't. >2. Of all the descriptions in the world, 99.9999999% of them are not in >DELTA. Probably 99.999% of them are textual (natural language) descriptions. >Surely this should form the basis of our first challenge, methinks. I think that's actually a rather harsh assessment of DELTA's uptake. Let's say there are about 4x10^6 known species. We'd like descriptions for them all, along with descriptions of higher taxonomic levels (genera, families). So the magnitude of total number of descriptions is about 10^7. I'd guess that the number of taxa with DELTA descriptions is of the magnitude of 10^5. So that means roughly 1% of all taxa already have a DELTA description. >3. Related to 2 above, many (though by no means all) DELTA datasets are >already abstractions from the source (a set of natural language >descriptions). We should start with the source. The source? Surely the ultimate source is observations made on individual specimens. If we wish to start with the source, the first thing that needs to be done is to make sure that we have a system which can adequately describe a specimen in hand. (Sorry if I get too defensive about DELTA. I just can't help myself...) Eric Zurcher CSIRO Livestock Industries Canberra, ACT Australia E-mail: Eric.Zurcher(a)pi.csiro.au

1 0