November 2001 - tdwg-content

Re: Modelling and XML
by Guillaume Rousse 23 Nov '01

23 Nov '01

Ainsi parlait Leigh Dodds : [..] > I'll also interject 2p/euro/cents at this point: > > I believe I'd tackle this problem firstly from a modelling perspective: > can we agree upon, and define a model for the data that we're > capturing. It seems like there is a conceptual model inherent in the > data currently captured by DELTA and other formats, that is > separate to the details of its syntax. I wholefully agree there: let's define first a conceptual model, then we should discuss of its implementation. However, we must agree on model extend: will it concerns only concept description (aka: characters) or also case description (aka: items) ? IMHO, only the first one can be generalized, or we'll have to validate the case description twice: against a generic model and against its concept. For using an exemple, if i have a description of the characters of Pociloporidae familiy, and a description of the items of Pociloporidae family, i'll have to make sure characters are really characters (validating against generic character model), to make sure items are really items (validating against generic items model), and make sure Pociloporidae items are really Pociloporida (validating items against characters). I would prefer to have only to validate characters againt a generic character model, and validate items against characters, meaning using a character description as a suitable model for items description. -- Guillaume Rousse <rousse(a)ccr.jussieu.fr> GPG key http://lis.snv.jussieu.fr/~rousse/gpgkey.html

1 0

Re: Morphological Data Representation
by Guillaume Rousse 23 Nov '01

23 Nov '01

Ainsi parlait Steve Shattuck : [..] > I've translated this into the XML file that is attached. (Even this fairly > simple example is moderately large and I would recommend using an XML > viewer such as Microsoft's XML Notepad when working with it - see > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnxml/html >/ xmlpaddownload.asp.) I'm always surprised seeing people recommend using proprietary stuff, when they are plenty of free-software equivalents: http://freshmeat.net/search/?site=Freshmeat&q=xml+editor&section=projects sourceforge.net also also provide a complete list vim and emacs also have xml support, and are available on many platforms > The basic structure of the attached file is: > > <Morphology> > <Characters> > <Character> - ANY NUMBER > <Type> > <ID> > <Short_Descriptor> > <Long_Descriptor> > <State_Descriptor> > <Order> > <State> - ANY NUMBER > <Value> > <ID> > <Order> > <Items> > <Item> - ANY NUMBER > <Type> > <ID> > <Descriptor> > <CodedCharacter> - ANY NUMBER > <CharacterID> > <Description> > <CodedState> - ANY NUMBER > <StateID> > <Value> This is a mixed approach, whereas Gregor's proposition is to separate characters description and items. Morevoer, it's closely related to delta (as xdelta), whereas everyone seems to agree on building something from scratch. > Note that I've treated everything as elements and haven't used attributes. > Simplicity is the only reason for this and some elements would be better as > attributes; these can be converted when the dust settles. No, there is also a performance argument for attributes: as they make files less verbose, parsing is quicker. See xerces-j/xalan-j FAQ. However, from a modeling point-of-view, the rule should be: use elements for what is has real-world meaning use attributes for modeling artifacts (id, idrefs, etc...) Default XSLT transformation enforces this by outputting elements contents and ignoring attributes. -- Guillaume Rousse <rousse(a)ccr.jussieu.fr> GPG key http://lis.snv.jussieu.fr/~rousse/gpgkey.html

1 0

Morphological Data Representation
by unknown＠example.com 23 Nov '01

23 Nov '01

Below and attached are a first attempt at representing simple, common DELTA-type data in an XML-based structure. I've used a selected set of characters and items from the Butterfly sample data on the DELTA web site. The DELTA-formatted data looks like this: ==== DELTA-Standard CHARS File ==== *SHOW: Lepidoptera demonstration characters. Revised 28-AUG-91. *CHARACTER LIST #1. main colour of inner part of front wing/ 1. white/ 2. cream/ 3. grey/ 4. brown/ 5. black/ 6. yellow/ 7. orange/ 8. blue/ 9. green/ #2. wings <transparency>/ 1. with transparent areas/ 2. without transparent areas/ #3. length of front wing/ mm/ #4. antennae <length>/ times length of front wing/ ==== DELTA-Standard ITEMS File ==== *SHOW: Lepidoptera demonstration items. Revised 18-OCT-94. *ITEM DESCRIPTIONS # Antheraea/ 1,4 2,2 3,43-50 4,0.15-0.2 # Ethmia/ 1,2-4 2,2 3,11-14 4,0.6-0.65 # Graphium/ 1,1-2/9 2,2 3,29-33 4,0.45-0.5 # Hecatesia/ 1,4 2,1<small, translucent window>/2 3,11-14 4,0.8-0.9 ==== DELTA-Standard SPECS File ==== *SHOW: Lepidoptera demonstration specifications. Revised 28-AUG-91. *NUMBER OF CHARACTERS 4 *MAXIMUM NUMBER OF STATES 9 *MAXIMUM NUMBER OF ITEMS 4 *CHARACTER TYPES 3,RN 4,RN *NUMBERS OF STATES 1,9 ==== For these files, the DELTA-generated natural language would look something like this: Antheraea Main colour of inner part of front wing brown. Wings without transparent areas. Length of front wing 43-50 mm. Antennae 0.15-0.2 times length of front wing. Ethmia Main colour of inner part of front wing cream to brown. Wings without transparent areas. Length of front wing 11-14 mm. Antennae 0.6-0.65 times length of front wing. Graphium Main colour of inner part of front wing white to cream, or green. Wings without transparent areas. Length of front wing 29-33 mm. Antennae 0.45-0.5 times length of front wing. Hecatesia Main colour of inner part of front wing brown. Wings with transparent areas (small, translucent window), or without transparent areas. Length of front wing 11-14 mm. Antennae 0.8-0.9 times length of front wing. ==== Hand-generated natural language would be essentially the same except for the last item, where it might look more like this: Hecatesia Main colour of inner part of front wing brown. Wings with or without transparent areas (when present, forming a small window). Length of front wing 11-14 mm. Antennae 0.8-0.9 times length of front wing. ============================================ I've translated this into the XML file that is attached. (Even this fairly simple example is moderately large and I would recommend using an XML viewer such as Microsoft's XML Notepad when working with it - see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnxml/html/ xmlpaddownload.asp.) The basic structure of the attached file is: <Morphology> <Characters> <Character> - ANY NUMBER <Type> <ID> <Short_Descriptor> <Long_Descriptor> <State_Descriptor> <Order> <State> - ANY NUMBER <Value> <ID> <Order> <Items> <Item> - ANY NUMBER <Type> <ID> <Descriptor> <CodedCharacter> - ANY NUMBER <CharacterID> <Description> <CodedState> - ANY NUMBER <StateID> <Value> Note that I've treated everything as elements and haven't used attributes. Simplicity is the only reason for this and some elements would be better as attributes; these can be converted when the dust settles. I've tried to generalise as much as possible and use only two main elements: <character> and <item>. Each character is assigned a <type> that tells what it is (ordered multistate, unordered multistate, real, integer, etc). Similarly the item <type> can be specified as taxon or specimen (or potentially something else). A couple of points are probably worth making: The <Short_Descriptor> and <Long_Descriptor> are used to support DELTA comments. This probably needs to be generalised further to support any number of alternate phrasings. <State_Descriptor> is used for the units of numeric characters and isn't needed (?) for other character types - it's an attempt at keeping <character> general. The <Description> element in <CodedCharacter> is used to house natural language representations. Codes for the states (when needed) are placed in square brackets, these being translated during generation. As noted above, this may been to be generalised to support any number of phrasings. In <CodedState>, <Value> is used to hold numeric values, the <character> <state> being used to define what the number means (minimum, maximum, etc., rather than using placement in the attribute string as in the DELTA standard). This element won't (?) be needed for multistate characters. I think/hope the remainder is fairly clear. ** The Next Step ** I would suggest the following path from here: 1) Make sure the above representation makes sense for the data given. 2) Expand the above data to support LucID-specific requirements (without adding additional complexity). Once this is finished we can: Add additional DELTA features (dependencies, default values, etc.) Add more complex data sets and examples Add new features on our assorted "wish lists" I look forward to comments and forward progress! Thanks, Steve Steve Shattuck CSIRO Entomology biolink(a)ento.csiro.au

1 0

Re: is there an "xml-include"
by unknown＠example.com 22 Nov '01

22 Nov '01

There seems to be a number of problems being addressed in the current discussion: 1) Gregor's original post asked for a solution ("how do I do an include in XML") without posing a problem ("how do I maintain a separate global character list and link a number of independent taxon descriptions to it"). I would suggest that using an "include" is only one of a range of solutions to this problem. And if this isn't the problem then we should revisit it before spending too much time coming up with a solution. 2) Whether XML is hierarchical or relational is of little importance in practice. All major relational databases can suck in and spit out XML - it's a non-issue from an implementation standpoint (which is what we are ultimately concerned with here). IDs and RefIDs are primary and foreign keys - the mapping is one-to-one if you want to structure the XML that way. 3) Leigh is spot-on when he says "define a model for the data" as this is "separate to the details of its [the model's] syntax". The syntax (= XML model) will be useless if it doesn't manage the information that's important to us no matter how "rigorous" or "technically accurate" it is. 4) At the Sydney meeting I thought we agreed to restart this discussion with specific examples based on real-life characters (in the biological sense). This current thread would seem to be very much a continuation of the one started 2 years ago: 80% focus on solutions (largely round XML syntax) and 20% on specific problems (addressing business needs). I fear we'll end up the same place we were before. Thanks, Steve Steve Shattuck CSIRO Entomology biolink(a)ento.csiro.au P.S. - A solution to Gregor's problem would be to use unique identifiers (GUIDs) in his global character list. Any one using this list would then include these identifiers as part of taxon descriptions along with metadata (a citation) to find where the global list is housed. If you then want to combine separate description datasets you can compare the character identifiers: if they are the same then it's the same character, if they are different then it's not the same character. It's a simple, foolproof solution that's easy to implement technically (whether people will actually do this is a completely different matter ;-) ). P.P.S. - I'm well aware that I have the power to direct this discussion by actually doing what I suggest in No. 4 above. And I really, really intend to do it. I just have one more meeting to get through, and then another 10 emails, and then ....

1 0

Re: is there an "xml-include"
by Robert A. (Bob) Morris 21 Nov '01

21 Nov '01

Steve Shattuck writes: > Date: Thu, 22 Nov 2001 10:41:55 +1100 >[...] > 4) At the Sydney meeting I thought we agreed to restart this discussion with > specific examples based on real-life characters (in the biological sense). > This current thread would seem to be very much a continuation of the one > started 2 years ago: 80% focus on solutions (largely round XML syntax) and > 20% on specific problems (addressing business needs). I fear we'll end up > the same place we were before. Amen. More explicitly, what was posed was to provide "data challenges", i.e. (as I understood) problems about characters that the poser feels important to model. Implicit (as I understood) in that discussion---and in almost all of this recent thread---is that a particular representation (e.g. XML) is NOT what is initially desired. If, indeed, my understandings were correct, and if a sufficiently inclusive set of challenges were identified, what would then be accomplished sets the stage exactly for the modeling Leigh urges. Further, this complies with one of the admonitions in Rob Atkinson's talk to first find out what you are trying to make a standard about, i.e.: <not> <adage>XML is the answer. What is the question?</adage> </not> Finally, I would paraphrase what I understood the subgroup meeting arrived at: The first job is to identify with these data challenges the strengths and the limitations of what DELTA-motivated applications can accomplish, rather than throw out the experience of DELTA and start over. Probably Gregor's meeting minutes will reveal whether that understanding is correct and perhaps how it came to be. --Bob Morris

1 0

Re: is there an "xml-include"
by Douglas Trainor 21 Nov '01

21 Nov '01

Amen. I'm not a taxonomist, but I suggest that several folks on this list take some of their current character data and make an example out of it in XML and comment on the example. It need not be perfect. This could quickly educate others about the strengths and potential weaknesses of an approach with a specific examplar. XML will make it very nice to move data between different applications, or between component parts of a large application. I just want to write nifty plant/insect applications against the XML-wrapped character data. Going right off for a perfect Holy Grail has sabotaged other group projects in the past. One of the best examples is how Algol language folks got into bitter debates about when an object is a chair and when it's a table. douglas Steve Shattuck wrote: > There seems to be a number of problems being addressed in the current > discussion: > > 1) Gregor's original post asked for a solution ("how do I do an include in > XML") without posing a problem ("how do I maintain a separate global > character list and link a number of independent taxon descriptions to it"). > I would suggest that using an "include" is only one of a range of solutions > to this problem. And if this isn't the problem then we should revisit it > before spending too much time coming up with a solution. > > 2) Whether XML is hierarchical or relational is of little importance in > practice. All major relational databases can suck in and spit out XML - > it's a non-issue from an implementation standpoint (which is what we are > ultimately concerned with here). IDs and RefIDs are primary and foreign > keys - the mapping is one-to-one if you want to structure the XML that way. > > 3) Leigh is spot-on when he says "define a model for the data" as this is > "separate to the details of its [the model's] syntax". The syntax (= XML > model) will be useless if it doesn't manage the information that's important > to us no matter how "rigorous" or "technically accurate" it is. > > 4) At the Sydney meeting I thought we agreed to restart this discussion with > specific examples based on real-life characters (in the biological sense). > This current thread would seem to be very much a continuation of the one > started 2 years ago: 80% focus on solutions (largely round XML syntax) and > 20% on specific problems (addressing business needs). I fear we'll end up > the same place we were before. > > Thanks, Steve > > Steve Shattuck > CSIRO Entomology > biolink(a)ento.csiro.au > > P.S. - A solution to Gregor's problem would be to use unique identifiers > (GUIDs) in his global character list. Any one using this list would then > include these identifiers as part of taxon descriptions along with metadata > (a citation) to find where the global list is housed. If you then want to > combine separate description datasets you can compare the character > identifiers: if they are the same then it's the same character, if they are > different then it's not the same character. It's a simple, foolproof > solution that's easy to implement technically (whether people will actually > do this is a completely different matter ;-) ). > > P.P.S. - I'm well aware that I have the power to direct this discussion by > actually doing what I suggest in No. 4 above. And I really, really intend > to do it. I just have one more meeting to get through, and then another 10 > emails, and then ....

1 0

Re: is there an "xml-include"
by Leigh Dodds 21 Nov '01

21 Nov '01

> I think this comes closest. Can you point me to information on > XInclude? As you say, it seems unclear whether a validating parser, > say msxml 4.0 would be able to check such id/idrefs. http://www.w3.org/TR/xinclude XInclude in Java: http://www.ibiblio.org/xml/XInclude/ Cheers, L.

1 0

Modelling and XML
by Leigh Dodds 21 Nov '01

21 Nov '01

As an outside, but interested, observer I'm keen to contribute to this discussion in any way that seems useful. E.g. technical help with XML. Judging by the recent discussion there is progress being made on this front. Is there any publically available documentation? I'll also interject 2p/euro/cents at this point: I believe I'd tackle this problem firstly from a modelling perspective: can we agree upon, and define a model for the data that we're capturing. It seems like there is a conceptual model inherent in the data currently captured by DELTA and other formats, that is separate to the details of its syntax. Having agreed a particular model, a suitable syntax for expressing that model would be the logical next step. XML seems like a good fit for this, and, as I've noted elsewhere, can serialise graphs of information. Cheers, L. -- Leigh Dodds, Research Group, Ingenta | "Pluralitas non est ponenda http://weblogs.userland.com/eclectic | sine necessitate" http://www.xml.com/pub/xmldeviant | -- William of Ockham

1 0

Re: is there an "xml-include"
by Leigh Dodds 21 Nov '01

21 Nov '01

At risk of distracting from the discussion at hand: > If you feel that descriptive data can only be modeled with > relations, then you can never get XML to work for you. It is not relational, it is strictly > hierarcical. It does not and, by design, cannot support the relational model. Do not > confuse XML with a database. It is -not- designed to be a database. The few databases that > use XML are all hierarcical XML is designed to model the structure of > documents. If you want to create a database, use a database > language. XML is what you use in the report. Well strictly speaking XML can be used to model/serialise graph structures, so while it is hierarchical, and the syntax does require a single root, there's no reason why this can't become a graph (or other) structure once parsed. There's been a lot of activity (unsurprisingly) in serialising XML into and out of databases. In general it's not an easy problem to solve, but it can be done. Also, I'd have to disagree with the statement that 'the few databases that use XML are all hierarchical'. Current XML-aware databases fall into two categories: XML-Enabled Databases (i.e. relational databases that have added XML features) and Native XML Databases (i.e. those specifically designed to store XML). Databases in the first category do tend to be primarily relational in nature. Those in the second are a mixture: they model the XML document internally, and this might be achieved using a relational model. Cheers, L. -- Leigh Dodds, Research Group, Ingenta | "Pluralitas non est ponenda http://weblogs.userland.com/eclectic | sine necessitate" http://www.xml.com/pub/xmldeviant | -- William of Ockham

1 0

Re: is there an "xml-include"
by Robert A. (Bob) Morris 20 Nov '01

20 Nov '01

Leigh Dodds writes: > Date: Wed, 21 Nov 2001 09:15:02 -0000 > From: Leigh Dodds <ldodds(a)INGENTA.COM> > To: TDWG-SDD(a)usobi.org > Subject: Re: is there an "xml-include" > > At risk of distracting from the discussion at hand: > >[...] > Well strictly speaking XML can be used to model/serialise graph structures, > so while it is hierarchical, and the syntax does require a single root, > there's > no reason why this can't become a graph (or other) structure once parsed. > > There's been a lot of activity (unsurprisingly) in serialising XML into and > out of databases. In general it's not an easy problem to solve, but it > can be done. People interested in how easily it can always be done, but why it is hard to do it well in general, might consult S. Abiteboul, P. Buneman, D. Suciu, "Data on the Web. From Relations to Semistructured Data and XML", Morgan Kaufmann Publishers, 2000, especially pp. 172 ff.

1 0