September 2000 - tdwg-content

Validation
by Kevin Thiele 02 Sep '00

02 Sep '00

Eric, I'm not sure about a requirement for strong validation in the spec. I agree that validation should be allowed, but not required. Your example: | <STATEMENT> | <ITEM> | <ITEM_NAME> Gouania exilis </ITEM NAME> | </ITEM> | <ELEMENT> | <ELEMENT_NAME> Flour colur </ELEMENT_NAME> | </ELEMENT> | <VALUE> Puerto Rico </VALUE> | <QUALIFIER> anchovies, please! </QUALIFIER> | </STATEMENT> There are three types of validation you suggest here: 1. Validate the element strings (and presumably also items, values and qualifiers) against predefined lists to guard against typographical and spelling errors; 2. Validate values against a list of allowable values for the character; 3. Validate qualifiers against a list of allowable qualifiers. Taking these in reverse order: 3. As I see it the spec itself would define a set of allowable qualifiers (such as "rare" (= "rarely"?), "by misinterpretation", "uncertain" etc). I think we could probably agree on a limited set of qualifiers, and stick to that (with allowance for extension). If we do this, then "anchovies, please!" will be out for several reasons. 2. Validation of allowable values is covered in the draft spec. One of the properties of an element (~"character") is a list of allowable values (~"states"). If such a property is filled, then validation can be done by the program that handles the data, just as in DELTA and Lucid. Two notes: * I'd like to allow but not enforce that a valid document have the allowable_values property filled. By not enforcing it, a simply marked-up natural-language description could be a valid document. This would perhaps mean that the spec could meet half-way the automated markup of legacy descriptions, and I'm keen to do this. Of course, a document without this property specified would not be able to be validated in the way you suggest and hence may not be very useful for some purposes, but this may be a price one is willing to pay for other benefits, and I think we need to keep this open. * I'm using "allowable values" rather than "states" as this seems to me to be more general, and subsumes the requirement to distinguish between, for instance, multistate and numeric "characters". A numeric "character" of course, doesn't have "states", but it does have allowable values (integers in the case of an integer numeric, real numbers in the case of a real numeric). 1. How strong is the requirement for this type of validation? Enforcing this seems to me to be like requiring that all word documents carry in their header a dictionary of the english language to allow validation of spellings. It seems to me that providing tools that allow people to check these strings against a predefined list (defined either within the document or in an external resource) would be useful, but not absolutely necessary. A document that is not or cannot be validated in this way would not be useless, and would perhaps be more free. Note that the spec as I see it would allow (but again, not enforce as DELTA and Lucid do) the encoding of descriptions. Thus, a valid document may be d1 as below. This would preempt the need for typographic validation, and allow allowable-values validation. But for some reason I don't want to disallow d2 as a valid document also. <DOCUMENT ID = "d1" Name = "Treatment of Gouania"> <ITEM_PROPERTIES> <ITEM ID = "1" NAME = "Gouania exilis"/> <ITEM ID = "2" NAME = "Gouania australiana"/> </ITEM_PROPERTIES> <ELEMENT_PROPERTIES> <ELEMENT ID = "1" > <ELEMENT_NAME> Flower colour <\ELEMENT NAME> <VALUE_LIST> <VALUE ID = "1"> <VALUE_NAME> "green" </VALUE_NAME> </VALUE> <VALUE ID = "2"> <VALUE_NAME> "yellow" </VALUE_NAME> </VALUE> </VALUE_LIST> </ELEMENT> <ELEMENT ID = "2" > <ELEMENT_NAME> Petal shape <\ELEMENT NAME> <VALUE_LIST> <VALUE ID = "1"> <VALUE_NAME> "ovate" </VALUE_NAME> </VALUE> <VALUE ID = "2"> <VALUE_NAME> "obovate" </VALUE_NAME> </VALUE> </VALUE_LIST> </ELEMENT> </ELEMENT_PROPERTIES> <COLLECTION Item_ID = "1"> <STATEMENT> <ELEMENT Element_ID="1/> <VALUE Value_ID = "1"/> </STATEMENT> <STATEMENT> <ELEMENT Element_ID="2/> <VALUE Value_ID = "1"/> </STATEMENT> <STATEMENT> <ELEMENT Element_ID="2/> <VALUE Value_ID = "2"/> </STATEMENT> </COLLECTION> </DOCUMENT> <DOCUMENT ID = "d2" Name = "One thing I know about a rose"> <STATEMENT> <ITEM> <ITEM_NAME> Rose </ITEM NAME> </ITEM> <ELEMENT> <ELEMENT_NAME> Smell </ELEMENT_NAME> </ELEMENT> <VALUE> Sweet </VALUE> </STATEMENT> <\DOCUMENT> Cheers - k

1 0

Re: Draft Spec mark 2
by Kevin Thiele 02 Sep '00

02 Sep '00

>>>From Eric's comments: | Kevin has offered us not just 1, but 4, different document models for | comment. That's a lot to consider, and I won't attempt to offer detailed | criticism on each alternative. But from my point of view, the first several | appear far too "weak" to be of much use. These are four (very early) examples, not four models. My intention is to develop a specification that can allow data represented with varying levels of complexity, from very simple statements to fully worked up integral data files such as Lucid and DELTA use. I'm not sure why, but I like the idea that the most basic document would simply say "Eric Zurcher has two feet". Perhaps current programs wouldn't be able to make much use of such a document, but maybe in the future we may have a way of collecting and organising clouds of such basic scraps (atoms) of description into something quite powerful. | In particular, I think it is | essential that there be a mechanism for specifying a universe of the | character and state values (or, mapping roughly from Kevin's terminology, | "elements" and "values", respectively) which may appear within a | description. That is, we need to be able to define (or, rather, provide a | mechanism for the dataset designer to define) the equivalent of a DELTA | "character list". This is needed to enforce consistency and to disallow | nonsensical constructs. For example, a portion of Document 1 reads: | | <STATEMENT> | <ITEM> | <ITEM_NAME> Gouania exilis </ITEM NAME> | </ITEM> | <ELEMENT> | <ELEMENT_NAME> Flower colour </ELEMENT_NAME> | </ELEMENT> | <VALUE> green </VALUE> | <QUALIFIER> rarely </QUALIFIER> | </STATEMENT> | | There needs to be a way to preclude a nonsensical entry like: | | <STATEMENT> | <ITEM> | <ITEM_NAME> Gouania exilis </ITEM NAME> | </ITEM> | <ELEMENT> | <ELEMENT_NAME> Flour colur </ELEMENT_NAME> | </ELEMENT> | <VALUE> Puerto Rico </VALUE> | <QUALIFIER> anchovies, please! </QUALIFIER> | </STATEMENT> Yes, I agree. Validation rules need to be built in. I think I said that I was putting this up for development, not as a finished product. Perhaps another reason for having character & taxon lists at the head of the document is to allow an agent that visits the document to determine which taxa and characters it's about without having to parse the whole thing? | I might also note that I strongly question the way the above is organized. | I think a rearrangement better expressing the relationships (but still not | really addressing the problem of a lack of meaningful validation) would be | more along the lines of: | | <ITEM> | <ITEM_NAME> Gouania exilis </ITEM NAME> | <ELEMENT> | <ELEMENT_NAME> Flower colour </ELEMENT_NAME> | <VALUE> green | <QUALIFIER> rarely </QUALIFIER> | </VALUE> | </ELEMENT> | </ITEM> This is perhaps a subtle point, perhaps a trivial one, and I'm glad you raised it. Why is this better that the above? What exactly is the advantage? I know yours is more DELTA-like and I agree it seems more intuitively correct, but is there more to your strong belief than that? In truth, I played with different ways of structuring unitary statements and presented the one I did to draw comment rather than just follow down the path of what we do already. | And I'd be inclined to make a bit more use of attributes, rather than | element content, though as Bob Morris points out, that is largely (though | not entirely) a matter of stylistic convention. See my general comment about true- and pseudo-XML | I don't really see any great difficulties is implementing some sort of | "character list" within the XML syntax. "Document 4" appears to be heading | in that direction, but doesn't (in my opinion) go quite far enough. I agree | What I think we need is a Schema definition that would allow a validating parser | to detect obvious data errors, and assist editing software in enforcing | "correctness" of the data. I've recently been looking through the | description of XML Schema, and it seems to have the expressive power needed | for this sort of thing. Very likely. My opinion is that we should work out our data requirements (one of which is the need for validation as you properly point out) and then determine the best vehicle to carry the data. Cheers - k

1 0

Re: Draft Spec mark 2
by Kevin Thiele 02 Sep '00

02 Sep '00

Following several comments from the list: Any and all comments about the syntax in my document and examples: I SHOULD have said that I'm not trying to make this "proper" XML yet. My intention is just to determine the data elements needed - it seems to me that once we've done that the best-practice XML (e.g. should this data element be content or attribute) can be done by you people with the knowledge to do it properly. So please read anything that looks like XML in the examples as pseudo-XML. Perhaps there's a better way of writing the examples to make it clear they're not supposed to be XML yet. I'm sure I'm going to regret not having done this. Cheers - k

1 0

Re: Philosophy; Was: Re: Draft Spec mark 2
by Kevin Thiele 02 Sep '00

02 Sep '00

| Bob Morris | | p.s. After about a year of hanging out with a variety of biologists, | I've come to think of the taxonomists as the of grammarians | and etymologists (not /entomologists/) of Life: Another way of looking at it: It's quite possible that the ability to construct (determine?) the etymology of Life is the best definer for the human condition - tool-using is no good, language is no good, even logic is no good, because animals do all these things. But there's no indication that a chimpanzee in a forest has a TAXONOMY of the forest (though they may have representations and tags in their minds for things that are good to eat, something quite different). If this is correct, then perhaps taxonomists are the highest form of organic life. |Taxonomists, God bless 'em. You can't live with 'em and you can't |live without 'em.

1 0

Re: Draft Spec mark 2
by Jim Croft 01 Sep '00

01 Sep '00

> >Plant descriptive data is much to complex to be captured in XML which is > >more geared towards presentation than representation. > >I'll second that. I use LaTeX, and previously used a precursor to SGML, >to typeset all my printed documents. These *typesetting* languages are >great for defining the appearance of the content of documents, but they >are utterly inadequate for representing the content (ie., data) itself. This is not my understanding at all... I thought the theory behind XML was to describe content, structure and relationships of data elements, quite independently of presentation as with typesetting languages... Having defined an issue with XML, or similar, you can present it however you want... Having defined an issue with a typesetting language, all you can do is typeset it... jim

1 0

Re: Draft Spec mark 2
by Eric Zurcher 01 Sep '00

01 Sep '00

Kevin has offered us not just 1, but 4, different document models for comment. That's a lot to consider, and I won't attempt to offer detailed criticism on each alternative. But from my point of view, the first several appear far too "weak" to be of much use. In particular, I think it is essential that there be a mechanism for specifying a universe of the character and state values (or, mapping roughly from Kevin's terminology, "elements" and "values", respectively) which may appear within a description. That is, we need to be able to define (or, rather, provide a mechanism for the dataset designer to define) the equivalent of a DELTA "character list". This is needed to enforce consistency and to disallow nonsensical constructs. For example, a portion of Document 1 reads: <STATEMENT> <ITEM> <ITEM_NAME> Gouania exilis </ITEM NAME> </ITEM> <ELEMENT> <ELEMENT_NAME> Flower colour </ELEMENT_NAME> </ELEMENT> <VALUE> green </VALUE> <QUALIFIER> rarely </QUALIFIER> </STATEMENT> There needs to be a way to preclude a nonsensical entry like: <STATEMENT> <ITEM> <ITEM_NAME> Gouania exilis </ITEM NAME> </ITEM> <ELEMENT> <ELEMENT_NAME> Flour colur </ELEMENT_NAME> </ELEMENT> <VALUE> Puerto Rico </VALUE> <QUALIFIER> anchovies, please! </QUALIFIER> </STATEMENT> I might also note that I strongly question the way the above is organized. I think a rearrangement better expressing the relationships (but still not really addressing the problem of a lack of meaningful validation) would be more along the lines of: <ITEM> <ITEM_NAME> Gouania exilis </ITEM NAME> <ELEMENT> <ELEMENT_NAME> Flower colour </ELEMENT_NAME> <VALUE> green <QUALIFIER> rarely </QUALIFIER> </VALUE> </ELEMENT> </ITEM> And I'd be inclined to make a bit more use of attributes, rather than element content, though as Bob Morris points out, that is largely (though not entirely) a matter of stylistic convention. I don't really see any great difficulties is implementing some sort of "character list" within the XML syntax. "Document 4" appears to be heading in that direction, but doesn't (in my opinion) go quite far enough. What I think we need is a Schema definition that would allow a validating parser to detect obvious data errors, and assist editing software in enforcing "correctness" of the data. I've recently been looking through the description of XML Schema, and it seems to have the expressive power needed for this sort of thing. Cheers, Eric Zurcher CSIRO Division of Entomology Canberra, Australia E-mail: ericz(a)ento.csiro.au

1 0

Re: Draft Spec mark 2
by Leigh Dodds 01 Sep '00

01 Sep '00

> -----Original Message----- > From: TDWG - Structure of Descriptive Data [mailto:TDWG-SDD@usobi.org]On > Behalf Of Una Smith > Sent: 01 September 2000 13:18 > To: TDWG-SDD(a)usobi.org > Subject: Re: Draft Spec mark 2 > > > On Fri, 1 Sep 2000, Erik Westlin wrote: > > >Plant descriptive data is much to complex to be captured in XML which is > >more geared towards presentation than representation. > > I'll second that. I use LaTeX, and previously used a precursor to SGML, > to typeset all my printed documents. These *typesetting* languages are > great for defining the appearance of the content of documents, but they > are utterly inadequate for representing the content (ie., data) itself. XML is *not* a typesetting language. Its a meta-language for defining other languages, just as SGML was. It is *not* limited to describing documents, and its far from being a limited to describing layout. My own doodlings with XML and Taxonomic data [1] shows that the DELTA format can be expressed as XML without loss of information. Its probably too early to get mired in syntax discussions - at present it may be better to assume that all examples merely that - not formal proposals for how the serialised data should look. The model is the important issue at the moment. Once that is nearing completion, decisions on a serialisation syntax (XML or something else) can then be made in highlight of concrete examples and data. Personally speaking, I'd be surprised if there were aspects of a taxonomic data model which *couldn't* be adequately expressed in XML. At most I'd expect there to be some contortions in the syntax, rather than a complete inability to represent the data. The question is then, how much does the syntactic 'sugar' outweigh the network effect of embracing a standard like XML. My 2p. [1]. http://www.ldodds.com/delta/index.html L. -- Leigh Dodds, Systems Architect | "Pluralitas non est ponenda http://weblogs.userland.com/eclectic | sine necessitate" http://www.xml.com/pub/xmldeviant | -- William of Ockham

1 0

Re: Draft Spec mark 2
by Robert A. (Bob) Morris 01 Sep '00

01 Sep '00

If you care about the intricacies of the difference between programming languages and markup languages for document and data reprepresentation, see the ironically named "The Origin of (Document) Species", by Khare et al. at http://www.cs.caltech.edu/~adam/papers/www/origin-of-species.html Despite being one of the perpetrators, I'd observe we are wondering of the TDWG charter a little here....

1 0

Re: Draft Spec mark 2
by Una Smith 01 Sep '00

01 Sep '00

Una Smith wrote: >>I'll second that. I use LaTeX, and previously used a precursor to SGML, >>to typeset all my printed documents. These *typesetting* languages are >>great for defining the appearance of the content of documents, but they >>are utterly inadequate for representing the content (ie., data) itself. Jim Croft wrote: >This is not my understanding at all... > >I thought the theory behind XML was to describe content, structure and >relationships of data elements, quite independently of presentation as with >typesetting languages... > >Having defined an issue with XML, or similar, you can present it however >you want... > >Having defined an issue with a typesetting language, all you can do is >typeset it... Jim, I think your definition of a typesetting language is not the same as mine. If by "typesetting language" you mean RTF, then I would agree with you that XML is not a typesetting language. But I don't consider RTF to be a typesetting language because it cannot (to my knowledge) be used to write other languages. That's the important difference between RTF and XML (and LaTeX). LaTeX is a language that was designed for typesetting, and it is good at it: typesetting is relatively painless, using LaTeX. LaTeX can be (and has been) used to write simple text-based user interfaces, and to write new programs (yes, you can write LaTeX programs to write LaTeX programs, ad infinitum; you could also write LaTeX programs to write C programs, and vice versa). These other uses of LaTeX are quite contorted, though; such programs are not pleasant to write. GML was a markup language, like RTF; SGML and its derivatives, including XML, are (as I see it) typesetting languages. Sure, if we all agree with Leigh Dodd that we are just using XML for the convenience of it (though I don't personally find XML convenient), that's okay. But the discussion here keeps slipping into implementation issues and forgetting about the underlying abstractions. We could conduct this discussion in PERL code too. (Hey, why not?) Why not forget the codes (or pseudo codes) and use ordinary words that we all understand? *Then* decide which language (if any) to implement it in, *after* we all agree what "it" is. Una Smith una.smith(a)yale.edu Department of Ecology and Evolutionary Biology Yale University New Haven, CT 06520-8106

1 0

Re: Draft Spec mark 2
by Erik Westlin 01 Sep '00

01 Sep '00

Eric Zurcher wrote: > In my mind it's totally wrong to create a language for descriptions using XML. Instead create a dedicated language for descriptions - then construct a XML generator for this language if you like. I'm in computer science, my interrest in botany is just a small part time hobby But i hate to see a lot of energy go to waste and i would like to see a computer flora whith a deep knowledge i could interact with. I think you should consult some people in the knowledge representation field. Plant descriptive data is much to complex to be captured in XML which is more geared towards presentation than representation. ------------------------------------------------------------------------ Erik Westlin Manne Siegbahn Laboratory email: westlin(a)msi.se

1 0