September 2000 - tdwg-content

Re: Text characters in DELTA
by Donovan Sharp 05 Sep '00

05 Sep '00

Mike Dallwitz wrote: > > From: Jim Croft <jrc(a)ANBG.GOV.AU> > > To: TDWG-SDD(a)USOBI.ORG > > > The DELTA spec would probably allow you to include a complete description > > as a lump of text, but it would not be particularly useful to do so. > > Yes, you could do this. It would be slightly more useful to break the text > up into text characters. For example, > > #1. <synonymy>/ > #2. <comments on habit>/ > #3. <comments on leaves>/ > #4. <comments on flowers>/ > #5. <comments on distribution>/ > #6. <general comments and discussion>/ > > These could still be combined into a single description, but could also be > displayed and searched separately if necessary. > > What is usually done is to intersperse such text characters at the > appropriate places in a list of coded characters. They can then be used as > repositories for information that the author can't or doesn't want to code. > The information they contain is accessible in Intkey, and can be used in > much the same way as coded information. > > -- > > Mike Dallwitz > > CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia > Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 > Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/ This is also useful for the preparation of written descriptions or associated text to be displayed with in a key. Observations on a species can greatly improve species descriptions. Things like 'grows on red loams with a northerly aspect' or 'Similar to species A but with longer thingies and rounder whatsits'. These are things which would not normally be coded as part of the data but are useful to the end user, and it is the needs of the end user which are ultimately important, and for this reason it is useful. Im not a computer scientist, I just build keys so perhaps I see this in a different light, and one thing which concerns me is the amount of double handling (and worse) which creeps in with coding, preparation of associated text, common links etc. The standard needs to be able to incorporate all useful entities, including 'hard' data, comments, images, sounds and other media. At the moment we build a key, then we add images and other associated things, these things should be considered part of the data and included from the start. The standard should then allow the elements which are not included for number crunching to be formatted and presented to the end user in any desired fashion.

1 0

Re: Text characters in DELTA
by Susan B. Farmer 05 Sep '00

05 Sep '00

>> The DELTA spec would probably allow you to include a complete description >> as a lump of text, but it would not be particularly useful to do so. > >Yes, you could do this. It would be slightly more useful to break the text >up into text characters. For example, > >#1. <synonymy>/ >#2. <comments on habit>/ >#3. <comments on leaves>/ >#4. <comments on flowers>/ >#5. <comments on distribution>/ >#6. <general comments and discussion>/ > >These could still be combined into a single description, but could also be >displayed and searched separately if necessary. > >What is usually done is to intersperse such text characters at the >appropriate places in a list of coded characters. They can then be used as >repositories for information that the author can't or doesn't want to code. >The information they contain is accessible in Intkey, and can be used in >much the same way as coded information. > I've done this. I use it as a check -- If I want/need to reverify the information again ...; then I don't have to go back to the original source because it's included in the data set. Susan ----- Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Minimalism AND functionalism
by Kevin Thiele 05 Sep '00

05 Sep '00

>>>From Jim Croft, | >The standard I'm working towards ALLOWS this degree of formality, but | >doesn't ENFORCE it. | | Although I think I understand and like what we are trying to do I am having | a slight problem here... a standard that does not enforce itself, is not | really a standard, is it? No, I don't think that's what's happening. The question I'm trying to ask is: what are the REQUIRED elements of the standard (ie those elements that must be present for a document to be valid under the standard)? I'm arguing that there is merit in using a minimal set of required elements, and of course also incorporating in the standard a larger set of optional elements. We all do this all the time, e.g. the property PRIVATE NOTES - no-one would suggest that every document to be valid under the standard should include private notes, but it's handy having the option to include these. It seems to me, for instance, that a CHARACTER LIST element should be optional, not required. Sure, having a character list allows validation and a document would be more valuable (in some contexts) for its inclusion, but I see advantage in not enforcing its presence. So I'm proposing, as ground-zero, that the only absolute requirements are that a document have DESCRIPTION elements, that these have ELEMENT elements (terminology is unfortunate here) and that these have VALUEs. This is a long way down the track towards minimalism, I know, but I'm floating it for comment. So, as I've exampled before, I'd like something like: <DOCUMENT ID = "d2" Name = "One thing I know about a rose"> <DESCRIPTION> <ITEM> <ITEM_NAME> Rose </ITEM NAME> </ITEM> <ELEMENT> <ELEMENT_NAME> Smell </ELEMENT_NAME> </ELEMENT> <VALUE> Sweet </VALUE> </DESCRIPTION> </DOCUMENT> to be a valid document. It contains information in a structured way that means something, and the information can be parsed. Sure, this document couldn't be used direct as input to Lucid or DELTA, but does that make it worthless? I'm working at the moment towards a future generation of the Lucid Builder (and perhaps future DELTA Builders will be similar) that will indeed be able to collect and interactively integrate such structured bits of information. Maybe wacky, but why not? Cheers - k

1 0

DELTA cf SDD
by Kevin Thiele 05 Sep '00

05 Sep '00

>>>From Mike... >In DELTA, character values in a description can't be separated from the >'attribute' which contains them, without losing information. This is because >the order of the values, and the words 'or', 'and', and 'to' which may >connect them, are significant. Also, the position of a qualifier (before or >after a value) may be significant, and some qualifiers, e.g. 'reliability', >may apply to the attribute as a whole rather than to a particular value. > >Example: > > 5,3/3&1<when infected with virus>/<occasionally>2<@reliability 3> > > = Flowers red, or red and white (when infected with virus), or > occasionally yellow. > >This doesn't seem to be accommodated in the proposal - at least, it's not >spelled out. Courple of things: 1. This raises one of the problems of DELTA as pointed out by Gregor and others at the TDWG meeting,which is that <comments> are being overloaded with meaning. There are various types of things here that are masquerading as being similar, when they're not at all. * "<when infected by a virus>" is perhaps a true comment. In the above example, could this perhaps be covered by content? <DESCRIPTION Name = "a taxon or thing"> <ELEMENT> Name = "Flower colour"> <VALUE> red <QUALIFIER> rarely </QUALIFER> </VALUE> when infected by a virus </ELEMENT> </DESCRIPTION> Note that these comments are only used for natural-language descriptions (they're stripped out when DELTA translates to an interactive key etc). I don't see why the ordering of these true comments can't be accomodated by their position in the content with respect to the tags - since these are "trivial" comments in most contexts, this doesn't seem weak to me. * <occasionally> is a qualifier as defined in the draft spec (only I'm calling it "rarely" in line with Lucid's terminology). Lucid is currently the only program that uses this type of qualifier in identification, and it's very important. But it's not to be confused with the common-or-garden variety of comment as above. * <@reliability 3> is a command for a program. This should be made clear by the tag so that other programs can ignore it. 2. &. I've always been bothered by the use of & (and) vs / (or) in DELTA. Using Mike's example: 5,3/3&1 = "Flowers red, or red and white" The use of & is handy for codifying natural-language descriptions, but in other applications it's highly problematical. This is a homology problem and has been discussed at length in the cladistic literature e.g. *CHARACTERS #1. Dorsolateral appendages/ 1. arms/ 2. wings/ *ITEMS #Humans 1,1 #Birds 1,2 #Angels 1,1&2 The discovery of angels (having both arms and wings) would prove the non-homology of arms~wings, so the character as stated is false and should be restated "Arms present/absent" "Wings present/absent". Similarly, in Mike's example, the red&white broken colouring of virus-infected tulips is actually non-homologous with uniform red or uniform white and should be a separate character (or at least a separate state). So pending further discussion (which I'm sure will come) I propose that the standard should not support the & coding of DELTA. Over to you, Mike... :-) Cheers - k

1 0

The basic statement
by Kevin Thiele 05 Sep '00

05 Sep '00

----- Original Message ----- From: Eric Zurcher <ericz(a)ENTO.CSIRO.AU> To: <TDWG-SDD(a)USOBI.ORG> Sent: Monday, September 04, 2000 11:29 AM Subject: Re: Draft Spec mark 2 | At 08:13 2/09/2000 +1000, Kevin Thiele wrote: | >These are four (very early) examples, not four models. My intention is to | >develop a | >specification that can allow data represented with varying levels of | >complexity, from very simple statements to fully worked up integral data | >files such as Lucid and DELTA use. I'm not sure why, but I like the idea | >that the most basic document would simply say "Eric Zurcher has two feet". | >Perhaps current programs wouldn't be able to make much use of such a | >document, but maybe in the future we may have a way of collecting and | >organising clouds of such basic scraps (atoms) of description into something | >quite powerful. | | Well, perhaps. But I think that our successors will appreciate it if we | apply a bit of rigour and take care to describe our data with as much | clarity and lack of ambiguity as possible. I don't think they'd find the | statement "Eric Zurcher has two feet" to be particularly interesting or | even unambiguous. Is "two feet" an anatomical statement, or a measurement | (e.g., has two feet of figurative rope, by which he can easily hang himself | in a forum such as this), or both <G>? True, so long as rigour doesn't become a straightjacket, at least at this stage where we are still in explore mode. Personally, I think that if we merely XMLify the Lucid or DELTA data format, we may have failed our successors...:-( | >| I might also note that I strongly question the way the above is organized. | >| I think a rearrangement better expressing the relationships (but still not | >| really addressing the problem of a lack of meaningful validation) would be | >| more along the lines of: | >| | >| <ITEM> | >| <ITEM_NAME> Gouania exilis </ITEM NAME> | >| <ELEMENT> | >| <ELEMENT_NAME> Flower colour </ELEMENT_NAME> | >| <VALUE> green | >| <QUALIFIER> rarely </QUALIFIER> | >| </VALUE> | >| </ELEMENT> | >| </ITEM> | > | >This is perhaps a subtle point, perhaps a trivial one, and I'm glad you | >raised it. Why is this better that the above? What exactly is the advantage? | >I know yours is more DELTA-like and I agree it seems more intuitively | >correct, but is there more to your strong belief than that? In truth, I | >played with different ways of structuring unitary statements and presented | >the one I did to draw comment rather than just follow down the path of what | >we do already. | | I think there is a real difference in terms of a reduction in ambiguity. We | should generally make an effort to clearly associate modifiers with the | object that they are intended to modify. Suppose we have something like the | following: | | <ELEMENT> | <ELEMENT_NAME> leaf </ELEMENT_NAME> | <VALUE> | lobed margins | </VALUE> | <VALUE> | with spines | </VALUE> | <QUALIFIER> | rarely | </QUALIFER> | </ELEMENT> | | How should such a construct be interpretted? Do spines cover the leaf as a | whole, or are they confined to the lobed margins? Is it the spines which | are rarely present, or the lobed margins? Yes, I agree with all over this. OK, so it was a dumb idea... Are we agreed then that the basic unit of a description is a statement of the form <DESCRIPTION Name = "a taxon or thing"> <ELEMENT> Name = "a character or feature that the taxon or thing has"> <VALUE> The value that the element has in this case <QUALIFIER> One of a small list of defined qualifiers (e.g. "rarely") </QUALIFER> </VALUE> <VALUE>....</VALUE> </ELEMENT> <ELEMENT>...</ELEMENT> </DESCRIPTION> ..and that this structure (with suitable properties for elements etc see the draft spec) can capture any and all parts of a description? Cheers - k

1 0

Text characters in DELTA
by Mike Dallwitz 05 Sep '00

05 Sep '00

> From: Jim Croft <jrc(a)ANBG.GOV.AU> > To: TDWG-SDD(a)USOBI.ORG > The DELTA spec would probably allow you to include a complete description > as a lump of text, but it would not be particularly useful to do so. Yes, you could do this. It would be slightly more useful to break the text up into text characters. For example, #1. <synonymy>/ #2. <comments on habit>/ #3. <comments on leaves>/ #4. <comments on flowers>/ #5. <comments on distribution>/ #6. <general comments and discussion>/ These could still be combined into a single description, but could also be displayed and searched separately if necessary. What is usually done is to intersperse such text characters at the appropriate places in a list of coded characters. They can then be used as repositories for information that the author can't or doesn't want to code. The information they contain is accessible in Intkey, and can be used in much the same way as coded information. -- Mike Dallwitz CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/

1 0

Re: What I'm trying to do
by Jim Croft 04 Sep '00

04 Sep '00

>The standard I'm working towards ALLOWS this degree of formality, but >doesn't ENFORCE it. Although I think I understand and like what we are trying to do I am having a slight problem here... a standard that does not enforce itself, is not really a standard, is it? It seems we are after a specification that says: 'a) This is how you describe the bits of organisms, and thus the whole organism.' 'b) If you wish not to describe organisms this way, you can, but in that case, this is how you must do it.' I can live with that as a standard... >If all descriptive data needs to be encoded to a strong specification, it will >never be so encoded (that's one reason, I think, why DELTA has failed as a >global specification). The DELTA spec would probably allow you to include a complete description as a lump of text, but it would not be particularly useful to do so, and I do not think Mike, Toni and Eric would be terribly impressed to see you using DELTA this way... :) >It seems to me that creating a standard like this would be more valuable >than simply XMLifying the DELTA of Lucid data file structure. Yes, as long as it can accommodate all the complexities, rigour and validation that is inherent in the DELTA and Lucid like data structures. jim

1 0

Re: Draft Spec mark 2
by Jim Croft 04 Sep '00

04 Sep '00

I agree with Eric on this one... in the example below, there is nothing in the structure to say where the qualifier belongs... If you read the data sequentially you might have a pretty good guess, but a specification that relies on sequence rather than the structural relationships and clear (explicit?) definitions of components for sense and meaning, is an bit fragile in my view... jim >I think there is a real difference in terms of a reduction in ambiguity. We >should generally make an effort to clearly associate modifiers with the >object that they are intended to modify. Suppose we have something like the >following: > ><ELEMENT> > <ELEMENT_NAME> leaf </ELEMENT_NAME> > <VALUE> > lobed margins > </VALUE> > <VALUE> > with spines > </VALUE> > <QUALIFIER> > rarely > </QUALIFER> ></ELEMENT> > >How should such a construct be interpretted? Do spines cover the leaf as a >whole, or are they confined to the lobed margins? Is it the spines which >are rarely present, or the lobed margins?

1 0

Re: Draft Spec mark 2
by Leigh Dodds 04 Sep '00

04 Sep '00

> I agree with Eric on this one... in the example below, there is > nothing in the structure to say where the qualifier belongs... If you read the data > sequentially you might have a pretty good guess, but a specification that > relies on sequence rather than the structural relationships and clear > (explicit?) definitions of components for sense and meaning, is an bit > fragile in my view... Exactly. Relying on ordering in the document is very shaky (CSV vs. XML). Far better to explicitly define how (and to what) the qualifier should be applied. This could perhaps be handled by statements about statements. i.e. a. leaf has lobed margins b. leaf has lobed margins with spines c. b is rare (I believe this is known as reification in RDF. The relationship between two objects can itself be modelled as an object, and thus properties can be attached to it, and additional statements made about it) Just musing... L. -- Leigh Dodds, Systems Architect | "Pluralitas non est ponenda http://weblogs.userland.com/eclectic | sine necessitate" http://www.xml.com/pub/xmldeviant | -- William of Ockham

1 0

Re: Validation
by Joseph H. Kirkbride, Jr. 04 Sep '00

04 Sep '00

The last few days of discussion have seemed to be drifting towards a 'free' structure which would not enforce comparablity. I totally support Jim that one of the key elements in DELTA format is the absolute enforcement of parallelism and compatibility. The idea that each species or specimen in a genus could be a separate document is anathema to me. I want rigor and compability in descriptions, keys, and data, not a retreat to the chaos of the past. Joseph H. Kirkbride, Jr. USDA, Agricultural Research Service Systematic Botany and Mycology Laboratory Room 304, Building 011A, BARC-West Beltsville, Maryland 20705-2350 USA Voice telephone: 301-504-9447 FAX: 301-504-5810 Internet: jkirkbri(a)asrr.arsusda.gov On Sun, 3 Sep 2000, Jim Croft wrote: > Kevin wrote: > >I'm not sure about a requirement for strong validation in the spec. I agree > >that validation should be allowed, but not required. > > But isn't that the main (only?) reason for bothering to encode descriptive > data in the first place? To try and impose a degree of consistency, > rigour, comparability and validity that is not possible in an > unstructured, unformatted blob of text? If the specifications are not > going to require this degree of strandardization, are we going to be > letting the tansferability of data and effort that we seek slip by? > > The value of the DELTA, Lucid and other character-based encoding of > descriptive data is that they do impose a high degree of validation, > whether it be by measurement ranges, allowable values or states, or > whatever. Without validation in biological datasets you end up with a mess > - just look at any herbarium or museum specimen dataset - who doesn't have > terrestrial specimens that plot in the middle of the ocean? > > >There are three types of validation you suggest here: > >1. Validate the element strings (and presumably also items, values and > >qualifiers) against predefined lists to guard against typographical and > >spelling errors; > > and for consistency... one can be right, but inconsistent, and although > humans can handle this, computers stuff it up... > > >2. Validate values against a list of allowable values for the character; > > This is the mechanism to addressing the above... > > >3. Validate qualifiers against a list of allowable qualifiers. > > It would be nice to put a cap on the number/type of 'qualifiers' allowed, > but you can bet that there will always be something on a list that people > will want/need to use, but at this stage we probaly should confine ouselve > to the need for the list itself, not to the content of the list? > > >Taking these in reverse order: > > as one would naturally want to do.. :) > > >3. As I see it the spec itself would define a set of allowable qualifiers > >(such as "rare" (= "rarely"?), "by misinterpretation", "uncertain" etc). I > >think we could probably agree on a limited set of qualifiers, and stick to > >that (with allowance for extension). If we do this, then "anchovies, > >please!" will be out for several reasons. > > in theory yes, but in practice, if you open this can of worms, the > discussion will probably go on for weeks... > > >2. Validation of allowable values is covered in the draft spec. One of the > >properties of an element (~"character") is a list of allowable values > >(~"states"). If such a property is filled, then validation can be done by > >the program that handles the data, just as in DELTA and Lucid. > > It is not just 'allowed', isn't it? Isn't this what the draft specs are > all about? > > >Two notes: > >* I'd like to allow but not enforce that a valid document have the > >allowable_values property filled. By not enforcing it, a simply marked-up > >natural-language description could be a valid document. This would perhaps > >mean that the spec could meet half-way the automated markup of legacy > >descriptions, and I'm keen to do this. Of course, a document without this > >property specified would not be able to be validated in the way you suggest > >and hence may not be very useful for some purposes, but this may be a price > >one is willing to pay for other benefits, and I think we need to keep this > >open. > > It would be pretty difficult to validate a blob of free text... but I can > see a case for wanting to include them in the exercise... so perhaps there > should be a deprecated class of elements that are essentially unable to be > validated but perhaps of interest in some instances? > > >* I'm using "allowable values" rather than "states" as this seems to me to > >be more general, and subsumes the requirement to distinguish between, for > >instance, multistate and numeric "characters". A numeric "character" of > >course, doesn't have "states", but it does have allowable values (integers > >in the case of an integer numeric, real numbers in the case of a real > >numeric). > > This would cover the use of text rather than various numeric values - there > is nothing that should say everything as to be represented numerically. It > should be possible to use text *and* to have it validated in some way. > > >1. How strong is the requirement for this type of validation? Enforcing this > >seems to me to be like requiring that all word documents carry in their > >header a dictionary of the english language to allow validation of > >spellings. It seems to me that providing tools that allow people to check > >these strings against a predefined list (defined either within the document > >or in an external resource) would be useful, but not absolutely necessary. A > >document that is not or cannot be validated in this way would not be > >useless, and would perhaps be more free. > > Documents do not have to carry their validation with them - they could > refer to an external source of validation (or dictionary in your above > example - this is what is implied and expected in Word documents at the moment) > > >Note that the spec as I see it would allow (but again, not enforce as DELTA > >and Lucid do) the encoding of descriptions. Thus, a valid document may be d1 > >as below. This would preempt the need for typographic validation, and allow > >allowable-values validation. But for some reason I don't want to disallow d2 > >as a valid document also. > > This may be a non threatening approach to gradually introducing validity > and rigour into descriptive data and is probably worth exploring some more... > > jim >

1 0