September 2000 - tdwg-content

Re: Minimalism AND functionalism
by Jim Croft 06 Sep '00

06 Sep '00

Kevin wrote: >I agree that having a DELTA-like degree of structure is a worthy goal, and >I'm not excluding such a degree of structure from the proposed standard, >just not requiring it as I keep saying. What I don't agree is that it's such >an obvious gold standard that we should say "for a description to be >regarded as a useful description, it MUST have this degree of structure". This comes back to the question of fitness for purpose... as part of the specs we need to define what 'useful' is.... or maybe abstractly, define how we define 'useful'... An unstructured blob of text might suit one particular need, but although containing all the appropriate information, might be totally useless for others. >I suppose I keep coming back to this: DELTA has just the degree of structure >you wish for, it's been around for what, 30 years?, it's even been given the >imprimatur as the "World Standard" already and - without putting too fine a >point on it and calling a spade a bloody shovel - it hasn't worked as a >standard. It may have if people hadn't gone around ignoring it and inventing others... or if it and the suite of products based upon it had been flexible enough and responsive enough to accommodate what people wanted to do when they wanted to do it... >Now I suppose I'm going to get howls of protest over this, >particularly from people who have been using DELTA for years, but I assay >that it's true. The only reason that this discussion is taking place at all >is that TDWG has realised for some time that DELTA as a standard hasn't >worked well, and we need to be open about this. My reading of the entrails at the time was the pressure to do the SDD thing was not that DELTA wasn't working, but rather that there were aggressively competing alternatives with champions working against each other inventing different an diverging mousetraps rather than collaborating to build a better one... SDD was an attempt to create an environment that would enable the definition of all descriptive data so it could be exchanged and shared by all the various application with minimal (zero?) information loss. It was to concentrate on data and data structure rather than the bells an whistles of applications and interfaces. We seem to be going in the right direction to do that... >One solution is to do the fixes to DELTA that have been proposed already >(and have been in draft now for ten years or so) and XMLify it. But it seems >to me that the main limitation of DELTA isn't simply that it isn't XML, or >that it's not complex enough (heaven forbid), or some easily-fixable thing >like that. It's more fundamental, and I suppose I'm exploring some sort of >fundament at the moment. I think part of the basic problem is that it's >tried to force too much structure and while this is a great promise it's >been an impediment in practice. I may well be wrong about this, but I think >there's something in it. A definition/specification that can accommodate both approaches would be nice, but it is very unlikely that we will be able to fully resolve the internal tension between rigour/structure and freedom/flexibility. They are incompatible and even if we can formulate a specification to handle both approaches, at the end of the day people have to apply the specs, and some will be control freaks, some will be anarchists and others will be schizophrenic - it is difficult to imagine a real conduit between the extremes. >Perhaps I'm trying to play lawn tennis with a table-tennis net! Would that >be fun? It'd certainly make it easier to get the bloody ball over the net so >the game can go on... and sometimes I get the impression the only thing common in the game is the grass - some are playing tennis on it (with or without a net), others are playing golf, others croquet, bowls, cricket, soccer, etc. But it is still only grass and we should be able to describe the stuff in some universally comprehensible fashion... jim

1 0

Re: Comparative data
by Bryan Heidorn 06 Sep '00

06 Sep '00

Given prior comments I think I should rephrase my prior comments about free text since if is not contradictory to what most others have said since, although it might not seem that way. I think that everyone should produce the most standardized, consistent and meaningful data possible, grounded only on fact and direct observation. If we had a perfect data model and everyone did that all our problems would be solved! Problem is that we'll never make a data model that can capture all of the semantics of the natural world. Even with the expressive power of Natural Languages we can not do that, certainly no taxonomic data standard will. So there will always be information that is important to communicate to other people that will not fit well through the thin pipe of the formal specifications. When that nasty and unfortunate thing happens, the standard needs to handle it. We don't want people on real projects avoiding using the standard because there is some critical piece of information that just will not fit through the data model. Then there is a legacy data... we'll never get enough money to fit it into a data model to the atomic level. Text blobs will have to do for some of that. Structure is good when we can get it. Bryan At 09:08 AM 9/7/00 +1000, Mike Dallwitz wrote: >> From: "P. F. Stevens" <peter.stevens(a)MOBOT.ORG> >> To: TDWG-SDD(a)USOBI.ORG > >>> producing comparative data is difficult. Nevertheless, >>> shouldn't this be one of the main objectives of taxonomy? > >> Absolutely. And surely this is what we are trying to facilitate here, and >> what guides the discussion. > >My comments were in response to Kevin Thiele's opinion: > >> I think part of the basic problem is [trying] to force too much structure >> and while this is a great promise it's been an impediment in practice. > >'Structure' apparently means a character list, i.e. the basis for producing >comparative data. Although Kevin says elsewhere that both structured and >unstructured data should be allowed (and I agree - they are allowed in >DELTA), the above statement seems to suggest that the 'impediment' should be >avoided by encouraging the use of non-comparative data. > >-- > >Mike Dallwitz > >CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia >Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 >Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/ >

1 0

Re: Comparative data
by P. F. Stevens 06 Sep '00

06 Sep '00

>>>From Mike Dallwitz: producing comparative data is difficult. Nevertheless, >shouldn't this be one of the main objectives of taxonomy? > Absolutely. And surely this is what we are trying to facilitate here, and what guides the discussion. Peter S.

1 0

Comparative data
by Mike Dallwitz 06 Sep '00

06 Sep '00

> From: Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> > To: TDWG-SDD(a)USOBI.ORG > I think part of the basic problem [with DELTA] is that it's tried to force > too much structure and while this is a great promise it's been an > impediment in practice. This is the problem with (or the virtue of) all character-based data systems (although DELTA does alleviate it by allowing comments and text 'characters', as discussed previously). I agree with Kevin that it is an impediment to acceptance of such systems. In fact, I think it's the main impediment - producing comparative data is difficult. Nevertheless, shouldn't this be one of the main objectives of taxonomy? Thirty years ago, Leslie Watson wrote: Perusal of the average taxonomic-descriptive work usually reveals that _as a source of comparative data_ it is hopeless. One genus will be described in terms of criteria that receive no mention in the next. Even species of the same genus may be described inconsistently. It is often impossible to distinguish with any degree of conviction between actual observation and extrapolation, between absence of a feature and mere failure to seek or comment on it. In general, the further one proceeds up the hierarchy, the less comparative the descriptions become. Given the situation prevailing in individual publications it is not surprising that scanning across them is even less satisfactory. ... If facts are wanted for reviewing the classification of a large group it is singularly disheartening to have to seek them in miscellaneous works of this kind. The labour is immense; worse, one sets out with the depressing knowledge that much of it will be wasted on discovering that the details thus compiled are not comparative. There is a welter of such material theoretically applicable to most major taxonomic problems, which will probably never be called upon because of its unpromising presentation and sheer intractability. L. Watson (1971). Basic taxonomic data: the need for organisation over presentation and accumulation. Taxon 20, 131-136. Things haven't improved much since, and they won't if we back away from teaching and strongly encouraging people to produce comparative data. Marking up free text into related chunks and making it available in electronic form would certainly reduce some of the labour, referred to above, of the person who is searching for comparative data. But it would not make non-comparative data comparative, and doing the markup would be a difficult and probably thankless task. I think that, in addition to the classificatory problems alluded to above, the majority of nomenclatural problems (proliferation of synonyms) are the result of not having readily accessible, comparative data. > It seems to me that the main limitation of DELTA isn't simply ... that > it's not complex enough. I'd say that that's exactly the problem - for 'not complex enough', read 'doesn't have enough features'. Of course, to add new features in the best way, and to minimize the complexity in some sense, you might have to abandon existing data structures. However, we should not throw the baby out with the bathwater. That is, we should retain the useful features of current systems. -- Mike Dallwitz CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/

1 0

Re: Draft Spec mark 2
by Eric Zurcher 06 Sep '00

06 Sep '00

At 09:40 6/09/2000 +1100, Greg Whitbread wrote: >Mixed content might also impact some of the validation >and context issues discussed earlier. > > From http://www.w3.org/TR/1998/REC-xml-19980210 , > Extensible Markup Language (XML) 1.0. > "An element type has mixed content when elements of that type may contain > character data, optionally interspersed with child elements. In this case, > the types of the child elements may be constrained, but not their order or > their number of occurrences:" > Greg, While validation of mixed content is indeed a problem when using XML 1.0 DTDs, XML Schema can handle it. Quoting from http://www.w3.org/TR/xmlschema-0/ , XML Schema Part 0: Primer (W3C Working Draft, 7 April 2000) "Note that the mixed model in XML Schema differs fundamentally from the mixed model in XML 1.0. Under the XML Schema mixed model, the order and number of child elements appearing in an instance must agree with the order and number of child elements specified in the model. In contrast, under the XML 1.0 mixed model, the order and number of child elements appearing in an instance cannot be constrained. In sum, XML Schema provides full schema validation of mixed models in contrast to the partial schema validation provided by XML 1.0." While mixed content is best avoided, I'd hate to rule it out completely. In particular, it might be good if those elements or attributes containing blobs of text could also contain some embedded HTML (or, rather, XHTML) markup, to allow for italics, sub- and super-scripts and so on. Eric Zurcher CSIRO Division of Entomology Canberra, Australia E-mail: ericz(a)ento.csiro.au

1 0

Re: DELTA cf SDD
by Mike Dallwitz 06 Sep '00

06 Sep '00

> From: Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> > To: TDWG-SDD(a)USOBI.ORG >> Example: >> >> 5,3/3&1<when infected with virus>/<occasionally>2<@reliability 3> >> >> = Flowers red, or red and white (when infected with virus), or >> occasionally yellow. > There are various types of things here that are masquerading as being > similar, when they're not at all. "<when infected by a virus>" is perhaps > a true comment. Note that these comments are only used for natural > language descriptions (they're stripped out when DELTA translates to an > interactive key etc). "<occasionally>" is a qualifier as defined in the > draft spec. The things are similar in that they qualify state values or attributes. The distinction between free-text and coded qualifiers is also made in the proposed DELTA enhancements (see http://biodiversity.uno.edu/delta/standard/proposal.exe). The coded qualifiers in the DELTA proposals are distinguished by being preceded by the symbol '@'. The above example could have been written 5,3/3&1<when infected with virus>/<@rarely>2<@reliability 3> The coded qualifiers could be checked for validity by programs, and could be used by programs for various purposes (including natural-language descriptions). Although the primary purpose of free-text qualifiers is for natural-language descriptions, they could also be used for other purposes (e.g. they could be searched). Their omission from the current version of Intkey is only for historical reasons (and is also irrelevant to the present discussion). > Lucid is currently the only program that uses this type of qualifier [like > @rarely] in identification, and it's very important. <@reliability 3> is a > command for a program. This should be made clear by the tag so that other > programs can ignore it. Of course, _any_ tag can be ignored or used by a program. I see no reason to distinguish, in the standard, between those tags that are used by the current version of LucID, and those that are not. Character and attribute reliabilities (which are not implemented in the current LucID) are arguably more important for identification than the '@rarely' qualifier. > I've always been bothered by the use of & (and) vs / (or) in DELTA. Using > Mike's example: > 5,3/3&1 = "Flowers red, or red and white" > The use of & is handy for codifying natural-language descriptions ... 'nuff said. The connectors 'and' and 'to' were put in DELTA for this purpose, in response to user demand. Omitting them from a new standard would be a retrograde step. > ... but in other applications it's highly problematical. This is a > homology problem ... The red & white broken colouring of virus-infected > tulips is actually non-homologous with uniform red or uniform white and > should be a separate character (or at least a separate state). Let's not confuse empirical facts with principles. 'red and white' _could_ be a homologous, intermediate condition between 'red' and 'white', in the same way that 'red to white' could be (cf. colour change in hydrangeas). You have tacitly admitted this by suggesting that it could be a separate state. In multi-purpose databases, you often have to make compromises in the way characters are defined, so that they will work reasonable well for all purposes. The possibilities in this example might be #1. flowers <colour>/ 1. red/ 2. white/ #2. flowers <colour>/ 1. entirely deep red/ 2. red and white/ 3. entirely white/ #3. entirely red flowers <presence>/ 1. present/ 2. absent/ #4. red and white flowers <presence>/ 1. present/ 2. absent/ #5. entirely white flowers <presence> 1. present/ 2. absent/ Character 1 is simple, produces the best natural-language descriptions, and is often acceptable for other applications. However, one of the other formulations would probably have to be used (depending on the capabilities of the software) if the distinction between 'red and white' and the other conditions was important for identification or classification. The natural-language descriptions would suffer, particularly if characters 3-5 were used. It's impossible to cater for the connector 'to' by redefining characters in this way. -- Mike Dallwitz CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/

1 0

Re: Draft Spec mark 2
by Greg Whitbread 06 Sep '00

06 Sep '00

Bob (Object) Morris might have an alternative solution but my relational database dislikes mixed ( though valid ) content like <!ELEMENT qualifier (#PCDATA ) > <!ELEMENT value ( #PCDATA | qualifier )* > as in Bryan's preferred example. ><ELEMENT> > <NAME> leaf </NAME> > <VALUE> > lobed margins > </VALUE> > <VALUE> > with spines > <QUALIFIER> > rarely > </QUALIFER> > </VALUE> ></ELEMENT> > My database wants that everything should have a name, otherwise it texts over and talks in blobs. Mixed content might also impact some of the validation and context issues discussed earlier. From http://www.w3.org/TR/1998/REC-xml-19980210 , Extensible Markup Language (XML) 1.0. "An element type has mixed content when elements of that type may contain character data, optionally interspersed with child elements. In this case, the types of the child elements may be constrained, but not their order or their number of occurrences:" So, to avoid unnecessary rewriting and to promote maximum utility I would prefer: <!ELEMENT value #PCDATA ) > <!ATTLIST value qualifier CDATA #IMPLIED> ... </ELEMENT> <NAME> leaf </NAME> <VALUE> lobed margins </VALUE> <VALUE qualifier="rarely"> with spines </VALUE> </ELEMENT> or <ELEMENT> <NAME> leaf </NAME> <STATE> <VALUE> lobed margins </VALUE> </STATE> <STATE> <VALUE> with spines </VALUE> <QUALIFIER> rarely </QUALIFER> </STATE> </ELEMENT> greg Australian ..................I................GPO Box 1777 Canberra 2601 National GREG WHITBREAD voice: +61 2 62509 482 Botanic Integrated Botanic Information System fax: +61 2 62509 599 Gardens......................S...........I.T. happens....ghw(a)anbg.gov.au

1 0

Re: Minimalism AND functionalism
by Kevin Thiele 06 Sep '00

06 Sep '00

>>>From Eric | I don't mean to be petty, but didn't we pretty much agree on a different | sort of nesting of elements in this general model? I thought we were still trying to work things out. What was it we agreed on... | Anyway, I can see that we will need to be cautious with terminology here. | "Element" (and "Attribute") have specific meanings within XML that could | potentially be confused with the terms as used for descriptive data. | Similarly, "valid" has a specific meaning within XML (basically, that a | document is both well-formed and compliant with a specified DTD or Schema), | that is slightly different from the notion of "validity" you are referring | to here (or is it?). I realize that in this context, none of us has much | difficulty using the appropriate mental "namespace", but if we ever reach | the stage of a formal number, we'll need to be careful of this. (Yes, yes, | I know I'm being terribly pedantic, and I'm sorry. But then I am addressing | taxonomists, after all.) Yes, I know. Can you suggest better terms and we'll use them. | >It contains information in a structured way that | >means something, and the information can be parsed. Sure, this document | >couldn't be used direct as input to Lucid or DELTA, but does that make it | >worthless? I'm working at the moment towards a future generation of the | >Lucid Builder (and perhaps future DELTA Builders will be similar) that will | >indeed be able to collect and interactively integrate such structured bits | >of information. | | Such a document is not "worthless", but it is definitely "worth less" than | it could be. Arguably, the chief goal in designing a descriptive data model | is to provide a format or structure which can be readily accessed or | manipulated by computer. We humans can usually cope fairly well with | conventional natural language descriptions, but we can do so because of our | substantial background knowledge of life, the universe, and everything. We | can know from a quick glance at the above what is intended (provided we can | read English!). But today's computers have no concept of "Smell", and has | no awareness that the term "Smell" in this description may be in any way | related to the term "fragrance" is some other description. In short, such a | "loosely" structured description has very little advantage over a | description written in natural language in terms of machine readability, | yet suffers some reduction in its human readability. I agree that having a DELTA-like degree of structure is a worthy goal, and I'm not excluding such a degree of structure from the proposed standard, just not requiring it as I keep saying. What I don't agree is that it's such an obvious gold standard that we should say "for a description to be regarded as a useful description, it MUST have this degree of structure". I suppose I keep coming back to this: DELTA has just the degree of structure you wish for, it's been around for what, 30 years?, it's even been given the imprimatur as the "World Standard" already and - without putting too fine a point on it and calling a spade a bloody shovel - it hasn't worked as a standard. Now I suppose I'm going to get howls of protest over this, particularly from people who have been using DELTA for years, but I assay that it's true. The only reason that this discussion is taking place at all is that TDWG has realised for some time that DELTA as a standard hasn't worked well, and we need to be open about this. One solution is to do the fixes to DELTA that have been proposed already (and have been in draft now for ten years or so) and XMLify it. But it seems to me that the main limitation of DELTA isn't simply that it isn't XML, or that it's not complex enough (heaven forbid), or some easily-fixable thing like that. It's more fundamental, and I suppose I'm exploring some sort of fundament at the moment. I think part of the basic problem is that it's tried to force too much structure and while this is a great promise it's been an impediment in practice. I may well be wrong about this, but I think there's something in it. | If you can actually create software that can make real sense of bits of | information like this, you will have accomplished something that has eluded | AI researchers for the past half century. I appreciate the irony, but I think that creating software that can handle the degree of structure I'm proposing (which is actually considerable, when you think about it) is a lot different from creating an AI that can precis Hamlet. | This discussion brings to mind a debate between the mid 20th century | American poets Robert Frost and Carl Sandberg. Frost argued that writing | poetry in free verse was like playing tennis without a net. Sandberg | countered that tennis would be a better game if there were no net to get in | the way. As one might guess, I've always been inclined to agree with Frost... Perhaps I'm trying to play lawn tennis with a table-tennis net! Would that be fun? It'd certainly make it easier to get the bloody ball over the net so the game can go on... Cheers - k

1 0

Re: Minimalism AND functionalism
by Jim Croft 05 Sep '00

05 Sep '00

>This discussion brings to mind a debate between the mid 20th century >American poets Robert Frost and Carl Sandberg. Frost argued that writing >poetry in free verse was like playing tennis without a net. Sandberg >countered that tennis would be a better game if there were no net to get in >the way. As one might guess, I've always been inclined to agree with Frost... He was the 'promises to keep, miles before sleep' guy wasn't he? He would probably identify fairly strongly with the SDD list... jim

1 0

Re: Minimalism AND functionalism
by Eric Zurcher 05 Sep '00

05 Sep '00

At 09:31 5/09/2000 +1000, Kevin Thiele wrote: >So I'm proposing, as ground-zero, that the only absolute requirements are >that a document have DESCRIPTION elements, that these have ELEMENT elements >(terminology is unfortunate here) and that these have VALUEs. This is a long >way down the track towards minimalism, I know, but I'm floating it for >comment. > >So, as I've exampled before, I'd like something like: > ><DOCUMENT ID = "d2" Name = "One thing I know about a rose"> > <DESCRIPTION> > <ITEM> > <ITEM_NAME> Rose </ITEM NAME> > </ITEM> > <ELEMENT> > <ELEMENT_NAME> Smell </ELEMENT_NAME> > </ELEMENT> > <VALUE> Sweet </VALUE> > </DESCRIPTION> ></DOCUMENT> > >to be a valid document. I don't mean to be petty, but didn't we pretty much agree on a different sort of nesting of elements in this general model? Anyway, I can see that we will need to be cautious with terminology here. "Element" (and "Attribute") have specific meanings within XML that could potentially be confused with the terms as used for descriptive data. Similarly, "valid" has a specific meaning within XML (basically, that a document is both well-formed and compliant with a specified DTD or Schema), that is slightly different from the notion of "validity" you are referring to here (or is it?). I realize that in this context, none of us has much difficulty using the appropriate mental "namespace", but if we ever reach the stage of a formal number, we'll need to be careful of this. (Yes, yes, I know I'm being terribly pedantic, and I'm sorry. But then I am addressing taxonomists, after all.) >It contains information in a structured way that >means something, and the information can be parsed. Sure, this document >couldn't be used direct as input to Lucid or DELTA, but does that make it >worthless? I'm working at the moment towards a future generation of the >Lucid Builder (and perhaps future DELTA Builders will be similar) that will >indeed be able to collect and interactively integrate such structured bits >of information. Such a document is not "worthless", but it is definitely "worth less" than it could be. Arguably, the chief goal in designing a descriptive data model is to provide a format or structure which can be readily accessed or manipulated by computer. We humans can usually cope fairly well with conventional natural language descriptions, but we can do so because of our substantial background knowledge of life, the universe, and everything. We can know from a quick glance at the above what is intended (provided we can read English!). But today's computers have no concept of "Smell", and has no awareness that the term "Smell" in this description may be in any way related to the term "fragrance" is some other description. In short, such a "loosely" structured description has very little advantage over a description written in natural language in terms of machine readability, yet suffers some reduction in its human readability. If you can actually create software that can make real sense of bits of information like this, you will have accomplished something that has eluded AI researchers for the past half century. Of course it's important to be able to store loosely structured bits of information, but in general the more rigour we can apply, the better. This discussion brings to mind a debate between the mid 20th century American poets Robert Frost and Carl Sandberg. Frost argued that writing poetry in free verse was like playing tennis without a net. Sandberg countered that tennis would be a better game if there were no net to get in the way. As one might guess, I've always been inclined to agree with Frost... Eric Zurcher CSIRO Division of Entomology Canberra, Australia E-mail: ericz(a)ento.csiro.au

1 0