December 1999 - tdwg-content

Re: (RQT) character/state/comment
by Gregor Hagedorn 20 Dec '99

20 Dec '99

Leigh wrote: > As a first cut I just transferred everything from DELTA comments into > XDELTA comments, although it was quite obvious that the 'comment' > component of DELTA was overloaded depending on context (a legacy > problem I guess, and one which raises the prospect of 'extensibility' > as a requirement of the new format). The concept of "overloading" is very good, I think it helps to understand the issue. I believe in the new standard, we will want to have as little overloading (which means context specific contents) as possible. In general, all data elements should be defined much more exactly than in Delta. Many things in Delta are really defined by result, not by purpose, which means that comments may have a different function in different applications (e.g. identification or nat. language reporting). > DELTA already allows dependencies between characters - such > that if a specific state has been selected for a character, other > characters are ignored (i.e. no wings, ignore anything to do > with wings). Is there any other dependency relationships that > might be required, or additional information about such a > relationship? > > Restricting states to particular options depending on the property > in question (e.g. leaf and/or wing shape) leads back to the > prior discussion on accepted standards for character description. If I understand this correctly, it assumes that there would be a general "property" entity. An instance of this entity would then be the property "shape", which has a list of generally applicable values. Some of these values would apply to leaves as well as to butterfly wings, other may be more specific. I wonder, whether terminology is actually well developed enough to really support this notion. The discussion is somewhat similar to the discussion on lexicons/globally applicable character definitions. My feeling is, that nowhere ever has "shape" been defined in a way that would be applicable to all areas. Defining dependency rules to create overlapping, or partly matching sets of character state seems to me a troublesome process. Further, one should realize that we need more than a name for such concepts. Two shapes in plants and butterfly wings that may be more or less identical in concept may be named identical in one language, but not in another. I believe that properties are useful (and from the viewpoint of information modeling are preferable to use), but dangerous to use in reality, simply because we as biologists never did any globally applicable terminology. Also, we have to make the concept clear to any user, e.g. that if she or he changes the name of a state, this change is actually applicable not only to his butterfly, but also to the plant the collegue is working on in the same database (a bit gross example, but more likely somewhere in a large institution for two people working on separate orders, or perhaps on ferns and conifers). Who thinks global state lists for something like shape (already separated into 2D-shape, 3D-shape), texture, smell, color, are possible and useful? Gregor ---------------------------------------------------------- Gregor Hagedorn G.Hagedorn(a)bba.de Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: (GEN) Lexicons
by Gregor Hagedorn 20 Dec '99

20 Dec '99

Originally, the definition of applied schema ("character definitions", lexica, terminology) is outside the scope of this discussion group. While judging from my own experience I think it unlikely to make fast progress here, I completely agree that it is most important to work on it. I would therefore endorse Alex's proposal to create a new top level subject: -> Please do use a subject line (LEX) if you want to talk about applied terminology for a certain group (all plants, all insects, etc.) If these discussion threads become too big, we should then open a new list for those interested in it. Gregor ---------------------------------------------------------- Gregor Hagedorn G.Hagedorn(a)bba.de Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Compression of XML files
by Robert A. (Bob) Morris 18 Dec '99

18 Dec '99

People interested in XML file size will be interested in a new free program XMill from AT&T Research Labs and UPenn. XMill is said to compress files up to twice as much as gzip on proprietary formats when data in those formats is first rendered in XML and then compressed with XMill. This is essentially because XMill can use the tag information to reorganize the data for greater redundancy. http://www.research.att.com/sw/tools/xmill/ points to the software and technical material about it. One of the authors, Dan Suciu, is also the a co-author of "Data on the Web : From Relations to Semistructured Data and XML", Morgan Kaufman. This is great book whose title is accurate. To read it, you have to be comfortable with graph theory, regular expressions, and finite state automata. If you are not, you can accept this a summary: XML is more general than relational databases and a little less general than pure object oriented databases.

1 0

Re: (RQT) Structure, Matrix, and Text
by Jim Croft 18 Dec '99

18 Dec '99

>Free comments anywhere, where it is not clear to which data item a >comment belongs, will not help. Also, in free text the same concept >will often be expressed with different words. This is a common >problem known by anybody who tried to capture free conventional >descriptions in a database. ... and is probably intractable. One person can often see data in another comments, while another may see others data as hardly worthy of comment. In many descriptive works the pearls of information are subtle and implicit rather than explicit and in your face (but we will not open the biological description as an art versus a science debate). Discussions about what is and is not data will go on forever and the critera will change fron genus to genus, gamily to family, order to order, and biologist to biologist. What is needed is a common standard format to store all descriptive data and information in an understandable and unambiguous way for exchange between applications so that nothing gets lost or made unavailable in transactions between applications and up and down the taxonomic hierachy. To ignore data and information is probably excusable, but to actually lose it is sinful and unpardonable. It is a shame that verbosity is the price we have to pay for this flexibility, but the tecnology can handle it. I did not think we were considering the 'direct processing' of these tagged XML (or whatever) files; they were to be just the lingua franca of applictions using descritpive data. A contemporary example would be the DELTA format: after Intkey, the DELTA Editor, MEKA or whatever have grabbbed the data they neither know nor care what a DELTA file looks like. What I think we are after is a format like this that all applications know about and can use without any extrenal hacking and preprocessing. There is a major differece between an effective archival and exchange format and an efficient arrangement of data for processing and analysis; I doubt if it will be posible to have both in the same package. My understanding was that this list was going to focus on the former, while application developers and closet hackers were going to whatever they needed or wanted with the latter. A recurring nightmare is the thought of pouring data backwards and forwards between applicatons, losing a little at each export/import like the fraying ends of an aging chromosome at each cell division, until eventually there is nothing left... jim __________________________________________________________________________ Jim Croft ~ jrc(a)anbg.gov.au ~ http://www.anbg.gov.au/people/croft.jim.html ph 02-6246-5500 ~ fx 02-6246-5248 ~ GPO Box 1600 Canberra ACT 2601

1 0

(RQT) Structure, Matrix, and Text
by Gregor Hagedorn 17 Dec '99

17 Dec '99

Although some time has passed, I would like to take up something up that was posed by Kevin under "(XML) XML?". > The problem to my mind is that in current formats, e.g. DELTA and > LucID, much information is implied by context. Thus, in > 1010 > 0101 > The taxa and character state numbers (identities) are implied by the > position of the data bit in the matrix. In XML this information is > verbosely explicit. > The following question: what could we do with such data as XML that we couldn't > do with the data as a simple structured file as above? > Is direct processing of XML data any easier than direct processing of the > data in a simpler format? Perhaps there will be off-the-shelf parsing tools, > but how much of a benefit will this be? I see among other things already mentioned, 2 benefits: 1. A matrix is inherently ordering dependent. Reordering characters or taxa is necessarily tied with the contents of data. In a more verbose presentation (be it XML or not), these actions can be decoupled. This is not currently an advantage as long as we mainly use isolated programs with import/export interaction alone. However, I believe in the future we will use programs working on networked data. This could mean that I locally modify only the character definition, perhaps translate it to German, define a character and taxon subset for my use, and change character hierarchy or ordering, but still cooperate with other groups using the same description data on the net. That requires of course, that the descriptions use, at least partially, standardized character definitions and that Globally unique identifiers have been assigned to characters. The advantage is not clear when thinking of DELTA, where the character ID is local, and also identical with the character ordering definition. These are understandable design limitation, which have the advantage to produce simple and compact files. In the future, I believe, we will want to be more flexible than this. In DeltaAccess, I followed the DELTA example and coupled ID and ordering, which does brings me quite some trouble currently... 2. Only with explicit and verbose tagging can be achieve forward AND backward compatibility. If LucID would want to support additional modifiers, perhaps change the 2 frequency states into a more specific and complex system, or if it would want to allow state specific reportable text and internal annotation (which are extremely difficult to support in matrix notation anyways) - the parser of a new application version would have to be changed quite a bit, perhaps rewritten - any older software would no longer be able to read these data. With a well designed verbose, fully tagged format, both these restrictions need not apply, I believe. > Is the following true?: once upon a time, computers could represent but not > efficiently analyse or process textual data, hence documents were stored as > text but "data" were stored as matrices etc. Now, XML has blurred the > boundary between these types of information ("textual" and "data") and we're > exploring the implications of that blurring. But are there now no > differences, and no further need for a matrix? I believe we need structure, and we need support for quality control. The distinction between text and data is blurred somewhat indeed, but text must be structured in ways that can be processed analytically. Free comments anywhere, where it is not clear to which data item a comment belongs, will not help. Also, in free text the same concept will often be expressed with different words. This is a common problem known by anybody who tried to capture free conventional descriptions in a database. For example, even in DELTA, frequency statements in free text "comments" may express the same frequency by multiple more or less synonymous wordings (is "mostly" and "usually" the same or not?). All frequency wordings are not really defined, and are therefore not accessible to analysis. DeltaAccess tries to use textual frequency modifiers, where the number of possible modifiers is restricted in the character definition, and where each frequency modifier is defined as to the exact upper and lower frequency range it presents. This is analytically accessible. Gregor ---------------------------------------------------------- Gregor Hagedorn G.Hagedorn(a)bba.de Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: (GEN) Lexicons
by Jim Croft 14 Dec '99

14 Dec '99

>I would like to suggest that if part of our group think it is worthwhile >to attempt to construct a standard character list for one or more (or all) >kingdoms (to which I am not opposed) that we have an additional first level >topic identifier for subject lines in our postings (currently we use GEN, >RQT and XML). Whatever happened to Richard Pankhurst's TDWG-based attempt to come up with a minimum character set (or something) for just the (vascular? seed? flowering?) plants? I seem to remember there was a fair bit of wailing, gnashing of teeth and slitting of wrists over this exercise... jim __________________________________________________________________________ Jim Croft ~ jrc(a)anbg.gov.au ~ http://www.anbg.gov.au/people/croft.jim.html ph 02-6246-5500 ~ fx 02-6246-5248 ~ GPO Box 1600 Canberra ACT 2601

1 0

Re: (GEN) Lexicons
by Alex R. Chapman 14 Dec '99

14 Dec '99

>>>From Alex R. Chapman (alexc(a)calm.wa.gov.au), 14/12/99, 10.46: Kevin's comments concur with my earlier posting on the scope of this list's topic. While reference to particular biological characters and attributes is essential to illustrate the potential functionality of any new standard system of representing descriptive data, it is surely beyond the scope of this group to attempt to develop or standardise the character lexicons themselves. A worthy and extremely useful task (and here I specifically differ from Kevin T.) but surely a major threat to reporting back to TDWG with a new data model in a timely manner! I would like to suggest that if part of our group think it is worthwhile to attempt to construct a standard character list for one or more (or all) kingdoms (to which I am not opposed) that we have an additional first level topic identifier for subject lines in our postings (currently we use GEN, RQT and XML). Perhaps LEX - for the discussion of developing standard lexicons or name-space vocabularies for one or more fundamental taxonomic groups. This way those most interested in this area can continue the debate. (It would be a fascinating outcome if most of the postings end up under this category!) In fact such a sub-subgroup would perhaps help to provide a range of character types or exemplars with which we could test any proposed data model arising from the main discussion. Cheers, Alex ____ Alex R. Chapman Email: alexc(a)calm.wa.gov.au Research Scientist Voice/Fax: +61 8 9334 0513 / 0515 WA Herbarium - Department of Conservation and Land Management Locked Bag 104 Bentley Delivery Centre Western Australia 6983 ---------- Original Text ---------- From: "Kevin Thiele" <kevin.thiele(a)PI.CSIRO.AU>, on 9/12/99 7:55: There is a subtext running in this discussion - whether part of our scope is the creation of lexicons or standard name-spaces - that to me is causing confusion. For instance: >>>From Leigh: >Restricting states to particular options depending on the property >in question (e.g. leaf and/or wing shape) leads back to the >prior discussion on accepted standards for character description. > >Defining, and agreeing upon these standard notations/descriptions are >A FUNDAMENTAL PART (my caps) of specifying this new format, and one that isn't >solved simply by deciding to use XML (for example). Its part of the >fundamental design and modelling, and is therefore something >that should be addressed early on. But from Leigh again: >I'd say that the DELTA approach - of avoiding domain (i.e. zoology, >virology, etc) specific notations in the format has worked well. >And I think this is the level that any initial work should be pitched at. >i.e. the data format should encode taxonomic *data* - just as DELTA >does. Any domain specific schema can be layered on top of this, or >include it. Begin with capturing the relevant data just as DELTA does, >and then progress from there. So do we or don't we? Am I misinterpreting these that they seem to say opposite things? >>>From Gregor: >It is true: Morphological structures may have containment >hierarchies, but I believe that these depent strongly on the >viewpoint of the author or user. > >EXAMPLE 2: Stuff can be in-between: The inflorescence contains part >of stem, part of leaves, and all flowers. Which leaves are part of >inflorescence and thus called bracts, and which aren't is often a >matter of taste, school, country... > >Thus: there are multiple concurrent or competing hierarchies, which >may overlap. The only problem with competing hierarchies is if we are trying to standardise and resolve the conflicts. If every worker resolves for their own project what to call bracts, this is not a problem for us. >>>From Jean-Marc: >We are designing XML vocabularies for the description of biological >species. Are we? I thought we were designing a format by which such a vocabulary can be represented. For the record, all the current systems (DELTA, LucID, NEXUS etc) enforce nothing lexically, they merely enforce a particular way of representing data. Two data sets for similar groups of plants may contain entirely different characters, or the same characters worded in different ways, or the same characters resolved into states in different ways, or (occasionally) identical characters. Comparing and combining datasets automatically is thus impossible. This seems such a shame, but is it perhaps unavoidable? Thus, if we are designing vocabularies, we are going a long way beyond what's been attempted before. Personally, I think designing domain-specific vocabularies will never work, unless the domain is the individual worker or group of collaborating workers. The popularity of lexicons is the old seductive universalism again. Great idea, but... There are two problems. Firstly, there are (broadly) two types of characters used in descriptions (and keys) - lets call them comparative and diagnostic characters. Comparative characters are the fairly general ones - e.g. leaf shape, ovary position - the sorts of characters that one would aim to describe consistently for all taxa in a monograph. Diagnostic characters are special characters that are useful for separating two or more taxa (of course, sometimes fairly general characters are diagnostic, but not always). A real example of a diagnostic character (from Synaphaea: Proteaceae): Ovary with an apical ring of translucent glands......S. bifurcata Ovary without glands.................................S. oulopha Clearly, no generalised lexicon or name-space will allow for capture of such diagnostic characters. BUT, perhaps we can have a standardised representation for the generalised characters using a lexicon and then use extensibility to allow user-specific diagnostic characters? To some extent, but perhaps not...: I will (foolishly) raise a challenge here that any generalised morphological character that anyone can come up with (in the plant domain) will be entirely inadequate for capturing data for some groups. For example, the most straightforward character I can think of is Leaves present absent But, a diagnostic difference between Discaria pubescens and Discaria nitida (Rhamnaceae) is the degree to which the leaves persist - in both, leaves tend to be absent in the adult plant, but in D. pubescens they are often completely absent while in D. nitida there are usually scattered reduced leaves on younger branchlets. And in Podostemaceae and Utricularia there's no guarantee that a leaf-like part is a leaf, because the conventional differentiation of vegetative parts into leaves/stems doesn't hold. Mother Nature's a tricky old dame, and any character definition will be inadequate to catch her. But do we put up with the inadequacy for the advantages that the universality brings? - if it means we constrain our ability to capture data, then I'd say no. So, I'd like to suggest that we try to develop a standardised data representation, but put no constraints on character definitions whatsoever. Cheers - k Beware the Universe - it bites

1 0

Re: RQTS : tag order
by Gregor Hagedorn 13 Dec '99

13 Dec '99

Bob Morris wrote: > Here's a proposed requirement for data exchange, brought home by one > of Leigh's examples: > > <RQT> It should be possible from the data itself, perhaps together > with external resources to which it refers, for software to determine > whether the order matters for any collection of tags in the > data.</RQT> I agree, especially that the software should be able to determine it. Regarding XML, tags, and DTDs, I believe, the assumption is that order does not matter, wherease an order can be imposed on attributes within a tag, correct? > One answer is "order always matters". That would be regrettable. For > example, being able to detect that some list is unordered can make it > easier to know whether two data streams represent the same > object. (That's probably a deeper desideratum.) Does order matters for our problem? Where would it matter, any examples from descriptive data? In fact, thinking in my own relational information model, we have a problem here: Some parts of the structure will be relating to records, i.e. instances of an entity type. In my thinking, the order of items to be described, and the order of characters used to perform the description should NOT matter. It is a plausible user requirement to report any of them in any order. That does not mean that characters or states do not have an order, but this order is secondary, and multiple orders are desirable. Being ordered should not be a requirement of the exchange format. However, there are other data elements that refer to attributes of entities. Ordering them makes some parsing operations easier. However, what fundamental advantages would it have to impose an order on them? Finally, some item are ordered, e.g. it will be desirable to impose an order on multiple states in a descriptions. Written in (loose) DELTA: 1,1/<rarely>2 -> Flowers blue or rarely white may be different from 1,2/1<rarely> -> Flowers white or rarely blue However, I believe that the DELTA method of using a text-sequence is not necessarily the best solution. Alternatively, a data element may be introduced to store the ordering information. For example, DeltaAccess handles this problem with the numeric SEQ attribute of the DESCR entity. Gregor ---------------------------------------------------------- Gregor Hagedorn G.Hagedorn(a)bba.de Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

My summary of "Why use XML"
by Gregor Hagedorn 13 Dec '99

13 Dec '99

Kevin brought up the question of advantages of XML. Somewhat belated, because I have been away for 10 days, here are some comments, trying to summarize the discussion as I see it. 1. I agree entirely with Noel that the discussions about XML on this list are intended to _explore_ the advantages of XML, not to settle on it. XML is well defined to be used in the context of text documents, but my impression is that for data that contain more structure, the XML variants are right now being developed (for example XML-Schema, XML-RDF). So it will be good to wait a moment before making decisions. And there is much to do regarding the information model itself! I hope that we will have more discussions about this. 2. I think Leigh Dodds did a good job in listing the advantages in his XDelta document. Generally I think that using standards and standard tools is a good idea. However, most certainly, XML will not be an "ultimate standard", as Jean-Marc thinks it is. XML will be certainly replaced by improved mechanisms. My feeling is that it is sufficiently relevant a step that it will be supported in a backwards compatible way even in 50 years. Of course, nobody can predict this. 3. Regarding the issue of size of data set, verbosity, and data compression: Some comments, esp. when it comes to "using it on PalmTops etc." confuse the issue of an exchange format with a native data format. Esp. given that XSLT will make it very easy to convert data, any program will most likely have a native, usually binary format that is optimized for operation speed and/or file size. I think we should not consider size any factor of an _exchange_ format. This format should be as robust and compatible for the future as possible. Many of us are developers of programs and tend to underestimate the effort that goes into the development of data sets. Our programs will certainly be replaced by better ones, but if the data can be maintained, revised, and improved as it is done for centuries with printed publications, I believe we will make better progress in biodiversity research. The data sets are the gold, and they should be preserved in a way that remains readable in 50 years. Because of this, however, using compression algorithms and then write it on a CD may be a bad idea for long term storage. Imagine having to decompress something that was compressed with some program on a Apple II machine... Gregor ---------------------------------------------------------- Gregor Hagedorn G.Hagedorn(a)bba.de Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: RQTS : tag order
by Robert A. (Bob) Morris 13 Dec '99

13 Dec '99

Gregor Hagedorn writes: > Date: Mon, 13 Dec 1999 16:21:15 +0100 > From: Gregor Hagedorn <G.Hagedorn(a)bba.de> > To: TDWG-SDD(a)usobi.org > Subject: Re: RQTS : tag order > > Bob Morris wrote: > > > Here's a proposed requirement for data exchange, brought home by one > > of Leigh's examples: > > > > <RQT> It should be possible from the data itself, perhaps together > > with external resources to which it refers, for software to determine > > whether the order matters for any collection of tags in the > > data.</RQT> > > I agree, especially that the software should be able to determine it. > Regarding XML, tags, and DTDs, I believe, the assumption is that > order does not matter, wherease an order can be imposed on attributes > within a tag, correct? No, it is exactly the opposite. A DTD imposes an order on tags, but attributes are unordered. However, the November 5 working draft of XML Schema provides mechanisms for settling tag order questions. XML Schema is a language for specifying tag constraints in XML itself, rather than in the DTD language which is not itself a special case of XML. Among other things, it provides strong data typing, including the ability to restrict the range of data, whereas in a DTD the only primitive data type is a string. It is slightly plausible that XML Schema would actually provide a good data modeling language independent of whether one wished to use it as an interchange language or as the native data store. Bob Morris

1 0