December 1999 - tdwg-content

(XML)shearing files between programs without data loss
by Jean-Marc Vanel 04 Dec '99

04 Dec '99

Shearing files between programs without data loss Several users have reported problems of data loss when passing files between programs. With XML and the Document Object Model, this will never happen, because it's easy for a programmer to work on the document tree, just adding nodes where additions have been made by user, and leaving the other parts unchanged. So a round-trip beetwen programs will be feasible, leading to a composite document having a common core of elements understood by both programs, some elements understood by program A only, and some by program B only. Cheers JMV "XML is not a problem, it's full of solutions"

1 0

Re: (XML) XML?
by Susan B. Farmer 04 Dec '99

04 Dec '99

*snip* I've got to catch up on my back reading ... But I saw this and *had* to respond! > >Example 2: Say I want to run NaviKey on my Palm Pilot*. The device has >limited resources and I would prefer to use very small files if at all >possible. > >As you say, these problems will all go away someday. > >Best, > >-Noel Cross > >*I only wish it were possible -- A few people have actually asked for this >capability, as it would be handy to use such as program in the field. > It's probably not that far off. I found the program that Larry Morse first did for interactive specimen identification before I entered Grad School. It takes 64K to run. (At the time it probably had a limit of 64K for data too -- and that stretched the limits of the computer.) Most of your hand-help calculators have a memory of at least 64K now. How programmable *is* a Palm Pilot? Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Re: (XML) XML?
by Noel Cross 04 Dec '99

04 Dec '99

On Sat, 4 Dec 1999, Jean-Marc Vanel wrote: > And now the figures about the "verbosity" of XML: > > My XML example (a single plant species from the flora of China) compresses from > 2500 bytes to 1090 (57%): > http://jmvanel.free.fr/Samples/DisplayDescriptions/species_example.xml > > An Xdelta example (Lepidoptera) on Leigh Dodds' site > compresses from 50500 bytes to 3600 (93%), because it has a large number of > repeted tags. > > I used winzip. > > Now we see that we can have all plant species on a CD (650 Mb), even without > compression: > 2500 bytes * 250 000 species = 625 Mb > > And on a DVD you can have 10 times more! And also on Internet with the new HTTP > 1.1 protocol, the files can be compressed during transmission. Still 2 more > arguments: the verbose repeted tags are not repeted when the XML file has been > parsed and is in memory; the last argument is financial: today you can buy a 10 > 000 Mega-bytes disk for 200 US$ , and it will decrease. > > So file size is NO problem for taxonomy with XML. You must understand that it's > the small price to pay for extensibility and interoperability. Agreed. In some cases large file size does present a problem though. Example 1: NaviKey is a Java applet that has to do the following: - download data - parse data - interact with user If the data are uncompressed XML files, then download time is obviously a major problem. If the XML is compressed, there is still the added time involved in compressing/uncompressing data, as well as new memory considerations. Example 2: Say I want to run NaviKey on my Palm Pilot*. The device has limited resources and I would prefer to use very small files if at all possible. As you say, these problems will all go away someday. Best, -Noel Cross *I only wish it were possible -- A few people have actually asked for this capability, as it would be handy to use such as program in the field.

1 0

Re: (RQT) Character and item hierarchy
by Stuart G. Poss 03 Dec '99

03 Dec '99

"Robert A. (Bob) Morris" wrote: > > The biggest advantage of XML for this group is its impending ubiquity, > not its formal structure, and (in my opinion) the discussion here > should return to discussing problems, not solutions because that > ubiquity will make the community want to use it whether it is the best > fit or not (just as has happened with HTML as a presentation vehicle). > > FYI I noticed this on the news lines: http://www.infoworld.com/articles/ic/xml/99/11/30/991130icxmlbiz.xml It seems we are NOT alone in the universe after all.

1 0

Re: (XML) XML?
by Noel Cross 03 Dec '99

03 Dec '99

On Thu, 2 Dec 1999, Kevin Thiele wrote: > Over the past couple of days I've partially implemented an export function > to produce Leigh's XDELTA documents (as a simple example of a possible XML > format) for the data in Lucid keys. I have a key to families of flowering > plants of Australia (240 taxa, 166 characters, 600 states). The data I'm > using are simple - basically a score matrix, a list of taxa and a list of > characters. The file sizes in three formats for these data are: > > LucID 166 kb > DELTA 240 kb > XML c3 Mb > > And this is only the most basic XML!. > By comparing three different encodings of a particular data set, I think that Kevin has given us a very graphic demonstration of the potential verbosity of an XML specification for descriptive data. This is definitely a syntactic issues that we'll need to address, even as we try to characterize the semantics of what it is that we need to encode. For this reason, I would suggest that examples giving in XML should not imply that we'll end up actually using XML as the recommended syntactic encoding of descriptive data, and that indeed no decision has been made on this matter. -Noel Cross

1 0

(XML) XML?
by Kevin Thiele 02 Dec '99

02 Dec '99

Will someone list for us exactly the benefits and disbenefits that will flow from using XML for a data interchange standard? Gregor wrote: >>These issues should be seen quite separately, and that we need both >> >>1. A markup language for text documents. This includes: >> - existing descriptions, captured by OCR or other means, where the >> markup would be manually (or automated/data mining?) >> - computer generated natural language descriptions that are >> published as electronic documents. A new CSIRO package could have >> a ToNat command that automatically adds the necessary, hidden >> markup code. >>2. A data language for new observations, including repeated >> measurements or repeated observations of categorical data, e.g. >> shape of multiple leaves in a single specimen. Further, many >> structures for knowledge managements (data revision, annotation, >> quality control and assessment) need to be implemented here. >> >>The issues overlap, and it would be beneficial to use as much common >>syntax as possible, but fundamentally I believe them to be quite >>different. To which Jim Croft replied: >Can you explain in more detail why they should be fundamentally different? >Aren't the differences just a matter a matter of degree, different points >on a continuum of data as it were? And aren't the basic principles >applicable across the whole? We need to explore this a lot more. I'm talking here about the data language, not the markup language, as it seems to me there are practical, even if not theoretical, differences. I can well understand the benefit from marking up a document (a text description) in XML. But when it comes to straight data, I'm less sure. By straight data, I mean data that can be represented most concisely as a matrix, for instance: Characters: Leaves ovate elliptic Flowers blue yellow Taxa: taxon1 taxon2 Data (taxa x states): 0101 1010 (...or a DELTA type disaggregated matrix) With this type of data, the XML markup becomes considerably more verbose than the data. Is this a problem? Over the past couple of days I've partially implemented an export function to produce Leigh's XDELTA documents (as a simple example of a possible XML format) for the data in Lucid keys. I have a key to families of flowering plants of Australia (240 taxa, 166 characters, 600 states). The data I'm using are simple - basically a score matrix, a list of taxa and a list of characters. The file sizes in three formats for these data are: LucID 166 kb DELTA 240 kb XML c3 Mb And this is only the most basic XML!. Now, perhaps in a few years we'll be carrying terabytes of data in our back pockets, but I would have thought that file size is important, particularly if I'm using this as an interchange format and want to email the data to someone (the xml file compresses very well into a zipfile, it's true). The following question: what could we do with such data as XML that we couldn't do with the data as a simple structured file as above? Leigh wrote: >Secondly I might not be generating anything visible at all. I can >well imagine an application that will take an XML document and >from the data within produce (say) a taxonomic tree or tress of that >data. Here I wouldn't use a stylesheet, I'd simply process the >data directly. Is direct processing of XML data any easier than direct processing of the data in a simpler format? Perhaps there will be off-the-shelf parsing tools, but how much of a benefit will this be? The problem to my mind is that in current formats, e.g. DELTA and LucID, much information is implied by context. Thus, in 1010 0101 The taxa and character state numbers (identities) are implied by the position of the data bit in the matrix. In XML this information is verbosely explicit. Is the following true?: once upon a time, computers could represent but not efficiently analyse or process textual data, hence documents were stored as text but "data" were stored as matrices etc. Now, XML has blurred the boundary between these types of information ("textual" and "data") and we're exploring the implications of that blurring. But are there now no differences, and no further need for a matrix? A final point. It seems to me that the discussion so far has been dominated by computer nuts (no offence intended, we need you!), with relatively little input from the community of taxonomists whose needs this standard is supposed to serve. Most of these will be happy to use a well-crafted tool, but won't want to know the intricacies of XML, and will be put off if things are not quite straightforward. This is a danger. We may end up with a wonderful, sophisticated, state-of-the-art, cutting-edge standard that nobody uses. Cheers - k

1 0

Re: (XML)Vocabularies getting out of hand
by Kevin Thiele 02 Dec '99

02 Dec '99

Jean-Marc writes: >Just say for the moment that we'll have several vocabularies by large >domains: > > * biological descriptions (generalities) > * botany > * zoology > * Virology > * Microbiology > * geography > * ecology > >May the forgotten disciplines forgive me... They'll have their XML >vocabularies too. Getting messy - I can't imagine that there's much shared vocabulary between, say, mammals, insects, nematodes, sponges..., just to take a few groups from the "zoology vocabulary". And if you asked taxomomists in each of these fields to use a common vocab they'd complain about square pegs in round holes. I think this whole idea of a universalist vocabulary (even several universes) is becoming silly. Better to work on the generalities and let collaborators work up their own vocabularies, with all the mess that that will entail. If this thing ever takes off, I think there will be no way of avoiding hundreds of vocabularies, not any handful that we can determine. Leigh comments: >Just to comment here, one approach being taken by the XHTML working group >is to take modular approach with future version of the specification (XHTML >1.1. >onwards). This allows defined portions of the spec to be replaced with >vendor specific extensions, e.g. replace all the form elements with >some extended/amended syntax, or the same with tables, etc. > >Such an approach might benefit this effort, in that a core set of >descriptive markup for characters can be defined, and then allow >additional modular extensions for the differing disciplines. > >This has the advantage of providing a small initial spec and data model >which can then be extended. (an approach thats worth considering >even without modularity included in the final design). Even a core set for mammals, insects, nematodes, sponges... would probably be a null set, except for cellular characters. Cheers - k

1 0

Re: (RQT) Character and item hierarchy
by P.Hovenkamp 02 Dec '99

02 Dec '99

At 04:26 PM 01-12-99 +0100, Gregor Hagedorn wrote: >> There is a containment hierarchy (e. g. a flower contains petals) >which >> raises no problems in XML. > >I would like to challenge this assumption, which seems to be >generally accepted in the discussion here. (...) >Thus: there are multiple concurrent or competing hierarchies, which >may overlap. (...) >*** Call for more examples: Can anybody come up with more good >examples (perhaps from animals) as to why a fixed hierarchy in the >form of a feature path may be unpractical? Here's my example: In ferns, leaves are often highly divided, into pinnae, pinnules, segments and even lower order elements. In most descriptions, a clear hierarchy is implied: pinnae are part of leaf, and inherit the properties, pinnules are part of pinnae and inherit... Programmers may be tempted to hard-code this hierarchy. However, for many characters, an opposite hierarchy can be devised, which may be much more informative: ultimate segments have margin, veins, and properties of those, which are inherited in the penultimate elements, and then onwards to the leaf. Especially in species where degree of dissection varies strongly (they occur), this may be a much better way of descibing a fern leaf. Hard-coding the usual hierarchy of leaf, pinnae, pinnules may therefore stand in the way of understanding and describing fern morphology. Is that clear enough for non-fern people? Best wishes, Peter P. Hovenkamp Rijksherbarium, Leiden The Netherlands hovenkamp(a)rhbcml.leidenuniv.nl http://rulrhb.leidenuniv.nl/

1 0

Re: (RQT) Character and item hierarchy
by Robert A. (Bob) Morris 01 Dec '99

01 Dec '99

P.Hovenkamp writes: >[...] > > Programmers may be tempted to hard-code this hierarchy. Programmers who hard-code /anything/ should be sent back to school for re-education. Not only should software itself behave according to the principle of least amazement, but we know that this is best accomplished if the programmers never build in assumptions about how the clients will use the software or present the data. The biggest advantage of XML for this group is its impending ubiquity, not its formal structure, and (in my opinion) the discussion here should return to discussing problems, not solutions because that ubiquity will make the community want to use it whether it is the best fit or not (just as has happened with HTML as a presentation vehicle). I am reminded of a talk I heard from a high-level member of the technical staff at Adobe a few years ago about an aborted three-year language development project. By his cleverly crafted presentation it became clearer and clearer to the audience that this new language had exactly the design features of another, just recently released language, he finally ended his talk by saying: "As we prepared to take the cloak off this project, we stepped into the street, looked up and saw that we were in the path of a huge, oncoming train. Its name was Java. We scrapped the project."

1 0

Re: (XML)Vocabularies getting out of hand
by Stuart G. Poss 01 Dec '99

01 Dec '99

: > Jean-Marc writes: > > >Just say for the moment that we'll have several vocabularies by large > >domains: > To which Kevin Thiele responds: > > Getting messy - I can't imagine that ... > I think this whole idea of a universalist vocabulary (even several > universes) is becoming silly. I believe the point was that we need to plan for a mechanism to handle consistently and with reasonable efficiency a wide variety of vocabularies in an extensible format, not that we would actually mandate any universal format. Any recommendation once implemented will, however, necessarily require us to forego some possible schemes. Lets just be sure that useful ideas are not lost as we collectively thrash things out. > Leigh comments: > > Even a core set for mammals, insects, nematodes, sponges... would probably > be a null set, except for cellular characters. > The fact that not every pair of taxa can be compared for a particular basis of comparison doesn't mean establishing a number of "vocabularies/name spaces" for particular levels of generality is a useless approach. Considering that such a common set or "core or intersection" might actually be several thousands of "protein" characters alone might suggest that this might be more fruitful than one might first suspect. I would agree, however, that it will likely take a potentially large number of "namespaces" to tease out all the interesting/potentially informative features in comparing widely disparate taxa, especially since one would like to characterize features along more than one basis of comparison (eg biochemical as well as morphological or ecological). We need not settle on a "final solution" as to name useage here, but only develop a mechanism that can be modified through time, according to conventional useage, so as to encompass a wide array of complex associations, some not yet discovered. If I have at least partially correctly digested the references on the RDF syntax correctly, we seem to need a mechanism to handle a wide variety of subject,predicate,object relations relating to both taxa and characters. The extent of these statments will likely arrange themselves according to the context (taxa) in question, at least to a large degree. Consequently it seems to me that we need some mechanism to specify taxa as a qualifier, or at least the "taxonomic range" (extent) over which the basis of comparison (character) implies. Likewise, we also need some mechanism (set of name spaces?) for communicating the basis(bases) of comparison implict for a given character (ie shape, molecular structure, fine structure, gross morphology, scale of comparison, physiology, etc.). >>>From my perspective, for statements of the form Taxon "A" HAS character "X", it would seem we need distinct "name spaces" (if my present conception of this term is the one being used by XML programmers) for both the taxa (subject) and character (object) part of the dialog. It remains unclear to me if this implies whether we may also need a potentially separate set of "name-spaces" for the reverse mappings (of the form character "X" can be found in/identified in/is representative of Taxon "A"). My suspicion is that we do, since such relations will usually not be isomorphic. However, this may depend upon the discipline (context). Likewise, the nature of the qualifiers for the predicates themselves (such as "is found in", "has", "has many of", "is a protein of", "has the shape of", "is a synapomorphy for" etc.) are even more unclear to me, as is whether there are any universal or common rules regarding such predicates that might usefully constrain the ways the taxon and character "vocabularies" are defined and used as an XML "grammar" potentially emerges. Also, it is also not yet clear to me how we can use XML to construct more complicated grammars implicit in the higher order grammatical relations of the type "Taxon "A" has character "X" according to Investigator(s) "Y" using observational method "Z" at time "T" (for rapidly evolving retroviruses or seaonal breeding colors, flowering condition. or molecular pathways perhaps). With respect to "messy" names-spaces/vocabularies I suggest we let the computers handle the large lists, multiple spellings, homonymy of terms, etc. They are particularly good at it and we will need a great many to characterize the millions of taxa, billions of molecules, and innumerable different bases for character definitions. I believe we need to concern ourselves more with how a consistent grammar for a "metadata language about characters and taxa" can be constructed/(implemented on the WWW) to facilitate consistent human input and precise meaning. Which specific character strings we ultimately use to formulate the tags or namespaces might take a bit longer to resolve, particularly should foreign colleagues feel strongly that they should be given unicode rather than soley ascii representations. Stuart

1 0