November 1999 - tdwg-content

Re: Types of data
by Leigh Dodds 24 Nov '99

24 Nov '99

> I too would argue that what we seek is a system that distinguishes between > "observations", measured in some sense (either directly using > some device), or perhaps "mapped" using known or widely used conventions and > higher level abstractions that often pass as qualitative "data", (ie leaf > "ovate" or state "relatively derived" or "1"). Would it be too simplistic to simply allow the format to express the source (qualitative/quantitative, or a more detailed description) or a particular data set? Deciding which sources you trust, value, or otherwise wish to manipulate is an option you then have at the application level. i.e. "I wish only direct measures of leaf shape, and not a description such as 'ovate'. If you also wanted to map between these measures ("give me an approximate textual description given these measurements") then this again lies at the application level and can be achieved assuming the data format is expressive enough. > In any case we need a means to distinguish between these two [maybe more?] > "fundamental [?] types" of "data", while at formulate a searching/description > language at the same time rich enough to characterize reasonably precisely the > context in which the representations were made, as well as the objects themselves. Again allowing the format to express source of data origin and means of origin allows this kind of processing to be deferred to the next processing layer (the application, or additional metadata information). Or am I missing the point here? L.

1 0

Whoa!
by Kevin Thiele 24 Nov '99

24 Nov '99

By golly what a start! Great stuff! Having read this morning the small flurry of contributions from Mauro, Leigh, Don, Jean-Marc and Noel, it's obvious that there are many ideas out there just waiting to burst forth. Perhaps we need a bit of structure now (or perhaps later). I'd suggest that there are several issues that can be used to structure the discussion. 1. WHAT'S THE INITIAL LIMIT FOR WHAT WE'RE TRYING TO DO? We could try to describe the universe that we're operating in by choosing which of the following is most true. a) We want an interchange format for current (and future) descriptive data/identification programs. These include LucID, DELTA's CONFOR, IntKey etc (the programs), XID, Linnaeus, and others (we should perhaps fill in this list sometime). The idea of an interchange format would be simple - each program would continue to store data in their own native format, but would implement import/export functions to the common standard so users would not be stuck in one format. The format should be at least a sum of all the types of data stored by each. Note that under this model if a user stored data using one program's data editor (e.g. the DELTA Builder) then exported to another program (e.g. the LucID Player) there will be data loss (since not all data stored will be useable by the recipient) and data inadequacy (since the recipient may support data that can't be stored by the storer). But having an exchange standard will increase pressure from users on all developers to do some competitive bootstrapping, and perhaps also encourage someone to develop a direct builder for the exchange format. b) We want a new data format for storing, parsing and manipulating descriptive data, without reference to existing programs. This would perhaps not differ from a) above, except that it would start more from scratch. It could perhaps subsume a) if such a format could also be used as an interchange format. But given the suggestion from Jean-Marc: >- The main existing formats, if we count by the number of species >covered, >are not DELTA, or other Computer Taxonomists's inventions, they are the >semi-formal formats of the existing floras in plain text, which contain >the >vast majority of existing species; >- the priority is to put this existing material in XML (this can be done >using standard parsing techniques); it would perhaps not serve the function of a) and those of us with existing programs would not advance beyond square 1. If we go this route a whole new suite of software tools will be needed for handling this, turnkey systems at the very least to put tools in the hands of ordinary users who don't know and don't want to know the intricacies of XML. c) We want to go the Universe and subsume all existing descriptive data formats into one superformat. This seems to be behind the comments about subsuming NEXUS. I think we should tread warily here. My impression is that users of NEXUS are more than happy with their format, there is no pressure for change coming from there, and we could merrily try to subsume NEXUS but find that the NEXUS community would ignore our efforts thankyou very much I'm quite alright. Are the Maddisons/Swoffords/Donoghues of the world listening? What do you think? d) any other statements? 2. WHAT ARE THE BASELINE REQUIREMENTS FOR THE SYSTEM I agree completely that step 1 whatever our goal must be a formal requirements analysis. Perhaps this would best proceed by the group contributing to an evolving list (or is there a better way?). We could use Leigh's as a starting point +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ REQUIREMENTS FOR A DESCRIPTIVE DATA FORMAT 1. - ease of use (i.e. authoring) 2. - ease of processing (parsing, validating, reading, converting) 3. - ease of sharing (i.e. distribution) 4. - open-ness (i.e. proprietary/non-proprietary) 5. - ease of extensibility (i.e. ability to add more information cleanly at a later data) 6. - internationalization 7. - un-abiguity of data representation 8. - unlimited size of data sets? (i.e. any limitation on character names, lengths, item names, numbers, etc) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ I'd vote that it should be non-proprietary (i.e. belonging to no one developer - that's been one limitation on DELTA as a "standard") and unlimited. I'd also say that under ease of authoring we should consider a mythical taxonomist who has data stored in a spreadsheet - can they easily fudge that data into the format (this is important in deciding between e.g. tagged formats like XML for which the fudge would be difficult and matrix formats). 3. WHAT ARE THE CORE DATA ELEMENTS Here's the beginning of a list, mostly a summation of LucID and DELTA. Note that this assumes that we're not simply tagging up descriptions. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ TYPES OF DATA NEEDED FOR A DESCRIPTIVE DATA STANDARD State-level scores: absent normally present rarely present (value?) uncertain present by misinterpretation Character-level scores: not scored yet unknown inapplicable uncertain .....do we need all these? .....more? Extra data: free-text comments attached to state scores state-score links ("/" vs "&" in DELTA) Character types: Ordered Multistate Unordered Multistate Real Numeric Integer Numeric? Ordered Cyclic Multistate Free Text Real Numeric Data: Minimum Normal Low Normal High Maximum Mean? Standard Deviation? N? Set-level data links to other data sets - links to lower-level data e.g. specimen-level data to be summed into taxon-level data e.g. subkeys - links to higher-level data e.g. passing of information up and down taxonomic hierarchies e.g. superkeys Multimedia data links from both character and taxon elements to any form of multimedia including blob text documents ++++++++++++++++++++++++++++++++++++++++++++++++ Finally, I think we need to bear in mind at all times the practical difficulties in wanting one format that's optimal for all things. My experience has been that a format that's good for storing atomised natural-language descriptions is not good for storing interactive key data and vice versa. The idea of a universal data store that can be used for many different purposes is a seductive one, and hence needs to be subject to frequent reality checks. Cheers - k

1 0

Re: so far...
by Eric Zurcher 24 Nov '99

24 Nov '99

At 19:06 23/11/1999 -0500, Susan Farmer wrote: >>From: Noel Cross <noelc(a)oeb.harvard.edu> >> >>Nexus is actually an extensible format so almost anything could be >>included. >> >>Unfortunately there is still doesn't seem to be an online description of >>Nexus, though the standard was printed in the journal Taxon a couple of >>years ago. > >Are you sure? I thought it was Systematic Botany. > Actually you're both wrong <G>. The formal description of the NEXUS format was in Systematic Biology. The full citation is: Maddison, D.R., Swofford, D.L. and W.P. Maddison. 1997. NEXUS: An extensible file format for systematic information. Syst. Biol. 46(4):590-621. For what it's worth, the paper itself refers to a Web site (http://phylogeny.arizona.edu/nexus/nexus.html) for "additions and proposed additions to the format", but nothing seems to have been placed there, as yet (though the site does provide "pseudo-code" for a NEXUS parser). Eric Zurcher CSIRO Division of Entomology Canberra, Australia E-mail: ericz(a)ento.csiro.au

1 0

Re: so far...
by Mauro J. Cavalcanti 24 Nov '99

24 Nov '99

Noel Cross wrote: > I know nothing about XDF. Mauro once pointed out this reference though: > > White, R.J. & Allkin, R. 1992. A language for the definition and exchange > of biological data sets. Mathl. Comput. Modelling, Vol. 16, No. 6/7, > pp. 199-223. This is the "official" publication for the XDF format, and is much less detailed than the equivalent publication for NEXUS. I have however a few detailed documents of the XDF specification, that were given to me by Bob Allkin, some years ago. As far as I know, XDF has not been used outside the scope of the ILDIS project, and I don't know if it is still being maintained by White & Allkin (I will ask to Bob, as he is now working here in Brazil). On peculiarity of this format is that it is intended to be used for transfering data between relational databases, as NEXUS is directed to the representation and transfer of phylogenetic datasets. I will be glad to send a copy of this documentation to someone in this list who could distribute it to anyone interested in it (maybe I could send it to Gregor?). PS Just as an aside, do this informal (and intended to be fun) treatment offends you (or anyone else in this list?). > ..."Computer Taxonomists" -- ouch! Best wishes, -- + - - - - - - - - - - - - Mauro J. Cavalcanti - - - - - - - - - - - - + | Setor de Paleovertebrados, Departamento de Geologia e Paleontologia | | Museu Nacional do Rio de Janeiro | | Quinta da Boa Vista, 20940-040, Rio de Janeiro, RJ, BRASIL | | E-mail: maurobio(a)acd.ufrj.br | | Home Page: http://read.at/digitax/personal.html | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + "Life is complex. It consists of real and imaginary parts."

1 0

Re: Types of data
by P. F. Stevens 24 Nov '99

24 Nov '99

>One of the data types that I would like to see modeled is a "matrix type" >data type. Consider the leaf shape. When I made up my DELTA data set, >I turned leaf shape into *3* characters -- one for leaf width, one for >the position of the widest shape, and a 3rd for the contour of the >sides. (narrow, broad; ovate, obovate; rounded, parallel). As the >basis for this division, I used the IAPT plain shape chart -- a two >dimensional representation. > >When you have a shape that varies from narrowly ovate to broadly obovate -- >how can you model that in *one* character? Variance from narrowly ovate >to narrowly obovate is possible just as variance from narrowly ovate to >narrowly obovate. > >You would define the data type by number and size of dimensions. (Two >would probably be sufficient for most cases). > Of course, the IAPT shapes are not "real", they are conventions (the actual circumscription of the various shapes was decided at a meeting back in the '50s, I think). As affording a communication device, the IAPT shape chart is excellent, and exactly what is needed (see below), as a means for measuring the world, it is definitely inferior to metrics such as millimeters and degrees. The states, the 0s and 1s, of phylogenetic analyses, are treated far more as IAPT shape charts than as the results of measured and analysed variation for my liking. Ideally - and I stress "ideally" - basic taxonomic data should be metric-type data (and images of things like pollen surface, perhaps wood anatomy) linked to specimen, rather as with d.n.a., where a sequence is deposited in GenBank and lnked to a specimen. I am not suggesting that all our taxonomic work should be so anchored, but I am suggesting that far more than at present should be so treated. Many morphological data sets consists of little more than unsubstantiated assertions about the world. If, however, somebody asked me for a flora treatment of some 'well" understood group for N. Amerias, I would probably have no reasonable alternative (because one's life is limited...) than to use whatever communication conventions exist for what I am looking at. (However, if a group had been monographed following more ideal standards, think how easy it would be to produce local accounts of variation, not to mention distribution maps, etc., etc.) And thinking of things like terms used in describing organisms, there certainly seems room for either synonymy tables and/or for more IAPT-type conventions. This is particularly true of genus- and family-"level" descriptions in plants - for teaching purposes, I am reading descriptions of genera and families in an edited compendium, and there may be three or four words for what is arguably the same thing, or terms that acutally make it more, not les, difficult, to get to grips with the basic variation. Peter S.

1 0

Re: Types of data
by Stuart G. Poss 24 Nov '99

24 Nov '99

I almost entirely agree with Peter. Most existing formats deal not with data per se, but rather with representations of data. 0's and 1's in phylogenetic analysis are an obvious example. While either value could represent a particular molecular base pair substitution, it need not and will, almost certainly, rarely reflect actual measurement of a particular specimen or series of specimens, even though such information would be required in some measure or estimation to actually reach the "higher-order inference". I too would argue that what we seek is a system that distinguishes between "observations", measured in some sense (either directly using some device), or perhaps "mapped" using known or widely used conventions and higher level abstractions that often pass as qualitative "data", (ie leaf "ovate" or state "relatively derived" or "1"). Because science requires assumptions be made, it is clear that in many areas, such representations are routinely assumed, with much or relatively little justification, as the issues require. Consequently, our "matrix" would be extremely sparse should one insist that only "measured variables" be specified as "character data" to reach the abstractions needed to conduct some kinds of investigations (ie. pooling museum records for a given species, whose taxonomy is established on the basis of morphology). Delta and Nexus serve well in variety of contexts. However, one would also like to be able to request the "intersection" of a variety of "weakly associated" or "semi-structured" information that might facilitate determining what the distribution and taxonomic conclusions might be for select sub-populations measured for a number of potentially widely different "features/properties/potential synonyms". To use the XML analogy, perhaps we need a tag language that "wrap" "representations of data", as well as other tags that indicate the circumstances under which "measured variables" were actually taken (eg. measurements taken from landmarks that may subtly differ among investigators; device used; preparation methods, etc.). In any case we need a means to distinguish between these two [maybe more?] "fundamental [?] types" of "data", while at formulate a searching/description language at the same time rich enough to characterize reasonably precisely the context in which the representations were made, as well as the objects themselves. If I disagree with Peter, it is that for at least a set of morphological features that can be imaged, we are rapidly approaching a day when it will be possible to measure thousands of objects each for thousands of measures quickly. For these, automated, well-defined definitions of shape will no longer be an issue (except perhaps which method/measure is most useful for a particular or general purpose). Rather, the issue is how can we set up a system of describing such measured data that allows us to evaluate it against taxonomic and morphological conclusions reached in the past using other methods, not to mention compare it against other molecular, developmental, physiological, and ecological data collected using a wide variety of techniques collected in the past and to be collected in the future. Also, comparison of methods for taking such data will become increasingly important as the devices (hardware and software) become ever more sophisticated and our language or interpreting their output becomes increasingly precise. Perhaps a useful approach might be to evaluate various "data types" separately to establish the appropriate set of "context" tags (the essence of DELTA, LUCID, NEXUS and related approaches), while also seeking to better understand the nature of the "conceptual wrappers" that will be needed to associate (tag) different contexts. At least such an approach might permit comparison of the controlled vocabularies of alternative tagging methods where there is content and concept overlap. Such an approach might also permit us to assess just what kinds of associations we need to be able to make, and hence what kind of language do we need to construct to permit "multi-dimensional" extensions. "P. F. Stevens" wrote: Of course, the IAPT shapes are not "real", they are conventions (the actual > circumscription of the various shapes was decided at a meeting back in the > '50s, I think). ...

1 0

Re: so far...
by Leigh Dodds 24 Nov '99

24 Nov '99

> As far as Delta goes, I think that the data are fairly close to what > taxonomists have been encoding into traditional taxonomic descriptions. > 'Success' is pretty good. > > Nexus can include charater/state data but also other things such as ways > to describe phylogenetic trees in the so-called "New Hampshire" format: > > ((raccoon:19.19959,bear:6.80041):0.84600,((sea_lion:11.99700, > seal:12.00300):7.52973,((monkey:100.85930,cat:47.14069):20.59201, > weasel:18.87953):2.09460):3.87382,dog:25.46154); If the basic format contains the relevant data, then a phylogenetic tree can be constructed from that data - so I'd have seen the above as something that should be derived from your data set rather than stored within it. I assume "New Hampshire" = "Newick" format? If so, heres an XML equivalent. <!ELEMENT label (#PCDATA)> <!ELEMENT title (#PCDATA)> <!ELEMENT branch (label?, branch*)> <!ATTLIST branch length CDATA #IMPLIED> <!ELEMENT newick (branch?)> Just something I was toying with before. > I sense that application developers who use Nexus are pretty happy with > the format since is was a collaborative effort to begin with, and that the > Nexus community would not see a great benefit in adopting a new data > format. Anybody else have this impression? Given this, and other similar comments on the list, I'd suggest that an additional requirement of any new format (if thats the way things go) is that it provides backwards-compatibility, as far as possible with other formats. Granted this might well be lossy, but given the numbers of users of Nexus, and the amount of software currently available, making efforts to provide for data conversion between formats means that we don't have to start completely from scratch. The first thing I did with XDELTA was provide a stylesheet with produce DELTA files. The same could be achieved for other formats. > > For example, how could retrieval of descriptive information across a > > department or institution be facilitated? Is it because of the sheer > > flexibility in how the characters can be defined using DELTA, > that querying > > across projects is difficult to say the least, unless the > character set is > > global that is (and therefore with a lot of redundacy)? > > I agree. XML only solves the syntax problem. The semantics problem can't > be solved as easily with technology, but would rely either on community > agreement, or on extensive mappings between various ontologies. I think. Definitely, and this is an important point to make. XML is an enabler, its not a magic-bullet. What types of problems have people met in attempting to share characters between research efforts? At a basic level, linking withing and between documents and data is easy. Its defining the semantics of this, and ensuring that the data retrieved is meaningful. > > This on the face of it would seem to map onto an XML > element/attribute/value > > schema pretty well. Would that help define more closely how we construct > > characters and maybe even prove universally applicable for all character > > types? > > Great question. Leigh? Anyone? :-) Sure element/attribute/value, but also element/element/content i.e. <leaf shape="obvate" /> or <leaf> <shape>obvate</shape> </leaf> Or did I miss something? > > Could one constrain further by expressing within the schema the > hierarchical > > relationships between the elements(eg 'blade' and 'petiole' as child > > elements of leaf') or would the introduction of terminology into the > > 'standard' be a step too far? > > This would be The Lexicon, no? It does seem to go beyond the task of > making a file format for data transfer. But it gets my vote > nevertheless. I'd suggest that something like this (expressing meta-data relationships amongst elements) could be layered on top of a basic data description format. RDF might provide the facility to do this effectively. I need to revisit the spec. cheers, L.

1 0

Correct reference for NEXUS
by Mauro J. Cavalcanti 24 Nov '99

24 Nov '99

"Susan B. Farmer" wrote: > > >From: Noel Cross <noelc(a)oeb.harvard.edu> > > > >Nexus is actually an extensible format so almost anything could be > >included. > > > >Unfortunately there is still doesn't seem to be an online description of > >Nexus, though the standard was printed in the journal Taxon a couple of > >years ago. > > Are you sure? I thought it was Systematic Botany. Dear Computer Taxonomists, The article by Maddison et al. on NEXUS didn't appeared either in "Taxon" or "Systematic Botany", but in "Systematic Biology" (1997). I posted the complete reference to the DELTA-L discussion list, about a month ago... Best wishes, -- + - - - - - - - - - - - - Mauro J. Cavalcanti - - - - - - - - - - - - + | Setor de Paleovertebrados, Departamento de Geologia e Paleontologia | | Museu Nacional do Rio de Janeiro | | Quinta da Boa Vista, 20940-040, Rio de Janeiro, RJ, BRASIL | | E-mail: maurobio(a)acd.ufrj.br | | Home Page: http://read.at/digitax/personal.html | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + "Life is complex. It consists of real and imaginary parts."

1 0

Re: so far ...
by Susan B. Farmer 24 Nov '99

24 Nov '99

>From: Leigh Dodds <ldodds(a)ingenta.com> > >> I sense that application developers who use Nexus are pretty happy with >> the format since is was a collaborative effort to begin with, and that the >> Nexus community would not see a great benefit in adopting a new data >> format. Anybody else have this impression? > >Given this, and other similar comments on the list, I'd suggest that an >additional requirement of any new format (if thats the way things go) >is that it provides backwards-compatibility, as far as possible with >other formats. > IMHO, this inter-compatibility of data sets is one of the Big Issues that needs to be addressed. I (initially) put all of my data into DELTA and then converted it to Nexus. The Nexus file has been heavily modified (and many of my character and state names truncated in the bargin). Until very recently, there was no way to take *this* file back into DELTA. But more than that; all of the continuous data in the original file (aka measurements) are now gone and will have to be added back into the DELTA data set by hand. :-( Why not extend the current standards that if there are data types that their particular application doesn't recognize, they stay around -- just ignored. MacClade does that even with Nexus data (maybe the newest one doesn't); but if I save my file from within MacClade, if I want to run PAUP again, there are commands that I must reinter into the file because MacClade has removed them because it didn't understand them. Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Re: list conventions
by Susan B. Farmer 23 Nov '99

23 Nov '99

> >It's configured that way now. Peter Ok. A couple of the posts that I've replied to have wanted to go to the original author, and the others have gone to the list. I wasn't sure. Thanks, Susan > >> Before we get started too heavily on this list, can we change it so that >> all messages default to the list rather than the originator? >> We are trying to foster discussion. >

1 0