- tdwg-content - lists.tdwg.org

Re: Types of data
by P. F. Stevens 24 Nov '99

24 Nov '99

>One of the data types that I would like to see modeled is a "matrix type" >data type. Consider the leaf shape. When I made up my DELTA data set, >I turned leaf shape into *3* characters -- one for leaf width, one for >the position of the widest shape, and a 3rd for the contour of the >sides. (narrow, broad; ovate, obovate; rounded, parallel). As the >basis for this division, I used the IAPT plain shape chart -- a two >dimensional representation. > >When you have a shape that varies from narrowly ovate to broadly obovate -- >how can you model that in *one* character? Variance from narrowly ovate >to narrowly obovate is possible just as variance from narrowly ovate to >narrowly obovate. > >You would define the data type by number and size of dimensions. (Two >would probably be sufficient for most cases). > Of course, the IAPT shapes are not "real", they are conventions (the actual circumscription of the various shapes was decided at a meeting back in the '50s, I think). As affording a communication device, the IAPT shape chart is excellent, and exactly what is needed (see below), as a means for measuring the world, it is definitely inferior to metrics such as millimeters and degrees. The states, the 0s and 1s, of phylogenetic analyses, are treated far more as IAPT shape charts than as the results of measured and analysed variation for my liking. Ideally - and I stress "ideally" - basic taxonomic data should be metric-type data (and images of things like pollen surface, perhaps wood anatomy) linked to specimen, rather as with d.n.a., where a sequence is deposited in GenBank and lnked to a specimen. I am not suggesting that all our taxonomic work should be so anchored, but I am suggesting that far more than at present should be so treated. Many morphological data sets consists of little more than unsubstantiated assertions about the world. If, however, somebody asked me for a flora treatment of some 'well" understood group for N. Amerias, I would probably have no reasonable alternative (because one's life is limited...) than to use whatever communication conventions exist for what I am looking at. (However, if a group had been monographed following more ideal standards, think how easy it would be to produce local accounts of variation, not to mention distribution maps, etc., etc.) And thinking of things like terms used in describing organisms, there certainly seems room for either synonymy tables and/or for more IAPT-type conventions. This is particularly true of genus- and family-"level" descriptions in plants - for teaching purposes, I am reading descriptions of genera and families in an edited compendium, and there may be three or four words for what is arguably the same thing, or terms that acutally make it more, not les, difficult, to get to grips with the basic variation. Peter S.

1 0

Re: Types of data
by Stuart G. Poss 24 Nov '99

24 Nov '99

I almost entirely agree with Peter. Most existing formats deal not with data per se, but rather with representations of data. 0's and 1's in phylogenetic analysis are an obvious example. While either value could represent a particular molecular base pair substitution, it need not and will, almost certainly, rarely reflect actual measurement of a particular specimen or series of specimens, even though such information would be required in some measure or estimation to actually reach the "higher-order inference". I too would argue that what we seek is a system that distinguishes between "observations", measured in some sense (either directly using some device), or perhaps "mapped" using known or widely used conventions and higher level abstractions that often pass as qualitative "data", (ie leaf "ovate" or state "relatively derived" or "1"). Because science requires assumptions be made, it is clear that in many areas, such representations are routinely assumed, with much or relatively little justification, as the issues require. Consequently, our "matrix" would be extremely sparse should one insist that only "measured variables" be specified as "character data" to reach the abstractions needed to conduct some kinds of investigations (ie. pooling museum records for a given species, whose taxonomy is established on the basis of morphology). Delta and Nexus serve well in variety of contexts. However, one would also like to be able to request the "intersection" of a variety of "weakly associated" or "semi-structured" information that might facilitate determining what the distribution and taxonomic conclusions might be for select sub-populations measured for a number of potentially widely different "features/properties/potential synonyms". To use the XML analogy, perhaps we need a tag language that "wrap" "representations of data", as well as other tags that indicate the circumstances under which "measured variables" were actually taken (eg. measurements taken from landmarks that may subtly differ among investigators; device used; preparation methods, etc.). In any case we need a means to distinguish between these two [maybe more?] "fundamental [?] types" of "data", while at formulate a searching/description language at the same time rich enough to characterize reasonably precisely the context in which the representations were made, as well as the objects themselves. If I disagree with Peter, it is that for at least a set of morphological features that can be imaged, we are rapidly approaching a day when it will be possible to measure thousands of objects each for thousands of measures quickly. For these, automated, well-defined definitions of shape will no longer be an issue (except perhaps which method/measure is most useful for a particular or general purpose). Rather, the issue is how can we set up a system of describing such measured data that allows us to evaluate it against taxonomic and morphological conclusions reached in the past using other methods, not to mention compare it against other molecular, developmental, physiological, and ecological data collected using a wide variety of techniques collected in the past and to be collected in the future. Also, comparison of methods for taking such data will become increasingly important as the devices (hardware and software) become ever more sophisticated and our language or interpreting their output becomes increasingly precise. Perhaps a useful approach might be to evaluate various "data types" separately to establish the appropriate set of "context" tags (the essence of DELTA, LUCID, NEXUS and related approaches), while also seeking to better understand the nature of the "conceptual wrappers" that will be needed to associate (tag) different contexts. At least such an approach might permit comparison of the controlled vocabularies of alternative tagging methods where there is content and concept overlap. Such an approach might also permit us to assess just what kinds of associations we need to be able to make, and hence what kind of language do we need to construct to permit "multi-dimensional" extensions. "P. F. Stevens" wrote: Of course, the IAPT shapes are not "real", they are conventions (the actual > circumscription of the various shapes was decided at a meeting back in the > '50s, I think). ...

1 0

Re: so far...
by Leigh Dodds 24 Nov '99

24 Nov '99

> As far as Delta goes, I think that the data are fairly close to what > taxonomists have been encoding into traditional taxonomic descriptions. > 'Success' is pretty good. > > Nexus can include charater/state data but also other things such as ways > to describe phylogenetic trees in the so-called "New Hampshire" format: > > ((raccoon:19.19959,bear:6.80041):0.84600,((sea_lion:11.99700, > seal:12.00300):7.52973,((monkey:100.85930,cat:47.14069):20.59201, > weasel:18.87953):2.09460):3.87382,dog:25.46154); If the basic format contains the relevant data, then a phylogenetic tree can be constructed from that data - so I'd have seen the above as something that should be derived from your data set rather than stored within it. I assume "New Hampshire" = "Newick" format? If so, heres an XML equivalent. <!ELEMENT label (#PCDATA)> <!ELEMENT title (#PCDATA)> <!ELEMENT branch (label?, branch*)> <!ATTLIST branch length CDATA #IMPLIED> <!ELEMENT newick (branch?)> Just something I was toying with before. > I sense that application developers who use Nexus are pretty happy with > the format since is was a collaborative effort to begin with, and that the > Nexus community would not see a great benefit in adopting a new data > format. Anybody else have this impression? Given this, and other similar comments on the list, I'd suggest that an additional requirement of any new format (if thats the way things go) is that it provides backwards-compatibility, as far as possible with other formats. Granted this might well be lossy, but given the numbers of users of Nexus, and the amount of software currently available, making efforts to provide for data conversion between formats means that we don't have to start completely from scratch. The first thing I did with XDELTA was provide a stylesheet with produce DELTA files. The same could be achieved for other formats. > > For example, how could retrieval of descriptive information across a > > department or institution be facilitated? Is it because of the sheer > > flexibility in how the characters can be defined using DELTA, > that querying > > across projects is difficult to say the least, unless the > character set is > > global that is (and therefore with a lot of redundacy)? > > I agree. XML only solves the syntax problem. The semantics problem can't > be solved as easily with technology, but would rely either on community > agreement, or on extensive mappings between various ontologies. I think. Definitely, and this is an important point to make. XML is an enabler, its not a magic-bullet. What types of problems have people met in attempting to share characters between research efforts? At a basic level, linking withing and between documents and data is easy. Its defining the semantics of this, and ensuring that the data retrieved is meaningful. > > This on the face of it would seem to map onto an XML > element/attribute/value > > schema pretty well. Would that help define more closely how we construct > > characters and maybe even prove universally applicable for all character > > types? > > Great question. Leigh? Anyone? :-) Sure element/attribute/value, but also element/element/content i.e. <leaf shape="obvate" /> or <leaf> <shape>obvate</shape> </leaf> Or did I miss something? > > Could one constrain further by expressing within the schema the > hierarchical > > relationships between the elements(eg 'blade' and 'petiole' as child > > elements of leaf') or would the introduction of terminology into the > > 'standard' be a step too far? > > This would be The Lexicon, no? It does seem to go beyond the task of > making a file format for data transfer. But it gets my vote > nevertheless. I'd suggest that something like this (expressing meta-data relationships amongst elements) could be layered on top of a basic data description format. RDF might provide the facility to do this effectively. I need to revisit the spec. cheers, L.

1 0

Correct reference for NEXUS
by Mauro J. Cavalcanti 24 Nov '99

24 Nov '99

"Susan B. Farmer" wrote: > > >From: Noel Cross <noelc(a)oeb.harvard.edu> > > > >Nexus is actually an extensible format so almost anything could be > >included. > > > >Unfortunately there is still doesn't seem to be an online description of > >Nexus, though the standard was printed in the journal Taxon a couple of > >years ago. > > Are you sure? I thought it was Systematic Botany. Dear Computer Taxonomists, The article by Maddison et al. on NEXUS didn't appeared either in "Taxon" or "Systematic Botany", but in "Systematic Biology" (1997). I posted the complete reference to the DELTA-L discussion list, about a month ago... Best wishes, -- + - - - - - - - - - - - - Mauro J. Cavalcanti - - - - - - - - - - - - + | Setor de Paleovertebrados, Departamento de Geologia e Paleontologia | | Museu Nacional do Rio de Janeiro | | Quinta da Boa Vista, 20940-040, Rio de Janeiro, RJ, BRASIL | | E-mail: maurobio(a)acd.ufrj.br | | Home Page: http://read.at/digitax/personal.html | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + "Life is complex. It consists of real and imaginary parts."

1 0

Re: so far ...
by Susan B. Farmer 24 Nov '99

24 Nov '99

>From: Leigh Dodds <ldodds(a)ingenta.com> > >> I sense that application developers who use Nexus are pretty happy with >> the format since is was a collaborative effort to begin with, and that the >> Nexus community would not see a great benefit in adopting a new data >> format. Anybody else have this impression? > >Given this, and other similar comments on the list, I'd suggest that an >additional requirement of any new format (if thats the way things go) >is that it provides backwards-compatibility, as far as possible with >other formats. > IMHO, this inter-compatibility of data sets is one of the Big Issues that needs to be addressed. I (initially) put all of my data into DELTA and then converted it to Nexus. The Nexus file has been heavily modified (and many of my character and state names truncated in the bargin). Until very recently, there was no way to take *this* file back into DELTA. But more than that; all of the continuous data in the original file (aka measurements) are now gone and will have to be added back into the DELTA data set by hand. :-( Why not extend the current standards that if there are data types that their particular application doesn't recognize, they stay around -- just ignored. MacClade does that even with Nexus data (maybe the newest one doesn't); but if I save my file from within MacClade, if I want to run PAUP again, there are commands that I must reinter into the file because MacClade has removed them because it didn't understand them. Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Re: list conventions
by Susan B. Farmer 23 Nov '99

23 Nov '99

> >It's configured that way now. Peter Ok. A couple of the posts that I've replied to have wanted to go to the original author, and the others have gone to the list. I wasn't sure. Thanks, Susan > >> Before we get started too heavily on this list, can we change it so that >> all messages default to the list rather than the originator? >> We are trying to foster discussion. >

1 0

Re: standard sloppy values (was) The end of DELTA?
by Susan B. Farmer 23 Nov '99

23 Nov '99

>This came up in several different forms already, but any hierarchy should be >able to be terminated at any level without generating a syntax error. >Semantics >need to be defined based on the depth of the hierarchy in any particular >instance. This will help to accommodate the large body of semi-structured text >descriptions that already exist and were discussed earlier. For example, the >"Leaf" node might be defined to contain a list of optional properties (or >sub-nodes) such as size, shape... but the standard should allow for free text >description as well, perhaps by allowing for a <NaturalLangage> as one of the >options, sort of an "other" category but with definable semantics. One of the things that I did when I was coding up my original DELTA dataset (since I was wortking from published floras, etc) was to code a text character at the end of the characters for a given organ (e.g., leaf) that included the text of the protolog, or whatever textual description that I was working from . This wasn't necessarily for searching or information retrieval; but rather it was to allow me to *later* come back and verify the data without having to go did the original source off the shelf. Lazy woman's option. Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Re: so far...
by Susan B. Farmer 23 Nov '99

23 Nov '99

>From: Noel Cross <noelc(a)oeb.harvard.edu> > >Nexus is actually an extensible format so almost anything could be >included. > >Unfortunately there is still doesn't seem to be an online description of >Nexus, though the standard was printed in the journal Taxon a couple of >years ago. Are you sure? I thought it was Systematic Botany. Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Re: The end of DELTA?
by Susan B. Farmer 23 Nov '99

23 Nov '99

>Could one constrain further by expressing within the schema the hierarchical >relationships between the elements(eg 'blade' and 'petiole' as child >elements of leaf') or would the introduction of terminology into the >'standard' be a step too far? > Putting on my programmer hat here, those kinds of things should be about to be included in whatever XML standards were set up relatively transparent to the user. For instance, "leaf" by default would contain certain properties (size, shape, petiole, base, apex, number, arrangement, location) that the user could code or mark as non-existant withought having to worry aout the "parent/child" or other object relationship. Folk who want to *write* software that uses the XML (or whatever) standard would be the ones to worry about those issues. Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Types of data
by Susan B. Farmer 23 Nov '99

23 Nov '99

>From: Leigh Dodds <ldodds(a)ingenta.com> > >So, what do people see as the basic requirements for this kind >of format? > >- ease of use (i.e. authoring) >- ease of processing (parsing, validating, reading, converting) >- ease of sharing (i.e. distribution) >- open-ness (i.e. proprietary/non-proprietary) >- ease of extensibility (i.e. ability to add more information cleanly at a >later >data) >- internationalization >- un-abiguity of data representation >- unlimited size of data sets? (i.e. any limitation on character names, >lengths, item names, numbers, etc) > >What types of data need to be modelled by the format? (this can >be a post-requirements gathering step, but some consideration needs >to be given early on to measure the 'success' of the current >formats) > >- what types of characters? (real, integer, text, etc) >- what types of data (text, images, other formats?) > One of the data types that I would like to see modeled is a "matrix type" data type. Consider the leaf shape. When I made up my DELTA data set, I turned leaf shape into *3* characters -- one for leaf width, one for the position of the widest shape, and a 3rd for the contour of the sides. (narrow, broad; ovate, obovate; rounded, parallel). As the basis for this division, I used the IAPT plain shape chart -- a two dimensional representation. When you have a shape that varies from narrowly ovate to broadly obovate -- how can you model that in *one* character? Variance from narrowly ovate to narrowly obovate is possible just as variance from narrowly ovate to narrowly obovate. You would define the data type by number and size of dimensions. (Two would probably be sufficient for most cases). Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0