November 1999 - tdwg-content

Re: Types of data
by Don Kirkup 25 Nov '99

25 Nov '99

Kevin Thiele wrote; > This worries me. Is the entity/property/value schema restricted to 3 > levels? (ditto for XMLs element/attribute/value). If so, it seems too much > of a straightjacket, especially if we want to express a hierarchy of > elements like this. No not at all, theoretically you can have as many levels in the hierarchy you like; element/element/...nth element/attribute/value > What to do about > leaf/lamina/abaxial_surface/vein_islands/indumentum/density. One (RDF centric) way to model this would be to treat leaf, lamina, abaxial_surface, vein_islands, indumentum as nested elements, and an attribute called density which will have a value (let's say "sparse"); <My_description> <leaf> <lamina> <vein_islands> <indumentum density="sparse"/> </vein_islands> </lamina> </leaf> </My_description> or more in line with the conventional (DELTA, NEXUS) approach, simply as a single element with an attribute (called density) and a value; <My_description> <leaf_lamina_abaxial_surface vein_islands_indumentum density="sparse"/> </My_description> In the latter case you've lost a lot of contextual information, but the approach (elements have properties that have values) might still offer improved consistency in definition of descriptors . Many of us, probably quite naturally, end up defining characters in this way, but there are may examples to the contrary, and none of the existing formats seem either expect or enforce this discipline. don

1 0

Re: Types of data
by Don Kirkup 25 Nov '99

25 Nov '99

Aren't observations made by looking at specimens and measurements taken from 'organs' (or whatever constituent parts of specimen)? The taxonomic philosophers might well argue that all characters are abstractions, that the concept that a leaf has shape is an abstraction. In terms of the 'reality' of statement you might make about the shape of a leaf, it doesn't really matter whether you describe shape using standard terms or as set of coordinates or as an elliptic fourier coefficient. Surely the way you measure an attribute depends on what you are going to use the measurements for, ultimately that's a question of precision. If there is a possibility that there will be a variety of uses (reuse) for the data, then an indictation of the degree of precision of the measurement would be highly desirable. Isn't it also important to distinguish whether the description refers to an actual leaf (ie. that leaf there, on that specimen is ovate) or whether you are making a generalisation about a taxon (ie leaves ovate to obovate)? A taxon cannot have a leaf shape, only leaves can, and an actual leaf does not usually vary in shape except through time. So if the reality is; "I looked at two specimens; from specimen x I measured 3 leaves and their shapes were ovate, ovate and obovate; from specimen y I measured 2 leaves, the first was ovate, the second obovate" Then why not say so? <my_taxon_description> <specimen> <leaf shape="ovate"/> <leaf shape="ovate"/> <leaf shape="obovate"/> </specimen> <specimen> <leaf shape="ovate"/> <leaf shape="obovate"/> </specimen> </my_taxon_description> For a generalised statement about my taxon, all I need is an aggregate function for the listed values of leaf shape. It seems that very few 'real' observations are formally recorded but we have a lot of generalisations. "leaves ovate to obovate, 5--31 mm long" What can be deduced about the relationship between leaf shape and leaf length? At what point in the exercise or recording observations should the decision be made that this information either is or is not important? <my_taxon_description> <specimen> <leaf shape="ovate" length_mm="10"/> <leaf shape="ovate" length_mm="8"/> <leaf shape="obovate" length_mm="25"/> </specimen> <specimen> <leaf shape="ovate" length_mm="5"/> <leaf shape="obovate" length_mm="31"/> </specimen> </my_taxon_description> Maybe the ovate leaves are all immature, or there's some enivironmental factor. Maybe I don't want to know, but maybe I do. don > Leigh Dodds writes: > > >> I too would argue that what we seek is a system that > distinguishes between > >> "observations", measured in some sense (either directly using > >> some device), or perhaps "mapped" using known or widely used > conventions > >and > >> higher level abstractions that often pass as qualitative > "data", (ie leaf > >> "ovate" or state "relatively derived" or "1"). > > > >Would it be too simplistic to simply allow the format to express > >the source (qualitative/quantitative, or a more detailed description) > >or a particular data set? Kevin Thiele wrote; > Too simplistic, I think. There will surely be many cases where in one data > set some characters will be "real" (i.e. low-level) observations while > others will be abstractions. For instance, sometimes it will be viewed as > worthwhile for the taxonomist to capture real data (leaf shapes are fairly > easy to do), other times anything other than abstraction is practically > impossible (try working out how to capture indumentum characteristics > quantitatively).

1 0

Looking for the list manager
by Mauro J. Cavalcanti 25 Nov '99

25 Nov '99

Dear Colleagues... I am in need of contacting directly the administrator of this discussion list. Could anyone kindly point her/his name and e-mail address to me? Thanks in advance for any assistance you can provide. Regards, -- + - - - - - - - - - - - - Mauro J. Cavalcanti - - - - - - - - - - - - + | Setor de Paleovertebrados, Departamento de Geologia e Paleontologia | | Museu Nacional do Rio de Janeiro | | Quinta da Boa Vista, 20940-040, Rio de Janeiro, RJ, BRASIL | | E-mail: maurobio(a)acd.ufrj.br | | Home Page: http://read.at/digitax/personal.html | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + "Life is complex. It consists of real and imaginary parts."

1 0

Re: so far ...
by Jim Croft 25 Nov '99

25 Nov '99

> This is typical of the level of malpractice of those who seem to be taking the > running with this discussion. Can you imagine being so clumsy and foolhardly > with hard-won data? I despair of this debate. Patience Nick - new lists take a while to settle down as participants try and establish a common agenda, and people strut personal favorite products and ideologies. Like Bernie, I am waiting for the discussion to focus itself. I had initially thought we were looking at a common and comprehensive (interchange?) format for biological descriptive data, perhaps involving an information model of the topic we are dealing with, and importantly, its boundaries. But all too quickly we have got to the level of all things to all people end-products, software and a degree of daunting complexity that we could probably do without at this stage. Approaching things from an 'if it aint broke dont fix it' point of view, is someone in a position to enunciate/tabulate exactly what it is we are trying to achieve and the shortcomings/limitations of exiting formats in reaching this goal? Having done this we might be better able to partition things into managable and achievable lumps. I was not at the Harvard meeting so I'm a bit reluctant to stick an oar in, but since when has ignorance been a reason not to have an opinion... :) All the suggestions of 'do it this way using this', while interesting and educational, are tending to obscure rather than clarify what it is we are tring to achieve. Well, I am getting a bit lost, and I love this data stuff... jim

1 0

Re: so far ...
by Nick Lander 25 Nov '99

25 Nov '99

Terry, Alex, Mike, Ben This is typical of the level of malpractice of those who seem to be taking the running with this discussion. Can you imagine being so clumsy and foolhardly with hard-won data? I despair of this debate. Nicholas Lander Principal Research Scientist Biological Information Group CALMScience ---------- Original Text ---------- From: "Susan B. Farmer" <sfarmer(a)GOLDSWORD.COM>, on 24/11/1999 9:31 PM: >From: Leigh Dodds <ldodds(a)ingenta.com> > >> I sense that application developers who use Nexus are pretty happy with >> the format since is was a collaborative effort to begin with, and that the >> Nexus community would not see a great benefit in adopting a new data >> format. Anybody else have this impression? > >Given this, and other similar comments on the list, I'd suggest that an >additional requirement of any new format (if thats the way things go) >is that it provides backwards-compatibility, as far as possible with >other formats. > IMHO, this inter-compatibility of data sets is one of the Big Issues that needs to be addressed. I (initially) put all of my data into DELTA and then converted it to Nexus. The Nexus file has been heavily modified (and many of my character and state names truncated in the bargin). Until very recently, there was no way to take *this* file back into DELTA. But more than that; all of the continuous data in the original file (aka measurements) are now gone and will have to be added back into the DELTA data set by hand. :-( Why not extend the current standards that if there are data types that their particular application doesn't recognize, they stay around -- just ignored. MacClade does that even with Nexus data (maybe the newest one doesn't); but if I save my file from within MacClade, if I want to run PAUP again, there are commands that I must reinter into the file because MacClade has removed them because it didn't understand them. Susan Farmer sfarmer(a)goldsword.com Botany Department, University of Tennessee http://www.goldsword.com/sfarmer/Trillium

1 0

Dallwitz's comparisons
by Mauro J. Cavalcanti 25 Nov '99

25 Nov '99

Dear Computer Taxonomists, I have just taken a look at the documents recently announced by Mike Dallwitz. They look nice, and the comparison of formats for descriptive data seems especially useful. However, I have noticed the complete absence of the XDF format. It seems that if we are engaging in a comprehensive comparison of formats for biological descriptive data, XDF deserves a little more attention. Perhaps it has not been widely used, but notwithstanding one of its main applications has been quite impressive - the importing of a large part of the Missouri Botanical Garden TROPICOS database to the use of the ILDIS Project. And this happened more than a decade ago, in the pre-Web era, and before all the current interest in distributed databases and metadata. I hope to find some time in the near future for trying to put some documentation on XDF on the Web, for our general use. Regards, -- + - - - - - - - - - - - - Mauro J. Cavalcanti - - - - - - - - - - - - + | Setor de Paleovertebrados, Departamento de Geologia e Paleontologia | | Museu Nacional do Rio de Janeiro | | Quinta da Boa Vista, 20940-040, Rio de Janeiro, RJ, BRASIL | | E-mail: maurobio(a)acd.ufrj.br | | Home Page: http://read.at/digitax/personal.html | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + "Life is complex. It consists of real and imaginary parts."

1 0

Re: Types of data
by Jean-Marc Vanel 25 Nov '99

25 Nov '99

You wrote: leaf/lamina/abaxial_surface/vein_islands/indumentum/density This is exactly an XPath ( http://www.w3.org/TR/xpath.html ) query, it will select all <density> elements children of <indumentum> elements children of ... etc. Other example: lamina[ancestor(species)//petal[contains(./text(),'white')]] will select <lamina> elements descendant of <species> elements, whose <petal> element's text content has 'white' string in it. The whole thing is to set a botanical vocabulary, allowing indumentum and disallowing (e.g.) pilosity. This is a secondary issue wether this vocabulary will be a DTD, an XML Schema, or a RDF Schema. Anyway, there are tools like XML Authority of Extensibility to translate beetwen these formats (not yet RDF Schema, alas), so we can provide all Schema dialects. This is what most business Consortia do. For the relational database fans, it is easy to export a many-columns table using a XSLT (http://www.w3.org/TR/xslt.html) transform, with columns names exactly like this: leaf/lamina/abaxial_surface/vein_islands/indumentum/density For the DELTA fans, the principle is to associate a character number to each leaf, like leaf/lamina/abaxial_surface/vein_islands/indumentum/density of the document Schema. It can be done even without DTD/Schema, see http://jmvanel.free.fr/Samples/DisplayDescriptions/speciesGUI.htm, where I list all the unique nested tags names in the document; this is what I call the implicit Schema. I do Javascript with the Document Object Model. I have proposals to extend the DELTA system; this will be another thread. see for more details: http://jmvanel.free.fr/ Regards Jean-Marc Kevin Thiele a écrit : > Is the entity/property/value schema restricted to 3 > levels? ........... > What to do about > leaf/lamina/abaxial_surface/vein_islands/indumentum/density. > I agree that a hierarchically structured schema may be wonderful, but in a > biological context I wouldn't like to see it constrained at 3 levels just > because that's what existing computer formats do. > > Cheers - k

1 0

Re: Types of data
by Bernie Hyland. 25 Nov '99

25 Nov '99

At 09:52 AM 11/24/1999 -0500, you wrote: >>One of the data types that I would like to see modeled is a "matrix type" >>data type. Consider the leaf shape. When I made up my DELTA data set, >>I turned leaf shape into *3* characters -- one for leaf width, one for >>the position of the widest shape, and a 3rd for the contour of the >>sides. (narrow, broad; ovate, obovate; rounded, parallel). As the >>basis for this division, I used the IAPT plain shape chart -- a two >>dimensional representation. >> >>When you have a shape that varies from narrowly ovate to broadly obovate -- >>how can you model that in *one* character? Variance from narrowly ovate >>to narrowly obovate is possible just as variance from narrowly ovate to >>narrowly obovate. >> >>You would define the data type by number and size of dimensions. (Two >>would probably be sufficient for most cases). >> > >Of course, the IAPT shapes are not "real", they are conventions (the actual >circumscription of the various shapes was decided at a meeting back in the >'50s, I think). As affording a communication device, the IAPT shape chart >is excellent, and exactly what is needed (see below), as a means for >measuring the world, it is definitely inferior to metrics such as >millimeters and degrees. The states, the 0s and 1s, of phylogenetic >analyses, are treated far more as IAPT shape charts than as the results of >measured and analysed variation for my liking. Ideally - and I stress >"ideally" - basic taxonomic data should be metric-type data (and images of >things like pollen surface, perhaps wood anatomy) linked to specimen, >rather as with d.n.a., where a sequence is deposited in GenBank and lnked >to a specimen. I am not suggesting that all our taxonomic work should be >so anchored, but I am suggesting that far more than at present should be so >treated. Many morphological data sets consists of little more than >unsubstantiated assertions about the world. If, however, somebody asked me >for a flora treatment of some 'well" understood group for N. Amerias, I >would probably have no reasonable alternative (because one's life is >limited...) than to use whatever communication conventions exist for what I >am looking at. (However, if a group had been monographed following more >ideal standards, think how easy it would be to produce local accounts of >variation, not to mention distribution maps, etc., etc.) > >And thinking of things like terms used in describing organisms, there >certainly seems room for either synonymy tables and/or for more IAPT-type >conventions. This is particularly true of genus- and family-"level" >descriptions in plants - for teaching purposes, I am reading descriptions >of genera and families in an edited compendium, and there may be three or >four words for what is arguably the same thing, or terms that acutally make >it more, not les, difficult, to get to grips with the basic variation. > > >Peter S. > > THANKS FOR BRINGING REALITY INTO THE DISCUSSION. I HAVE ONLY JUST JOINED THIS GROUP, SO I AM NOT PREPARED TO TRUMPET MY THOUGHTS TO THE WHOLE WORLD AT THIS STAGE. --------------------------------------------------------------------------- bernie.hyland(a)tfrc.csiro.au Bernie Hyland, CSIRO - Plant Industry, Box 780, ATHERTON 4883 Queensland, Australia. Ph 07 40918805 Int. 61 740918805 FAX 07 40918888 Int. 61 740918888

1 0

Some useful references
by Mauro J. Cavalcanti 25 Nov '99

25 Nov '99

Dear Computer Taxonomists, Here are some references that you may find potentially useful to throw some light on the recent (quite impressive!) discussions. From a more theoretical viewpoint, I especially recomend the papers by Diederich. Allkin, R. & White, R.J. 1989. XDF: A Language for the Definition and Exchange of Biological Data Sets. Description & Manual. Version 3.3. Unpublished. 49 p. Diederich, J. 1997. Basic properties for biological databases: character development and support. Mathematical and Computer Modelling 25: 109-127. Diederich, J., Fortuner, R. & Milton, J. 1997. Construction and integration of large character sets for nematode morpho-anatomical data. Fundamental and Applied Nematology 20: 409-424. Maddison, D.R., Swofford, D.L. & Maddison, W.P. 1997. NEXUS: An extensible file format for systematic information. Systematic Biology 46: 590-621. White, R.J. & Allkin, R. 1992. A language for the definition and exchange of biological data sets. Mathematical and Computer Modelling 16: 199-223. Winfield, P.J. & Allkin, R. 1991. ALICE Transfer Format (ATF) Definition. Version 1.2. ALICE Software Partnership, The Royal Botanic Gardens, Kew, Richmond. 17 p. Hope this helps. Best wishes, -- + - - - - - - - - - - - - Mauro J. Cavalcanti - - - - - - - - - - - - + | Setor de Paleovertebrados, Departamento de Geologia e Paleontologia | | Museu Nacional do Rio de Janeiro | | Quinta da Boa Vista, 20940-040, Rio de Janeiro, RJ, BRASIL | | E-mail: maurobio(a)acd.ufrj.br | | Home Page: http://read.at/digitax/personal.html | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + "Life is complex. It consists of real and imaginary parts."

1 0

Re: so far...
by Alex R. Chapman 24 Nov '99

24 Nov '99

Firstly, can I just say that the level of interest and debate on this topic already is impressive - there are clearly a lot of us around the world banging our heads against this one! I guess it will take a little while to settle down to a rhythm of debate, and finding the level and especially scope of the task at hand will necessarily require some brain-storming in the early stages. But I would like to make the general comment that we should be very clear in our minds about keeping quite separate and distinct the task of defining a new standard for descriptive data exchange, from the specific nature of the data that the standard will be communicating. The standard is required for capturing and exchanging data across all kingdoms (and occasionally winds up being used for abiotic entities like archaeological pottery shards - see Louhivuori, DELTA Newsletter 12, April 1996)! Examples drawn from particular disciplines are extremely useful for illustrating the requirements or functionality of elements within the existing or proposed data standard, but have the potential to draw us into another debate entirely - that of data sharing across projects. As far as I can see this is the summary of this sub-groups task, as presented in the welcome message on joining this list: " the subgroup should attempt to analyze the requirements for a new standard for descriptive data. The metaformat for the standard will probably be based on XML or related formats. It is hoped that this standard will reach universal recognition to become at some point a successor to existing standards like DELTA, NEXUS, or XDF." The general mechanism for exchanging descriptive data between software applications and databases will have to be kept distinct from the definition and meaning of specific instances of descriptive data content, which I think should (currently) be outside the scope of this list. It's pretty clear from Don, Noel and now Leigh's comments that combining data meaningfully across projects is a whole 'nother ballgame: >> > For example, how could retrieval of descriptive information across a >> > department or institution be facilitated? Is it because of the sheer >> > flexibility in how the characters can be defined using DELTA, that querying >> > across projects is difficult to say the least, unless the character set is >> > global that is (and therefore with a lot of redundacy)? >> >> I agree. XML only solves the syntax problem. The semantics problem can't >> be solved as easily with technology, but would rely either on community >> agreement, or on extensive mappings between various ontologies. I think. > Definitely, and this is an important point to make. XML is an enabler, > its not a magic-bullet. What types of problems have people met in attempting > to share characters between research efforts? > At a basic level, linking withing and between documents and data is > easy. Its defining the semantics of this, and ensuring that the data > retrieved is meaningful. Just to answer Leigh's question from the previous paragraph, here in PERTH we have made a serious attempt over the last seven years to capture descriptive data in DELTA format in a number of projects with an underlying aim of sharing characters across all projects. Management of nested sets of characters and continued adherence to character definition across projects over time and at various levels of the taxonomic hierarchy has certainly been arduous at times, but the benefits of a 'corporate' approach to data capture are now becoming clear as the data is seeing the light of day (see our FloraBase web site for some readily available samples: http://www.calm.wa.gov.au/science/florabase.html ) And we have made some useful progress in developing database applications to aid in the process of amalgamating data reliably across projects. For an early discussion of some of the issues in managing institutional descriptive datasets see Chapman and Choo, DELTA Newsletter 12, April 1996). (NB. PDF versions of recent DELTA Newsletters are available at: http://www.calm.wa.gov.au/science/delta/news/index.html ) So, I'm not saying its not a rivetting subject - just that one headache at a time is enough! Cheers, Alex ____ Alex R. Chapman Email: alexc(a)calm.wa.gov.au Research Scientist Voice/Fax: +61 8 9334 0506 / 0515 WA Herbarium - Department of Conservation and Land Management Locked Bag 104 Bentley Delivery Centre Western Australia 6983 ---------- Original Text ---------- From: "Leigh Dodds" <ldodds(a)ingenta.com>, on 24/11/99 18:52: > As far as Delta goes, I think that the data are fairly close to what > taxonomists have been encoding into traditional taxonomic descriptions. > 'Success' is pretty good. > > Nexus can include charater/state data but also other things such as ways > to describe phylogenetic trees in the so-called "New Hampshire" format: > > ((raccoon:19.19959,bear:6.80041):0.84600,((sea_lion:11.99700, > seal:12.00300):7.52973,((monkey:100.85930,cat:47.14069):20.59201, > weasel:18.87953):2.09460):3.87382,dog:25.46154); If the basic format contains the relevant data, then a phylogenetic tree can be constructed from that data - so I'd have seen the above as something that should be derived from your data set rather than stored within it. I assume "New Hampshire" = "Newick" format? If so, heres an XML equivalent. <!ELEMENT label (#PCDATA)> <!ELEMENT title (#PCDATA)> <!ELEMENT branch (label?, branch*)> <!ATTLIST branch length CDATA #IMPLIED> <!ELEMENT newick (branch?)> Just something I was toying with before. > I sense that application developers who use Nexus are pretty happy with > the format since is was a collaborative effort to begin with, and that the > Nexus community would not see a great benefit in adopting a new data > format. Anybody else have this impression? Given this, and other similar comments on the list, I'd suggest that an additional requirement of any new format (if thats the way things go) is that it provides backwards-compatibility, as far as possible with other formats. Granted this might well be lossy, but given the numbers of users of Nexus, and the amount of software currently available, making efforts to provide for data conversion between formats means that we don't have to start completely from scratch. The first thing I did with XDELTA was provide a stylesheet with produce DELTA files. The same could be achieved for other formats. > > For example, how could retrieval of descriptive information across a > > department or institution be facilitated? Is it because of the sheer > > flexibility in how the characters can be defined using DELTA, > that querying > > across projects is difficult to say the least, unless the > character set is > > global that is (and therefore with a lot of redundacy)? > > I agree. XML only solves the syntax problem. The semantics problem can't > be solved as easily with technology, but would rely either on community > agreement, or on extensive mappings between various ontologies. I think. Definitely, and this is an important point to make. XML is an enabler, its not a magic-bullet. What types of problems have people met in attempting to share characters between research efforts? At a basic level, linking withing and between documents and data is easy. Its defining the semantics of this, and ensuring that the data retrieved is meaningful. > > This on the face of it would seem to map onto an XML > element/attribute/value > > schema pretty well. Would that help define more closely how we construct > > characters and maybe even prove universally applicable for all character > > types? > > Great question. Leigh? Anyone? :-) Sure element/attribute/value, but also element/element/content i.e. <leaf shape="obvate" /> or <leaf> <shape>obvate</shape> </leaf> Or did I miss something? > > Could one constrain further by expressing within the schema the > hierarchical > > relationships between the elements(eg 'blade' and 'petiole' as child > > elements of leaf') or would the introduction of terminology into the > > 'standard' be a step too far? > > This would be The Lexicon, no? It does seem to go beyond the task of > making a file format for data transfer. But it gets my vote > nevertheless. I'd suggest that something like this (expressing meta-data relationships amongst elements) could be layered on top of a basic data description format. RDF might provide the facility to do this effectively. I need to revisit the spec. cheers, L.

1 0