November 1999 - tdwg-content

Re: Types of data
by Jim Croft 25 Nov '99

25 Nov '99

>Surely >the way you measure an attribute depends on what you are going to use the >measurements for, ultimately that's a question of precision. If there is a >possibility that there will be a variety of uses (reuse) for the data, then >an indictation of the degree of precision of the measurement would be highly >desirable. Source and precision stuff is surely important, but it is quite possible to end up with major bloat and more meta than meaningful data. This may not be a problem but there is certainly a practical limit as to how far we can or should go in this direction. >Isn't it also important to distinguish whether the description refers to an >actual leaf (ie. that leaf there, on that specimen is ovate) or whether you >are making a generalisation about a taxon (ie leaves ovate to obovate)? Most certainly. I must admit I thought we were talking about taxon descriptive data, but therie is probalby no reason at all why the same principles and data structure/definitions can not be used forthe structures/features themselves. >A taxon cannot have a leaf shape, only leaves can, and an actual leaf does not >usually vary in shape except through time. Conversely, only a taxon can have, or not have, ovate leaves. Is this a generalization or a fact? (and does it matter?) ><my_taxon_description> > <specimen> > <leaf shape="ovate" length_mm="10"/> > <leaf shape="ovate" length_mm="8"/> > <leaf shape="obovate" length_mm="25"/> > </specimen> > <specimen> > <leaf shape="ovate" length_mm="5"/> > <leaf shape="obovate" length_mm="31"/> > </specimen> ></my_taxon_description> Conceptually I like this reperesentation - it is what we all do and may or may not write down. But as you say, maybe people want to know this, maybe they don't... jim __________________________________________________________________________ Jim Croft ~ jrc(a)anbg.gov.au ~ http://www.anbg.gov.au/people/croft.jim.html ph 02-6246-5500 ~ fx 02-6246-5248 ~ GPO Box 1600 Canberra ACT 2601

1 0

FW: GEN RE: Types of data
by Don Kirkup 25 Nov '99

25 Nov '99

> but ovate is rather difficult to parameterize. It may be possible to > get ellipsoidal parameters, but width and length are clearly > insufficient. > > Does anybody know about good parameterization of shapes, so they can > be stored in more objective forms? Wouldn't the most objective representation of shape be given by a set of coordinates for points placed around the outline? If orientation of the object is important, then give landmark coordinates too eg. mark the organ base and apex. From that info, parameters such as object area, perimeter length, circularity (perimeter length*perimeter length/area), centroid, longest and shortest axes, aspect ratios etc. can be calculated. You'd also be able to standardise your outlines (eg. using Bookstein's transformation) and compare against reference shapes (eg elliptic fourier analysis), pairwise comaprisons using (thin plate splines) or fit to average shape using Procrustes metric. Feature extraction from digital images (eg defining the object outline, sampling of points around outline)and calculation of the shape parameters and export to spreadsheet is virtually automatic in packages like OPTIMAS. James Rolf's NTsysPC will perform the basic standardisations, thin plate splines and EFA. don

1 0

First level topics on this list (GEN)
by Gregor Hagedorn 25 Nov '99

25 Nov '99

The following is an attempt to structure the topics that must be discussed. On a first level, I believe we need to discuss: (GEN) General topics, e.g. discussion about the agenda in general, standardization process in general, perhaps exploration of funding possibilites to organize one or several workshops. (RQT) Requirement analysis and information model for descriptive information in biology: Stepping back and asking ourselves what we need in terms of information model, entities/attributes or functionality. We should try to create an outline here to further structure this discussion (XML) Exploration of the suitability or desirability of XML to be used as a metaformat for a new exchange format for descriptive data (incl. discussion of XML-Data, XML-Schema, XML-RDF, names spaces, Dublin Core, etc. What else? RQT and XML should definitely be structured further. I will make a proposal for RQT tomorrow. Please: - Always use a subject line - If possible, add the outline code (GEN, RQT, XML) and perhaps the number in the outline for this topic after your own subject in brackets. See the example for (GEN) in this posting. - When responding to a posting, please delete any information you are not commenting on. Do not just leave the entire posting as quoted text before or after your own text! I believe that structuring will help the discussion, but please feel free at any time to submit a discussion paper on any topic you consider important! Gregor ---------------------------------------------------------- Inst. for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Gregor Hagedorn Net: G.Hagedorn(a)bba.de Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: Types of data
by Stuart G. Poss 25 Nov '99

25 Nov '99

Kevin Thiele wrote: > > >Could one constrain further by expressing within the schema the hierarchical > >relationships between the elements(eg 'blade' and 'petiole' as child > >elements of leaf') or would the introduction of terminology into the > >'standard' be a step too far? > > This worries me. Is the entity/property/value schema restricted to 3 > levels? (ditto for XMLs element/attribute/value). If so, it seems too much > of a straightjacket, especially if we want to express a hierarchy of > elements like this. What to do about > leaf/lamina/abaxial_surface/vein_islands/indumentum/density. > I agree that a hierarchically structured schema may be wonderful, but in a > biological context I wouldn't like to see it constrained at 3 levels just > because that's what existing computer formats do. > > Cheers - k I don't think there is any internal constraint within XML that would restrict one to evaluating hierarchial relationships, if these were the most suitable, to only 3 levels. Some parsers may actually be constrained to work with a finite level of nestedness, but I would suspect that it would be much greater than 3 deep. I would think as is the case for object or relational modeling one could instantiate one to many or many to one relationships as required. My original point here was only that it is often difficult to decide how to represent concepts that associate objects/entities/roles. In one context it would be appropriate to say regard a leaf as an entity, its shape as an attribute, and ovate as a value. However, (and again this is probably not be the best example) in another context a leaf of the same plant might be regarded as a attribute of yet another object/entity, perhaps a tree, and not considered an entity on its own. A particular implementation, such as one using XML, will often provide a variety choices for representing the underlying ideas, but until we model/conceptualize how the various conceptual components interoperate/relate, a particular approach can often be difficult to evaluate or to extend. Further, I didn't mean to imply that all conceptual associations among character data types would be hierarchial. It was probably a poor choice of words on my part and I believe such a restriction would greatly limit the range of investigators who might be able to contribute and substantially limit the number of taxa or morphs that could be so characterized. There could be a large number of potential representations, including: trees, cyclic acyclic or directional graphs, lattices, semilattices, ordered sets, partially ordered sets, Galois connections, tensorial decompositions, etc. My point was that I believe we need to identify the critical conceptual relations and their context in a "standard" model for character data and to then formulate a language (implementation [in XML?]) for describing the most meaningful or useful among them. This language would presumably be a superset of existing methods. To use the terminology of Ganter and Wille (1996. Formal Concept Analysis. Springer.) in its simplest form, it might be useful to speak, as closely as possible, in terms of formal contexts, such as K := (G, M, I), which consists of two sets G (objects)and M (attributes) and I the incidence relation betwen G and M of the context, AND the formal concept of the context (G, M, I), namely a pair (A, B), where A is the extent and B the intent, and where A C G (C meaning subset not "C") and B C M, A' = B and B' = A, where A' := {m e M | gIm for all g e A} and B' := {g e G | gIm for all m e B}. (reading e as "is an element of"). Fortunately, even for many-valued contexts, these can be represented as simple cross tables. Already, we have considered several including 1) matrices of 0's and 1's 2) taxon by measure matrices, 3) range matricies for a given specimen, 4) specimen x location matricies, 5) species by location matrices... No doubt others may think of lots of other possibilities. A useful approach might then be to approximate the "quasi"-complete concept lattice B*(G, M, I) of a context derived from gluing (union of) concept lattices of a large number of relevant contexts and then identifying the most meaningful/useful/easily encoded congruent sublattices, factors, and implications between attributes, ignoring, at least initially, issues of scale, closure, and incomparability. Such an approach might thus permit working subgroups to focus on particular forms/types of character data and their most appropriate/useful/efficient representation. It might also permit us to collate useful lists of works/websites employing particular kinds of representations for an eventual website characterizing the emerging potential standard or recommendation. Admittedly, such a more general view of what would be required to model "character data" creates some tangential considerations with respect to straightforward conversion among existing formats, certainly a worthy and perhaps more immediately necessary goal. Such a subdiscussion would be roughly equivalent to enumerating the rules by which existing approaches encode various contexts and concepts and then establishing a uniform syntax to describe the common rules among them. However, as Peter and Gregor (at the meeting) indicated, it would be desirable to permit extensions that allow observational data (ie things that are measured) to be recognized as distinct from, as well as related to, what we might be loosely call conceptual data (representations that bear a greater degree of inference, such as 0's and 1's in phylogenetic data, leaf "shape" "ovate", or leaf color "green" etc.), so that one is in a better position to 1) verify the accuracy of the former, perhaps using alternate techniques, 2) study the relations between and properties of the two kinds of data, 3) test assumptions explict or implicit in the latter, 4) develop machine-assisted search/display/evaluation techniques for 1) 2) 3) across a wide range of independent data sets in a variety of contexts. If this is too radical a restructuring of the original intent of the discussion that may have been more narrowly focused (I didn't attend the initial setup meeting at Harvard that took place outside the main meeting room), I would be content to continue discussion of such general considerations among a smaller or alternate subgroup, assuming there is anyone else who feels likewise inclined. In any event, hopefully discussions of character data representations focused toward a potential emerging standard will be robust enough to include relatively novel formats, such as that involved in range data, and their conceptual relations to more traditional data types.

1 0

TDWG99 (Harvard), Minutes of SDD meeting
by Gregor Hagedorn 25 Nov '99

25 Nov '99

Minutes of the Subgroup "Structure of descriptive data" workshop at TDWG 1999 in Harvard ---------------------- The subgroup convened on Sunday, 31.10.99 during the annual TDWG meeting at Harvard University. The following topics were discussed: 1. Diederich's "Basic properties" General agreement was reached that a direct application of the "structure-property-value" model would be too restrictive, applicable mainly to morphological descriptions. A more general model, including cultural/physiological and molecular descriptions should be developed. The term "feature" was proposed as a more general replacement for structure. 2. Standardization of schemas (character definitons), user defined metadata This was considered desirable in the long run, but, esp.considering the experiences made by Richard Pankhurst with plant descriptors, it should be considered a long term goal. The problems were shortly discussed. Agreement was reached that these topics are quite separate and should be discussed in separate discussion groups. 3. New interchange format for descriptors It was proposed that a new "Superset format", encompassing the functionality of the existing DELTA and NEXUS interchange formats, and with extensions to support the requirements of LucID, DeltaAccess, and possible further software packages should be developed. However, it was agreed that we should first step back to review the current use of descriptors. The new format should not be a simple superset with possible redundancy, but the principles of biological descriptions should be reevaluated to cover them with appropriate and general mechanisms. To achieve a clearer picture, a requirement analysis was proposed (see next point). A distinction was made between the data model and the metaformat. It was agreed that XML may be a good vehicle for a new standard. However, alternatives should be carefully evaluated. A decision for or against XML would have to be made only after the requirement analysis and information model had sufficiently far progressed. An important point was made that the standard should belong to TDWG, and should neither in name nor in the preparation of future versions be too intimately tied to any developer of a software application. The relationship to existing standards TDWG-DELTA or NEXUS should be clearly defined and mapping definitions provided. 4. Requirement analysis A requirement analysis is to be made, separately for general and application type specific requirements. The general requirements should be discussed according to a preliminary information model. 5. How to proceed The group agreed to communicate over an email listserver (which will be installed at the University of Kansas with the help of Jim Beach). It was agreed that the documents to be discussed should in simple cases be send directly as plain text in the body of an email, or in the case of longer, structured and formatted documents, in the form of HTML attachments. The use of PDF should be avoided, since it can not be directly read and is difficult to annotate and consolidate annotations. A web page will document the consolidated discussion papers and the emendations or criticisms regarding these papers. In addition, the list server will directly provide an unedited archive of the original discussion. Time frame: It was considered desirable to have a definite proposal by a years time. ---------------- Any additions or corrections from those present are welcome! Gregor ---------------------------------------------------------- Inst. for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Gregor Hagedorn Net: G.Hagedorn(a)bba.de Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Nexus
by Gregor Hagedorn 25 Nov '99

25 Nov '99

Susan Farmer wrote: > I (initially) put all of my data into > DELTA and then converted it to Nexus. The Nexus file has been heavily > modified (and many of my character and state names truncated in the > bargin). Until very recently, there was no way to take *this* file > back into DELTA. But more than that; all of the continuous data > in the original file (aka measurements) are now gone and will have to > be added back into the DELTA data set by hand. :-( We had several comments on NEXUS so far. I agree with Kevin (and Leigh), that "users of NEXUS are more than happy with their format". However, I don't think they should be, exactly because the situation Susan is commenting on. NEXUS is by far more specialized and that much information is lost when restricting a data set to NEXUS. If NEXUS does not change, it would sooner or later become a one-way format, and would not be used for data storage any more. That does not mean that NEXUS is not a good format, but I believe we do need a format that is covering all the common ground. We should try to learn from NEXUS as we should try to learn from DELTA. >Given this, and other similar comments on the list, I'd suggest that an >additional requirement of any new format (if thats the way things go) >is that it provides backwards-compatibility, as far as possible with >other formats. I believe this backward compatibility will be possible, at least in a one- way situation. With this I mean that a simple XML program (given the new program is based on XML) will be able to provide DELTA or NEXUS formatted, that can be read by older programs. However, if the new format is really a redesign and not just a superset of competing directives, some information may be lost, and some information may have to be mapped in a way that will usually provide the best results. > Why not extend the current standards that if there are data types that > their particular application doesn't recognize, they stay around -- just > ignored. MacClade does that even with Nexus data (maybe the newest one > doesn't); but if I save my file from within MacClade, if I want to run > PAUP again, there are commands that I must reinter into the file > because MacClade has removed them because it didn't understand them. I believe that a new format is more desirable than blowing up existing formats beyond their design specifications. I think the latter is the main problem that I have with the proposals for "New Delta": The discussion and proposals correctly identify many shortcomings, which I consider very valuable work. However, resolving these issue in the context of the existing DELTA format structures, and especially in close context to the way the CSIRO DELTA programs currently work, has lead to many solutions which I consider undesirable. Gregor ---------------------------------------------------------- Inst. for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Gregor Hagedorn Net: G.Hagedorn(a)bba.de Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Welcome
by Gregor Hagedorn 25 Nov '99

25 Nov '99

Dear Colleagues I welcome all the new members of the list! Currently 96 people have subscribed, that is considerably more than I expected. However, only few people from the NEXUS community have subscribed. As it should be, the discussion is already in full flow. If those people who have signed up late want to catch up, please can go to the web archive (http://usobi.org/archives/tdwg-sdd.html). Perhaps we should try to add some more structure (Kevin already made some suggestions), but in any case just keep posting everything you think relevant. Regarding the general purpose of this list and the manager (Mauro's question), I post the welcome message below. Most of you have already received this, but some early subscribers did not or received an earlier version. Separate postings with minutes and a provisional outline will follow. ---------------------------------------- TDWG-SDD, Discussion List on the Structure of Descriptive Data To post a message to the list, send email to tdwg-sdd(a)usobi.org from your subscribed email address. Include a meaningful Subject line. In general, use only plain text in the main body of your message, and put HTML or other formatted text/graphics in attachments. This will make it easier for other subscribers to read the message, and very importantly, it will allow the LISTSERVer to create useful search terms from you message's main body. TDWG is the Taxonomic Databases Working Group. The mission of TDWG can be found at http://www.tdwg.org/. The subgroup "Structure of Descriptive Data" was started 1998 at the TDWG meeting in Reading. At the TDWG meeting in Harvard in Oct 1999 it was agreed that the subgroup should attempt to analyze the requirements for a new standard for descriptive data. The metaformat for the standard will probably be based on XML or related formats. It is hoped that this standard will reach universal recognition to become at some point a successor to existing standards like DELTA, NEXUS, or XDF. The purpose of this discussion list is to foster communication and collaboration among those involved in the creation of a new standard. The new standard should be carefully drafted and agreed upon by consensus. The discussion list provides an email- and www-accessible archive (available at http://usobi.org/archives/tdwg-sdd.html) so that the discussion is adequately documented. The list is expected to produce some traffic, including attached formatted documents (HTML + graphics). If you are concerned about too many emails flooding in, please consider whether your email client offers an automatic filter function to directly copy all mails originating from the list into a separate folder. To unsubscribe from the list you should send an email to LISTSERV(a)USOBI.ORG with the message: UNSUBSCRIBE TDWG-SDD If you have any question about the TDWG-SDD list, write to the list owners at the generic address: TDWG-SDD-Request(a)USOBI.ORG ---------- If you have any suggestion or additions please send them to me! ---------------------------------------------------------- Inst. for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Gregor Hagedorn Net: G.Hagedorn(a)bba.de Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: Types of data
by Gregor Hagedorn 25 Nov '99

25 Nov '99

> > I too would argue that what we seek is a system that distinguishes between > > "observations", measured in some sense (either directly using > > some device), or perhaps "mapped" using known or widely used conventions and > > higher level abstractions that often pass as qualitative "data", (ie leaf > > "ovate" or state "relatively derived" or "1"). > > Would it be too simplistic to simply allow the format to express > the source (qualitative/quantitative, or a more detailed description) > or a particular data set? I believe that original measurement data should be part of the data set, together with rules how to map them unto categorical data (DELTA: "Keystate directive"). Knowing that categorical states for, e.g. length-to-width-ratio of a leaf have been measured rather than just guessed, solves only a small part of the problem. > > Deciding which sources you trust, value, or otherwise wish to > manipulate is an option you then have at the application level. > > i.e. "I wish only direct measures of leaf shape, and not > a description such as 'ovate'. but ovate is rather difficult to parameterize. It may be possible to get ellipsoidal parameters, but width and length are clearly insufficient. Does anybody know about good parameterization of shapes, so they can be stored in more objective forms? Gregor ---------------------------------------------------------- Inst. for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Gregor Hagedorn Net: G.Hagedorn(a)bba.de Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

A comparison of formats for descriptive data
by Mike Dallwitz 25 Nov '99

25 Nov '99

> From: Jim Croft > To: TDWG-SDD > enunciate/tabulate exactly what it is we are trying to achieve and the > shortcomings/limitations of exiting formats in reaching this goal No sooner said than done :-). See A comparison of formats for descriptive data http://biodiversity.uno.edu/delta/www/compdata.htm Actually, I've been working on this for a few days, but have been hampered by some administrative work (a review of our project). The document is rather sketchy at the moment, and I will correct and add to it as time permits. There may be quite a few errors in the information on Nexus/Paup/MacClade. Also, I haven't yet had time to incorporate some of the suggestions made on this list. The paper I presented at the recent TDWG meeting is also available: Introduction to data requirements for natural-language descriptions and identification. http://biodiversity.uno.edu/delta/www/descdata.htm -- Mike Dallwitz CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/

1 0

Types of data
by Kevin Thiele 25 Nov '99

25 Nov '99

Leigh Dodds writes: >> I too would argue that what we seek is a system that distinguishes between >> "observations", measured in some sense (either directly using >> some device), or perhaps "mapped" using known or widely used conventions >and >> higher level abstractions that often pass as qualitative "data", (ie leaf >> "ovate" or state "relatively derived" or "1"). > >Would it be too simplistic to simply allow the format to express >the source (qualitative/quantitative, or a more detailed description) >or a particular data set? Too simplistic, I think. There will surely be many cases where in one data set some characters will be "real" (i.e. low-level) observations while others will be abstractions. For instance, sometimes it will be viewed as worthwhile for the taxonomist to capture real data (leaf shapes are fairly easy to do), other times anything other than abstraction is practically impossible (try working out how to capture indumentum characteristics quantitatively). More on the entity/property/value model. Don wrote: >Can character definitions be constrained without making too tight a >straight-jacket for oneself? Might it be possible to represent taxon >character descriptors by for example an entity/property/value (eg. >leaf/shape/ovate) based schema? >This on the face of it would seem to map onto an XML element/attribute/value >schema pretty well. Would that help define more closely how we construct >characters and maybe even prove universally applicable for all character >types? > >Could one constrain further by expressing within the schema the hierarchical >relationships between the elements(eg 'blade' and 'petiole' as child >elements of leaf') or would the introduction of terminology into the >'standard' be a step too far? This worries me. Is the entity/property/value schema restricted to 3 levels? (ditto for XMLs element/attribute/value). If so, it seems too much of a straightjacket, especially if we want to express a hierarchy of elements like this. What to do about leaf/lamina/abaxial_surface/vein_islands/indumentum/density. I agree that a hierarchically structured schema may be wonderful, but in a biological context I wouldn't like to see it constrained at 3 levels just because that's what existing computer formats do. Cheers - k

1 0