- tdwg-content - lists.tdwg.org

Re: characters/states and measurements and other hoary problems
by P. F. Stevens 03 Aug '00

03 Aug '00

>A couple of points. > >With regard to footnote 1. > >1 Attribution and sources for an item datum overides that for a character or >taxon, which override that for the treatment as a whole. Attribution for >characters and taxa are equivalent and additive. > >Don't we also need to say that item data attribution at the specimen level >may override that for a character or taxon as a whole? Otherwise, the system >has no way of dealing with misidentifications, particularly if some but not >all parts have been associated with the wrong ID, as might be frequently >encountered in fossils, or when dealing with taxa whose character state >definitions are later found applicable for only a specific size range (ie >differentially break down at small sizes). Specimen or parts level >data/attributions also would presumably be additive, even if potentially >contradictory (subject to differences of opinion), would they not? > This seems a reasonable point . > >If we are to presume nested levels of groupings of descriptor elements, then >we need to be able to clearly distinguish data values that are regarded as >referring to "individual units" (specimen, species, higher level taxon) at >one level, but are "collective" when evaluated at a different level. Not >only with the "collation rules" be different for different levels, but may be >different depending on whether a given feature (possibly "same feature but >defined differently") is regarded as a data item refering to a specific >"individual unit", or as a "collective unit". That is, the data item is a >representation of data that might apply to a collection, rather than a >measurable value that may have a scope no larger than a specific measure of a >specific specimen. Perhaps some treatments might include "collections of >collections" that would imply a mixing of both situations. > I presume you mean Specimen with ID - set of individual measurement or set of measurements (say in mm.) all associated with this specimen (maybe at a particular institution). Specimens of species: grouped, a larger series of measurements, and/or variously processed (e.g. mean + s.d.) the measurement(s) of either the specimen or the species could be assigned to states; those states might encompass a range of variation such that the attributuion of a specimen/species to that state would mean that, when refering to the state, one loses sight of the original measurements. P.

1 0

Re: characters/states and measurements and other hoary problems
by Stuart G. Poss 03 Aug '00

03 Aug '00

A couple of points. With regard to footnote 1. 1 Attribution and sources for an item datum overides that for a character or taxon, which override that for the treatment as a whole. Attribution for characters and taxa are equivalent and additive. Don't we also need to say that item data attribution at the specimen level may override that for a character or taxon as a whole? Otherwise, the system has no way of dealing with misidentifications, particularly if some but not all parts have been associated with the wrong ID, as might be frequently encountered in fossils, or when dealing with taxa whose character state definitions are later found applicable for only a specific size range (ie differentially break down at small sizes). Specimen or parts level data/attributions also would presumably be additive, even if potentially contradictory (subject to differences of opinion), would they not? Statement: One file will comprise one treatment, the basic unit of which is one or more characters describing one or more taxa or individuals. If we are to presume nested levels of groupings of descriptor elements, then we need to be able to clearly distinguish data values that are regarded as referring to "individual units" (specimen, species, higher level taxon) at one level, but are "collective" when evaluated at a different level. Not only with the "collation rules" be different for different levels, but may be different depending on whether a given feature (possibly "same feature but defined differently") is regarded as a data item refering to a specific "individual unit", or as a "collective unit". That is, the data item is a representation of data that might apply to a collection, rather than a measurable value that may have a scope no larger than a specific measure of a specific specimen. Perhaps some treatments might include "collections of collections" that would imply a mixing of both situations. Seems to me we need a means of distinguishing between collective representations and "unit values" (for lack of a better word), if for no other reason than to be able to track which data items apply to particular "features of general interest" (eg. leaves, roots, head, foot, etc) and which taxa are involved (higher-level taxon, species, individuals, parts of individuals). Its not clear to me how the current DDST can be used to associate "basic units" that might be reasonably differentially defined at different "levels of composition" by different investigators. Perhaps we need some general means to assign the "scope" over which the definitions of "unit" apply. Kevin Thiele wrote: > Dear List'eners > > attached find DDST Specifications.htm. See if this works. > > Cheers - k > > ------------------------------------------------------------------------ > Name: DDST Specifications.htm > DDST Specifications.htm Type: Hypertext Markup Language (text/html) > Encoding: quoted-printable

1 0

Re: characters/states and measurements and other hoary problems
by Una Smith 03 Aug '00

03 Aug '00

Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> wrote: > attached find DDST Specifications.htm. Here is the plain text equivalent, courtesy of lynx. Una Smith -------------------------------- snip ------------------------------------ Draft Specifications for a Descriptive Data Standard for Taxonomy Version History: Version 1.0 February 24, 2000, K.Thiele Version 1.1 revised July 18, 2000, K.Thiele General Requirements The DDST will be a data file structure that allows the capture and management of all types of data required for describing the morphology and anatomy of an organism or taxon. All data and metadata needed will be stored in one file, structured into several blocks (character lists, taxon lists, items data etc.). One file will comprise one treatment, the basic unit of which is one or more characters describing one or more taxa or individuals. The DDST will support the following: External lexica: these are externally-referenced lists of characters and states, or taxa, shared between several treatments. Lexica may be used without modification, or with one or more characters, states or taxa added internally (e.g. global vs local characters). Collation of data: data in the DDST may be captured and managed at several levels. One treatment (see above for definition of treatment) may store descriptive data for individual specimens, another may store data for species-level taxa, while another may store data for higher-level taxa. These individual treatments may be linked into a nested hierarchy, with specified collation rules allowing collation of data up the hierarchy, and passing of data down the hierarchy. Thus, some characters in the species-level treatment may be scored directly in that treatment, while others will collate data (e.g. leaf measurements) from items in the specimen-level treatment. Conversely, some characters may be scored in a genus-level treatment, and these become implicitly true for all taxa in a linked species-level treatment. Rich Attribution: all data elements in the DDST may be fully attributed to a source (e.g. contributor, published reference, specimen etc). Attribution will be optional at any level. Attribution will allow data-tracking and house-keepng, especially in circumstances when several contributors work on one treatment. The list of data elements below is structured using tabbed levels. Items tabbed across one level and enclosed in square parentheses are replicable within the higher level. Items in bold are required within their level (although the higher-level structure to which they belong may not be required) Comments are in curly parentheses. Note that this draft specification does not imply any particular structure for the data file used. It should be read as a list of required data elements for the final specification. ______________________________________________________________________ Treatment Name {Free-text title for the treatment} Description {Free-text description of the treatment} Treatment build/revision number {A real numeric e.g. 4.1 used for version control} Treatment build/revision date {Date string (standardised format?)} Contributors List {List of contributors to the treatment, including the principal builder} [ ID {Unique (in the context of this treatment) number for the contributor} Name {contributors name} Contact details {contributors address, email etc} Private notes {internal notes on contributor}7 ] Attribution {ID of principal treatment builder - this is the default attribution unless a lower-level item is specifically attributed}1 List of sources [ ID {Number for the source} Description {e.g. reference, description of specimen set etc} ] Principal Source {ID of the principal (default) source for the data}1 Treatment attachments {General information topics applicable to the treatment as a whole} [ Attachment name Attachment type {e.g. xml,html,txt,rtf,jpeg,gif} Attachment path/URL Public attachment notes7 Private attachment notes7 ] Private treatment notes {internal freeform notes for treatment}7 Character list source {path to an external lexicon that defines the character list for the treatment} Character set names list {list of set names for characters} [ Name {name string for a character set} ] Character List {required unless an external lexicon resource has been specified above} [ Character Name3 Character ID Set membership {list of sets to which the character belongs; a character must be able to belong to more than one set} Attribution1 {reference to a contributors ID from the Contributors list} Source1 (reference to a sources ID from the Sources list) Collated Character source {path name for another treatment that contains lower-level data for this character} Collation rule name {Name of a collation rule as defined in the Collation Rules list}2 Character type {ordered multistate, unordered multistate etc} Character dependencies (up) 4 Applies To list (or global/restricted type definition, then leave it to program to extract) 5 Character attachments [ attachment name attachment type attachment path/URL Public notes7 Private notes7 ] Private notes {internal notes for character}7 Character State List [ Character state name | Character state ID Character dependencies (down) Character state attachments [ attachment name attachment type attachment path/URL Public notes7 Private notes7 ] Private notes7 ] ] Taxon list source {path to an external resource that defines the taxon list for the treatment} Taxon set names {defines a list of allowable names for taxon sets} [ name ] Taxon List [ Name | Taxon ID Taxon set membership {list of sets to which the taxon belongs; a taxon must be able to belong to more than one set?} Taxon attribution1 Taxon attachments [ attachment name attachment type attachment path/URL Public notes7 Private notes7 ] Private notes7 ] Item Data {This will hold the "score matrix"} Taxon Name|ID/Character Name|ID6 Character Name|ID/Taxon Name|ID6 State Name|ID Score {normally present, rare, present by misinterpretation etc} Score Attribution1 Public Notes7 Private Notes7 ______________________________________________________________________ 1 Attribution and sources for an item datum overides that for a character or taxon, which override that for the treatment as a whole. Attribution for characters and taxa are equivalent and additive. 2 Treatments are nestable. That is, one treatment may contain data on specimens, a higher-level treatment on taxa. The higher-level treatment gathers information for some characters from lower-level treatments, using a specified collation rule. Collation rules will be specified externally to the treatment, and will cover e.g. how to merge scores, calculate values, deal with conflicts in source data etc 3 Character names may be hierarchically nested. Character properties (e.g. sets, dependencies, attachments) are only specified for the lowest level characters. e.g. Leaves margins teeth orientation }only these have shape } properties 4 Dependencies may be defined either up or down (but not both?). An up dependency lists the character states that make this character inapplicable; a down dependency lists characters that become inapplicable when this state is chosen. 5 The idea here is to specify a subset of taxa for which this character is scored, or to specify that the character is non-global, then leave it to the parsing program to determine the taxon list. This feature would be used by future identification programs that employ the Progressive Revelation model. 6 The item data may be stored as the equivalent of either a taxon-state matrix or a state-taxon matrix, depending upon whether taxa are nested within characters or characters are nested within taxa. There will need to be a way of specifying which of these is operative. 7 Public Notes are available for parsing, Private Notes are not, and are designed for private housekeeping within the treatment

1 0

Re: characters/states and measurements and other hoary problems
by Susan B. Farmer 02 Aug '00

02 Aug '00

>Dear List'eners > >attached find DDST Specifications.htm. See if this works. > Uh, not really. Same scenario as for Word Docs for me. Susan > ><!doctype html public "-//w3c//dtd html 4.0 transitional//en"> ><html> ><head> > <meta http-equiv=3D"Content-Type" content=3D"text/html;= > charset=3Diso-8859-1"> >

1 0

Re: Where are we?
by Susan B. Farmer 02 Aug '00

02 Aug '00

>On Mon, 31 Jul 2000, Peter Rauch wrote: > >>(Can it be posted in plain-ascii? Maybe not everyone could view >>it?) > >I second that request. I can view Word docs, but it is a hassle. > > Una Smith > Make that a third ... Susan Farmer

1 0

Re: Where are we?
by Una Smith 02 Aug '00

02 Aug '00

On Mon, 31 Jul 2000, Peter Rauch wrote: >(Can it be posted in plain-ascii? Maybe not everyone could view >it?) I second that request. I can view Word docs, but it is a hassle. Una Smith

1 0

Where are we?
by Kevin Thiele 01 Aug '00

01 Aug '00

Dear List'eners I don't want to push things, but it seems to me that the discussion's not gaining much more focus that we had last time. I think mostly we're reiterating and discussing the scope of the problem, and while this is important and valid, we also need to move beyond this to specific attempts at dealing with the problem. We're discussing the need for a standard rather than actually creating one. I put up a while ago an initial attempt at a proposal (the document DDST Specifications.doc, you all will have this buried in your attachments folder somewhere). Some people have looked at this, I know, but I think we need to decide: 1. Is this a start, or is it the wrong way to proceed and should be ditched in favour of something better 2. If it's an adequate start, what concrete changes are needed to incorporate the ideas people are discussing? 3. If it should be ditched, can anyone produce a better draft specification that we can build on. I think at this stage we need a draft document around which the discussion can focus. Contributions that point out complications and problems can then make concrete suggestions as to changes to the draft proposal, rather than be mere discussion points. It seems to me we should be at this stage around now - we've discussed the problems for close on 12 months now. Now it may be that the structure of this group is poorly suited for the task, and a recommendation should go to TDWG that this isn't the way to proceed. As it stands, it seems to me that we're functioning well as a discussion group but not well as a working group (actually working on the problem). This is undoubtedly partly because we're all busy - I know that I'm doing all this in my "spare" time and I'm sure most of us are in the same boat. Perhaps the only way that this can happen properly is for funds to be sought to employ someone on a contract basis to turn the ideas raised here into concrete output, using the list as an expert group/listening post/scratch-pad resource or whatever. We need to be honest as to whether we're going to achieve something. Any idea? Cheers - k

1 0

Oh phooey... Re: [TDWG-SDD] Where are we?
by Peter Rauch 31 Jul '00

31 Jul '00

On Mon, 31 Jul 2000, Peter Rauch wrote: > Send email to listserv(a)usobi.org with > GETPOST TDWD-SDD 183 X Make that: GETPOST TDWG-SDD 183

1 0

Re: Where are we?
by Peter Rauch 31 Jul '00

31 Jul '00

On Tue, 1 Aug 2000, Kevin Thiele wrote: > > I put up a while ago an initial attempt at a proposal (the document DDST > Specifications.doc, you all will have this buried in your attachments > folder somewhere). Some people have looked at this, ... Send email to listserv(a)usobi.org with GETPOST TDWD-SDD 183 to retrieve another copy of this Macintosh-created, MS Word formatted, base64-encoded-attachment document. :>) (Can it be posted in plain-ascii? Maybe not everyone could view it?) Peter

1 0

Re: characters/states and measurements and other hoary problems
by Stuart G. Poss 30 Jul '00

30 Jul '00

While I am philosophically most sympathetic to Peter's agruements, I must agree with Kevin with regard to the need to distinguish measurements as "characters" from qualitative characters. The bases of comparison for the former are typically fundamentally different with respect to asignment to state of the latter, and would be dependent not only on the exact definition [criteria for recognition] of the endpoints, which may differ systematically between investigators (or method of measurement), but also with respect to temporal scope (ie some endpoints may not be identifiable, except within a particular size range for a given structure in a given taxon). As pointed out by Kevin, they would also be dependent on sampling, although one would hope that most investigtors are atuned to the need to increase sample sizes to avoid bias or mischaracterization. With respect to measurements, the bases of comparison are geometrically constrained, whereas qualitative descriptions are not so constrained, even though both may be used to characterize the "same" features. Nonetheless, one could in some cases meaningfully tag measurements with some of the same tags as qualitative characters to indicate that they are proxies for the "same" features. This would allow data matrices conforming to these "generalized" features to be associated and the user able to then assess whether the association is "sufficiently tight" to warrant regarding them as proxies for the "same" feature. The difficulty comes in consistently applying/defining the basis of comparison and in establishing criteria to recognize when constraints imposed on using measurements create inconsistency with various qualitative definitions of "state". Perhaps attributes of a tag element could be used to indicate (hint at) different shades of meaning when measurements are associated with qualitative states (eg <leaf character><measurement character endpoint1="regionally localized, endpoint1_precision=">>5%" endpoint2 ="precisely localized" endpoint2precision="<<1%"> 45.2</measurement character></leaf character>, or some such construction, where in this case, <leaf character> may take the same tag name as might the qualitive character. However, we may need to also include additional refining elements <data measured by>Peter Stevens</data measured by>Peter Stevens</data measured by><data measured using>calipers</data measured using>, or some such to fully relate fundamentally diffferent data types or qualify the measurement context. In any event, such shades of meaning would require that "appropriate" context be established and context could vary enormously depending on the features being evaluated. In image analysis, one of the reasons image algebra was developed was to assist in standard charcterization (transformation) of images and image fragments ("features"). However, this only works when there is a consistent basis for comparison (typically pixel size and pixel neighborhood) that underlies the sets being defined. Unfortunately, (or perhaps fortunately) the human vision system is vastly more variable and complex. In the near term, Kevin is probably correct to suggest that the onus is on those who would care to tackle the "hellish difficulties" of making such associations. It may be best to fork the dialog with regard to this point, which is not to say that we shouldn't seek a general approach that could be extended to include possible solutions to such "issues of association" at a later time (ie better establish a useful lexicon and grammar for <mesurement character> elements, as opposed to <qualitative character> elements). Nonetheless, one of the unfortunate practices in systematics has been the relegation of measurement data to "notebooks" or more recently "floppy/CD disks" that are not properly archived in any coordinate fashion, except perhaps for the duration of a study or some unspecified time after that, and often at best only partially summarized in the original publication. The discipline could be much, much richer, if there were a consistent standard for archival and association between measurement data and qualitative description, if for no other reason than to recognize the various contexts within which these data might be useful in the future, not to mention to encourage workers to let others know "exactly" what they did to arrive at their conclusions. Measurment data can be extremely useful in providing insight into how different investigators actually recognize the various states of their characters and may suggest why different investigators may not recognize the same states for the "same" characters. Kevin Thiele wrote: > At 10:23 AM 25/7/00 -0500, Peter Stevens & Stinger commented on the hoary > issue of characters/states vs measurements: > > It's completely true that a set of measurements is a more fundamental data > element than a character state. But we need to allow for both ways of > recording data, for two reasons. Firstly, some characters don't lend > themselves to measurement (e.g. indumentum - it would be possible but > hellishly difficult). Secondly, if I'm writing a key to Families, I'm > afraid I probably won't be recording measurements of individual specimens > very often, as the data blowout would be alarming. And what specimens would > I use? I see no way around the problem (for higher-order taxa at least) of > doing the collation of data (measurements->states) in my head, with all the > admitted problems that ensue. > > The draft standard document I've put to the group allows for some > characters to be scored after a head-wise (hopefully wise, anyway) > collation, others to be collated from individual measurements using some > form of collation rule. > > As to data attribution, the draft standard document allows (I hope) all > data elements to be attributed (to a specimen, taxon, literature citation, > person etc), but doesn't force the issue. > > >Kevin: >Or can we have a set of linked standards - one for describing in > >the boring > > >old characters/states/taxa way, and others for the more space-shuttle ways > > >that can be linked in as they develop. > > > >and Eric >it is rather difficult to > > >combine datasets based on differing character lists, even when those > > >character lists are fairly similar. There is no mechanism for "mapping" > > >character states from one dataset onto those of another. > > > >We are back to an issue that was raised earlier, so at the risk of boring > >people by repetition, the way to record data is not (if at all possible) > >as states, but as measurements. Measurements of the same character from > >different studies may then be combinable, and states extracted from the > >measurements. But if one is combining states, then one is likely to be in > >trouble. The "approved shapes" that a committee of the Systematics > >Association came up with some years back are fine for communication in > >descriptions and conversations, but the best way to record shape data, > >whether for phylogenetic analysis or deciding if one heap of specimens > >really is distinct from another, is as measurements. > > > >(I agree that a list of "approved" characters is probably not attainable > >at present.) > > > >This raises another issue that Eric mentioned: > > > > >Another aspect > > >might be the inclusion of meta-data (e.g., who recorded this observation, > > >and on what material?) While this sort of information can be placed into > > >DELTA "text" characters and comments (as could virtually anything > > >representable as a string of bytes, with a bit of effort), the use of text > > >characters does not provide for well-structured access to the information. > > > >This issue of metadata is very important: if a DNA sequence is linked to a > >voucher, then all measurements/observations should be linked to > >specimens. In an "ideal" database, there would be links to specimens, if > >only through a literature citation; not being able to make easy links in > >the literature can make it difficult or impossible to use, and not having > >such links in a database greatly reduces its value. If I go wrong in > >including a specimen in a particular taxon, I want to be able to pull out > >all information associated with this specimen, whether it is simply length > >of leaf blade or how the anther wall develops, when I put it in its > >cotrrect place. > > > >Peter S. > ></blockquote></x-html>

1 0