March 2000 - tdwg-content - lists.tdwg.org

Re: SDD Specifications Document
by Mike Dallwitz 08 Mar '00

08 Mar '00

> From: Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> >> From: Leigh Dodds <ldodds(a)INGENTA.COM> >> Is it reasonable to include character dependencies 'upwards'? > There may be a problem if you mix and match "up" and "down" dependency > definitions. The thing is to choose the one that will be most easily > defined. Both forms are necessary for efficient internal use in programs. It's also convenient to have both forms available in the user interface of a data editor (our DELTA Editor does this). However, one form is enough for a data-exchange format - to have both (especially both together in the same data set) would make reading and checking the data more difficult. -- Mike Dallwitz CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/

1 0

Re: SDD Specifications Document
by Kevin Thiele 07 Mar '00

07 Mar '00

At 13:57 6/03/00 -0500, Peter Stevens wrote: >>If the rules need to be formulated rather then just named the challenge is >>to do it in such a way that an XML parser could do the collation, and a >>klutz taxonomist could create their own if an existing one is not sufficient. >> >There are many ways one can go from measurements to some sort of collation, >and this is an area where there is a fair amount of activity - I would have >thogutht that specifying rules was not the way to go. By rule here I just mean a quasimathematical rule that the computer uses to effect the collation. Some rules should be common and straightforward, such as the "gather all values" rule, others may be specific to a treatment/builder. As I envisage it, you (Peter) could formulate your own rule for your own treatment to do the collation just as you want. If there's a lot of activity in this area, can you help with this bit? >>>2. Collated Character source. I see these as essentially >>>a drill down mechanism that further identifies a 'bottom-level' >>>taxon. >> >>The necessary thing is to identify a path to the treatment that contains the >>data for collation, and the taxa of the lower-level treatment that will >>provide the data for a taxon in the current treatment. In the example above >>it's perhaps deceptively simple, because the target taxa in the lower-level >>treatment (in this case, the specimens) are nested within a taxon with the >>same name as the current taxon in the higher-level treatment (Macrolepiota >>clelandii or Lilium turkestanicum). This is perhaps the easiest way to do it >>(thinking aloud now) - require that the taxon of interest exist somewhere in >>the taxon hierarchy in both treatments, then step one down in in the >>lower-level treatment to find the elements of interest. >> >Are you then going to have to specify all ranks in advance, so that one >knows what is up or down? And how would one proceed if unranked naming >does indeed become popular? No, the system as I envisage it will be rank-free (I agree, we need to stay ahead of the game here), and "up" and "down" will be locally defined relative to the hierarchy as defined in the treatment. E.g. Treatment 1: Key to Australian species of Rhamnaceae Rhamnaceae Colletieae Discaria Discaria nitida Discaria pubescens Treatment 2: Morphometric data for Discaria specimens Discaria pubescens CANB4452885 CANB4788710 Discaria nitida CANB9633902 CANB9644921 Treatment 1 here defines Treatment 2 as the source for some collated characters (e.g. leaf length) for the taxa of Discaria. When it's looking for data for Discaria nitida it drills through Treatment 2 until it finds Discaria nitida in it's hierarchy (in this case, at the top level). It then assumes that all items at the next level down (specimens here) are source items for collation. Similarly, if another treatment were a key to genera of Rhamnaceae, it may use Treatment 2 as its source, drill down to Discaria and collate data from all items at the next level (in this case, species of Discaria). The actual taxa in the hierarchies are rank-free and not constrained in their definition. The only constraint would be that since the "taxon" name is used for identification purposes it would need to be unique in a treatment and common to both treatments. "Taxa" in this sense may equate to formal taxa or they may be species groups, whatever. Note that I still don't know whether all this collation business is actually possible - I just thinks it's important to try something like this. I take my cues in this from Peter (who's pointed out the importance of transparently recording all the steps that lead to a "character" and its "score" since these often hide a multitude of sins) and Bernie Hyland (are you still out there Bernie?), who has built a key to rainforest trees of Australia recording every measurement on every specimen used along the way. We need a way of 1. recording all these data if they're available and 2. automating the process of collating these data to higher levels. For instance, in the above example, if one of the specimens measured as Discaria nitida were redetermined to D. pubescencs, that change could be made at the lowest level treatment, then the higher-level treatments would automatically reflect the possible change in the "circumscription" of the taxa. >> >>>4. Its possible to provide a Character list internally to the >>>treatment or reference an external one. Will it be a requirement >>>that both could be used (i.e. combine the internal and external lists.) >> >>Yes. The way I see it an external character list is a resource, but not a >>constraint. If one is referenced a treatment builder should also be able to >>1. define their own internal characters, >>2. perhaps pick and choose amongst the characters from the referenced list, >>rather than have to use them all. >>I think such flexibility is necessary if lexica are to grow. > >This issue of character lists in advance gives me the heebie-jeebies. >Perhaps I am confusing the issue, but to a certain extent one could argue >that characters flow out of/depend on observations/measurements, so would a >character list in advance unduly constrain? I am thinking of characters >like helobial/cellular/nuclear endosperm, which can and should be >decomposed, embryo development, anther wall development, etc. I agree Peter - I'm worried by predefined characters also. That's why I've been arguing against building lexica into the standard. BUT, it seems to me that if one is to proceed with these treatments there need to be a priori defined characters. I don't know how to leave characters to define themselves (flow out of) the observations/measurements. I've suggested the structure of remote and local character lists to allow people to share resources. If someone creates a list of characters for Rhamnaceae and someone else finds that much of that list will be just the ticket for Vitaceae, then they should be able to share. Perhaps progressively global lexica will grow from this, perhaps not. There seems to be such enthusiasm for lexica that we need to allow for their possibility, but with a bottom-up rather then top-down approach to their creation. Conversely, if other treatment builders want to opt out of constraining lexica altogether, they should be alowed to do so also. My understanding of your problem (I may be wrong) is that you have difficulty with people using qualitative characters as handy (but dangerous) shortcuts for characters that are in reality quantitative. There are no Platonic ideals in the qualitative world, but the quantitative world gets closer. Thus, "crassinucellate" is really an abstraction of a set of measurements. I may argue that it's a handy abstraction, you'd argue that it's a dangerous abstraction, and we'd both be right (this may be a bad example as we'd probably both argue for the danger in crassinucellate, but you get my drift). As the standard stands, I could use Leaves: ovate as a character state in one treatment, and you could use leaf length, width and distance to widest point in yours (or some more sophisticated mathematical descriptor). Can we do any better? Cheers - k

1 0

Re: SDD Specifications Document
by P. F. Stevens 06 Mar '00

06 Mar '00

> >If the rules need to be formulated rather then just named the challenge is >to do it in such a way that an XML parser could do the collation, and a >klutz taxonomist could create their own if an existing one is not sufficient. > There are many ways one can go from measurements to some sort of collation, and this is an area where there is a fair amount of activity - I would have thogutht that specifying rules was not the way to go. >Is this possible or am I talking out of my stovepipe? > >>2. Collated Character source. I see these as essentially >>a drill down mechanism that further identifies a 'bottom-level' >>taxon. > >The necessary thing is to identify a path to the treatment that contains the >data for collation, and the taxa of the lower-level treatment that will >provide the data for a taxon in the current treatment. In the example above >it's perhaps deceptively simple, because the target taxa in the lower-level >treatment (in this case, the specimens) are nested within a taxon with the >same name as the current taxon in the higher-level treatment (Macrolepiota >clelandii or Lilium turkestanicum). This is perhaps the easiest way to do it >(thinking aloud now) - require that the taxon of interest exist somewhere in >the taxon hierarchy in both treatments, then step one down in in the >lower-level treatment to find the elements of interest. Does any of this >make sense to anyone - I'd quite like some support here, if anyone's still >out there. > Are you then going to have to specify all ranks in advance, so that one knows what is up or down? And how would one proceed if unranked naming does indeed become popular? > >>4. Its possible to provide a Character list internally to the >>treatment or reference an external one. Will it be a requirement >>that both could be used (i.e. combine the internal and external lists.) > >Yes. The way I see it an external character list is a resource, but not a >constraint. If one is referenced a treatment builder should also be able to >1. define their own internal characters, >2. perhaps pick and choose amongst the characters from the referenced list, >rather than have to use them all. >I think such flexibility is necessary if lexica are to grow. This issue of character lists in advance gives me the heebie-jeebies. Perhaps I am confusing the issue, but to a certain extent one could argue that characters flow out of/depend on observations/measurements, so would a character list in advance unduly constrain? I am thinking of characters like helobial/cellular/nuclear endosperm, which can and should be decomposed, embryo development, anther wall development, etc. Peter S.

1 0

Re: SDD Specifications Document
by Kevin Thiele 06 Mar '00

06 Mar '00

Leigh, >1. Collation rules. These are currently unspecified. Any objections >if I leave them out of an attempt to express these in XML? We >can revisit and revise this portion as time goes on. I think it's an important thing to try to implement this part, as it's a fundanmental part of the structure but also new and difficult and we need to give people a chance to comment & revise. The three main aspects of the draft standard as documented are 1. The ability (but not necessity) to attribute every data element to a source 2. The ability to collate data from lower-level (e.g. real-data) to higher-level (e.g. synthesized) sources 3. The open-ended hierarchical structure for character and taxon lists. I haven't given a lot of thought to how the collation rules would work in practice. The basic idea is this. Suppose you have one treatment that stores base data (measurements etc) for observed specimens of your taxa, then another treatment that wants to collate parts of these data to a taxon-level character. e.g. One file has Character = "Spore length (um)" Taxon = "Macrolepiota clelandii" Specimen = "CANB4545601" Value = 15 Value = 18 ... Specimen = "CANB233976" Value = 16 ..... The higher-level file wants to collate from this Character = "Spore length (um) Taxon = "Macrolepiota clelandii" Minvalue = 15 Maxvalue = 18 The collation necessary here is simple - "Find the limits of the range". Other simple rules may be "Find the limits and mean", "Find the limits and 25% quartiles" etc. For multistate characters the most common rule will be an additive one - "collect all values for the character and concatenate them": Character = "Petal colour" Taxon = "Lilium turkestanicum" Specimen = "CANB4545601" Value = "Pink" Specimen = "CANB233976" Value = "Purple" Now, I don't know what's the best way of formulating the rules. Would you simply have a set of standard names for rules as above, then leave it to the processing program to do the work, or can the whole thing be embedded in the XML (so that looking at the higher-level treatment with an XML interpreter would show you the collated data)? If the rules need to be formulated rather then just named the challenge is to do it in such a way that an XML parser could do the collation, and a klutz taxonomist could create their own if an existing one is not sufficient. Is this possible or am I talking out of my stovepipe? >2. Collated Character source. I see these as essentially >a drill down mechanism that further identifies a 'bottom-level' >taxon. The necessary thing is to identify a path to the treatment that contains the data for collation, and the taxa of the lower-level treatment that will provide the data for a taxon in the current treatment. In the example above it's perhaps deceptively simple, because the target taxa in the lower-level treatment (in this case, the specimens) are nested within a taxon with the same name as the current taxon in the higher-level treatment (Macrolepiota clelandii or Lilium turkestanicum). This is perhaps the easiest way to do it (thinking aloud now) - require that the taxon of interest exist somewhere in the taxon hierarchy in both treatments, then step one down in in the lower-level treatment to find the elements of interest. Does any of this make sense to anyone - I'd quite like some support here, if anyone's still out there. >Is it reasonable to include character dependencies 'upwards'? Don't know - I need to think about this more. There may be a problem if you mix and match "up" and "down" dependency definitions. The thing is to choose the one that will be most easily defined. >Am I right in believing that multiple character sets could drill >down into the same source (perhaps produced by different >organisations, researchers, techniques). In this case, is >there a principal source for going back upwards? It seems >unfair to expect a treatment to keep track of all other treatments >which point to it (I may be infering too much here). I think you may be off the track (or I may be). A character dependency is a property of a character or a state (depending whether it's an up or down dependency). It should be internal to a treatment, so there's no "keeping track" to worry about. >3. Why should characters only have properties at the lowest level? >What is the 'lowest level' given that drill-down can occur? Could >you outline the reasoning here? You're right - characters at higher levels may have properties also e.g. Leaves venation prominence pattern It may be worth providing notes, illustrations etc for "Leaves" so basic information can be moved up the hierarchy. Similarly, an "up" dependency set to "Leaves" would apply to all lower-level character items. >4. Its possible to provide a Character list internally to the >treatment or reference an external one. Will it be a requirement >that both could be used (i.e. combine the internal and external lists.) Yes. The way I see it an external character list is a resource, but not a constraint. If one is referenced a treatment builder should also be able to 1. define their own internal characters, 2. perhaps pick and choose amongst the characters from the referenced list, rather than have to use them all. I think such flexibility is necessary if lexica are to grow. Possible? Cheers - k

1 0

Re: Progressive Revelation
by Kevin Thiele 03 Mar '00

03 Mar '00

Dear Mike, you're right of course that a Best algorithm can to some extent emulate Progressive Revelation as I described it, but I think a fully worked-out PR model would go further. Also, of course, it's silly to work from a fully populated rectangular matrix if there's no intention of using all of it. Better perhaps to use a data model that has a "matrix" representation or equivalent at its core for the global data and a cloud of "data bits" for the non-global stuff. Cheers - k

1 0

(RQT) DELTA, XDELTA and LucID
by Mike Dallwitz 03 Mar '00

03 Mar '00

> Date: Fri, 7 Jan 2000 > From: Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> > To: TDWG-SDD(a)USOBI.ORG Sorry for taking so long to reply to this - I've been on holiday. > I've been looking through your XDELTA DTD. ... LucID can do a number of > things that DELTA can't, and perhaps XDELTA as it stands can't either. You > should consider these (quotes are from your xdelta.dtd document). We need to clarify what is meant by 'DELTA' in this context. The definition of DELTA data in the 'DELTA Standard' document (included in the program distribution and also available separately as http://biodiversity.uno.edu/delta/standard/standard.exe) is deliberately quite narrow: "The essential components of DELTA-format data are normally the 'character list', the 'taxon or item descriptions', the 'character types', the 'implicit values', and the 'character dependencies'." It was intended that only the 'data matrix' (ITEM DESCRIPTIONS), and a few other directive that were necessary for interpreting the matrix would be defined in the standard (e.g. 'implicit values' and 'character dependencies' affect the interpretation of missing values). In recent discussions, 'DELTA' has been used in the broader sense of some or all of the features (directives) implemented in CONFOR, DeltaAccess, etc. This is appropriate, as standardization requirements now go far beyond the basic data matrix. > 1. Character groups > >> A character group allows the grouping of related characters together. >> DELTA does not enforce the grouping of characters, and neither does >> XDELTA. However the CHARACTER HEADINGS directive seems to assume that >> related characters will be grouped together within a DELTA file (as it >> only specifies the first character that falls under each heading). >> XDELTA therefore allows characters to be grouped in this way. > > I assume that character groups here are equivalent to character sets in > LucID - groupings of characters so that you can e.g. display only leaf > characters in a random-access key. In LucID there is no requirement that > characters in a set are consecutive in the character list. Further, any > character may belong to more than one set - this is very useful so that > e.g. you can have the sets "leaves", "flowers", "petals", "simple" in > which "petal" characters also belong to the "flowers" set, and the > "simple" set comprises a selection of characters from all the other sets. Intkey also implements arbitrary groupings of characters, and I agree with Kevin that this is a useful feature, which needs to be retained. There may also be a need for hierarchical groupings, but the CHARACTER HEADINGS directive does not fill this need. These headings are purely for reading by humans in certain contexts, they are not hierarchical (except perhaps visually by means of font changes), and different ones may be needed in different circumstances. They are often, but not always, the same as the ITEM SUBHEADINGS, which appear in natural-language descriptions. Intkey also implements arbitrary groupings of taxa. > 2. Character state notes > > DELTA stores all notes as plain text, sometimes marked up with its own > typesetting marks, but pretty plain nonetheless. LucID allows builders of > keys to associate a formatted file (txt, rtf, html) with a character > state, to hold explanatory notes. DELTA associates notes with characters, not states. The notes can have RTF markup. Associating notes with the character as a whole is sometimes necessary, and is usually more convenient for the end user. > 3. Taxon multimedia > > LucID allows any number of multimedia files (images, videos, sounds, txt, > rtf, html) files to be associated with a taxon. Each file has a title for > display in a drop down menu. ... It's useful to allow an image to have > both a title and associated text e.g. notes, credits, copyright notice. All of these features are also available in DELTA. In addition, text overlays on images are supported. > 6. Taxon subkeys > > In LucID keys one key may be linked as a child to another. A taxon thus > has the name of a subkey associated with it, so that a user on reaching > the taxon can ask the program to automatically drop to the subkey to > continue the identification. In Intkey, this is handled by the same mechanism described in (3) - the subkeys are just another type of associated file. These would typically be other Intkey keys, but could be any file with an associated application, e.g. a conventional key in HTML, or, presumably, a LucID key. -- Mike Dallwitz CSIRO Entomology, GPO Box 1700, Canberra ACT 2601, Australia Phone: +61 2 6246 4075 Fax: +61 2 6246 4000 Email: md(a)ento.csiro.au Internet: biodiversity.uno.edu/delta/

1 0

Re: Progressive Revelation
by Robert A. (Bob) Morris 03 Mar '00

03 Mar '00

Kevin Thiele writes: > Date: Fri, 3 Mar 2000 15:58:08 +1100 > From: Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> > To: TDWG-SDD(a)usobi.org > Subject: Re: Progressive Revelation > > Dear Mike, > > you're right of course that a Best algorithm can to some extent emulate > Progressive Revelation as I described it, but I think a fully worked-out PR > model would go further. Also, of course, it's silly to work from a fully > populated rectangular matrix if there's no intention of using all of it. > Better perhaps to use a data model that has a "matrix" representation or > equivalent at its core for the global data and a cloud of "data bits" for > the non-global stuff. i.e., an object oriented database ..... :-) Bob Morris > > Cheers - k

1 0

Re: SDD Specifications Document
by Jean-Marc Vanel 02 Mar '00

02 Mar '00

Kevin Thiele a écrit : > Jean-Marc's work does indeed need appraisal by the group in relation to our > goals. My understanding from a brief perusal of his site is that he's trying > to establish a franework for creating a set (one day he hopes for a global > set) of descriptions of plants on the web, Yes, by merging Flora descriptions with character-based data à la Delta. > the data structured in such a way > as to be queryable as well as readable. Yes, it will make biological knowledge really accessible. For the query aspect, I try to structure data in a way suitable to export to several kind of programs: - Artificial Intelligence - Delta and LucID - databases, relational and OO > Two likely differences between the present attempt and his may be > > 1. that we need to create something more flexible so that any botanist can > capture any aspect of systematic data for any taxon - i.e. a Stipa can be > described just as fully as a Taraxacum and a Nepenthes, with data ranging > from morphology to anatomy, phytochemisty and gene sequences (the latter is > the only easy one). Jean-Marc's thing seems to me to be of the fairly > inflexible variety This assertion is "present by misinterpretation". The set of properties and organs is of course not limited, although for convenience a standard set will be provided. > 2. Jean-Marc's model is directed towards capturing descriptions. We need to > also allow for identification data for the range of key programs available > and in the future. Thus, things like character sets, dependencies, allowing > for misinterpretations etc may perhaps have little meaning in Jean-Marc's > domain, but are critical in ours. You must not doubt about my commitment to integrate those things in a general model. I'm currently searching funding/sponsorship because now I have to work on this after normal work hours. Anyway next monday I'll present a more detailed model and example, showing how data and meta-data published on different sites can be used together. Cheers JMV -- <person> <first_name>Jean-Marc</first_name> <name>Vanel</name> <project>Worlwide Botanical Knowledge Base - making botany available on Internet <a href="http://wwbota.free.fr/" >site</a> </project> <homePage>http://jmvanel.free.fr/</homePage> <a href="mailto:jmvanel@free.fr">mail (eventually put "wwbota" in subject to route your mail in relevant folder)</a> </person>

1 0

Re: SDD Specifications Document
by Leigh Dodds 02 Mar '00

02 Mar '00

Hi, I got some time last night to start giving this some thought. I've got some questions: 1. Collation rules. These are currently unspecified. Any objections if I leave them out of an attempt to express these in XML? We can revisit and revise this portion as time goes on. 2. Collated Character source. I see these as essentially a drill down mechanism that further identifies a 'bottom-level' taxon. Is it reasonable to include character dependencies 'upwards'? Am I right in believing that multiple character sets could drill down into the same source (perhaps produced by different organisations, researchers, techniques). In this case, is there a principal source for going back upwards? It seems unfair to expect a treatment to keep track of all other treatments which point to it (I may be infering too much here). 3. Why should characters only have properties at the lowest level? What is the 'lowest level' given that drill-down can occur? Could you outline the reasoning here? 4. Its possible to provide a Character list internally to the treatment or reference an external one. Will it be a requirement that both could be used (i.e. combine the internal and external lists.) Cheers, L.

1 0

Re: SDD Specifications Document
by Leigh Dodds 02 Mar '00

02 Mar '00

> At the very least, we should make sure that they're compatible, perhaps > seeing Jean-Marc's as a special-purpose subset of the general. Thats my current interpretation. Jean-Marc's model seems to presuppose that the identification of the organism has already taken place. With that identification involving the allocation of properties. Or have I mis-interpreted it? Jean-Marc, perhaps you could provide some additional examples? L.

1 0