January 2000 - tdwg-content

(LEX) Containment Hierarchies
by Vincent, Leszek 31 Jan '00

31 Jan '00

Dear subscribers to TDWG I've been 'listening' to the contributions on Lexicons, XML etc. for a while now and thought I should seek your comments because your contributions have sparked several developments in my thinking. I'm wrestling with the development of a hierarchy (containment hierarchies) of systematic descriptors for corn/maize/ Zea mays for the Missouri Maize Project (see URLs below). While this is challenging for maize/corn on its own, an added challenge is my desire to develop containment hierarchies which are compatible with databases for other grass crops e.g. rice (Oryza). To add to the challenge I have the ideal of trying to accommodate the 'interests' of similar databases for various DICOT crops e.g. soybean, cotton. While I do appreciate that such an all encompassing approach is maybe impossible to achieve I thought that it would still be good to get some other folks opinions on this. Having some kind of 'generic' structure which accommodates monocots & dicots should greatly facilitate the interoperability of these databases. It's the interoperability scenario which really got me on to thinking about this issue. Conducting simultaneous searches of dicot & monocot databases for morphological data down to molecular (e.g. sequence) data is not too far fetched. But I'm battling to find info. which addresses/refers to such 'generic' standards. I'd be grateful to have your thoughts on the above. Sincerely, - Leszek Vincent xoxoxoxoxoxoxoxoxoxoxoxoxoxoxox P. Leszek D. Vincent Ph.D., FLS Plant Science Unit, Dept. of Agronomy, 209 Curtis Hall, University of Missouri-Columbia, MO 65211-7020, USA. Ph: (573) 884-3716 (Agronomy), (573) Fax:(573) 884-7850; Ph/Fax (Home): (573) 441-1228; Email: Leszek(a)missouri.edu Plant Systematist on the Missouri Maize Project - NSF award 9872655 - (http://www.cafnr.missouri.edu/mmp/ and http://www.agron.missouri.edu/) xoxoxoxoxoxoxoxoxoxoxoxoxoxoxox

1 0

Re: (LEX) Global but non-universal lexicons
by Gregor Hagedorn 17 Jan '00

17 Jan '00

> There are a number of issues: > - how are character lists imported if a computer is not connected to > the net? It was already said that we need a local copy. I think this is quite natural especially because I believe we will have proprietary database formats the most applications will work on, and not directly XML. > - how are multiple imported character lists merged - which takes > precedence (local or remote) We need to be able to combine both character definition and item description in a way that some characters may go across multiple item subsets but also some character used only in certain items. If items are then reduced to the level of taxa in a summarize/synthesize operation, even more complex structures can result. > - versioning of character lists Of course, with major versions in which meaning is changed, minor version in which only corrections and additions are made, and perhaps subminor with corrections only. > - what if a remote character list is (or becomes) invalid (related > to versioning) Globally unique state IDS may consist of a local ID (not the sequence number), the name and version of the standard. I believe the merging, updating, controlling business of versioned character definition standards is one of the main things why we need more structure in the character definition (see basic property discussion etc.). --- I believe we need: Globally unique IDs for _states_ (perhaps for character and headings as well, not sure). The ID should be independend of ID numbers like in DELTA and DeltaAccess, that are simultaneously ordering numbers. A local copy that refers to a published standard. I believe it would be good if the future standard allows manual links e.g. to a standard published in print. I imagine an attribute in which the author of a character definition acertains that in his or her opinion the meaning of this data item is identical to a standard, even if the wording may have been slightly changed. The problem is that ANY character definition standard needs to be partly changed locally. Certain attributes of the character definition, esp. those that are language specific (character and state wordings, including character and state specific number and date formats) or refer to formatting and structuring (ParagraphLink, SentenceLink, CommaLink for natural language, Sequential ordering numbers, Heading, etc.) are highly likely to be in need for change. It may be desirable to split the character definition into a logical part, changes of which imply changes of contents, and a formatting part, which is always used as template only, and can be freely changed. (This implies that the character name needs to be doubled/split into a global development name, probably in English, and a name that is displayed in local language on the user interface.) Gregor ---------------------------------------------------------- Gregor Hagedorn G.Hagedorn(a)bba.de Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

(LEX) Global but non-universal lexicons
by Kevin Thiele 15 Jan '00

15 Jan '00

Question for XMLHeads - can an XML DTD refer to a remote XML document as a resource? If it can, could we solve the lexicon problem (which is, that some people like the idea of a lexicon and others don't) like this? A lexicographer puts onto the web an XML resource that provides a character/state list for a given universe of interest (seed plants/all life/the flagellated sporozoans of deepsea hydrothermal vents, whatever). If other workers like the character list and think they can live with it, then they include a URL to the lexicon in their document, and the lexicon is used as their character list. But if no available lexicon works for a given user, then they can define their own character list. That way, perhaps we'll all be happy. If this model works, it could perhaps also allow for extensibility ie mixed lists comprising remote lexica and extra document-defined characters. A problem with such lexica would be data security - what would happen if the lexicon's server crashes permanently - and how to allow for evolving lexica without invalidating previous treatments. Any good? Cheers - k

1 0

Re: (LEX) Global but non-universal lexicons
by Leigh Dodds 15 Jan '00

15 Jan '00

> Question for XMLHeads - can an XML DTD refer to a remote XML document as a > resource? Judging by the example you outline below I believe you want to link together two XML documents, rather than a DTD and a document. The latter doesn't bring any benefits in this case... > A lexicographer puts onto the web an XML resource that provides a > character/state list for a given universe of interest (seed plants/all > life/the flagellated sporozoans of deepsea hydrothermal vents, > whatever). If other workers like the character list and think they can live > with it, then they include a URL to the lexicon in their document, and the > lexicon is used as their character list. But if no available lexicon works for a > given user, then they can define their own character list. That way, perhaps we'll all > be happy. This is exactly the sort of thing I was hoping would happen. The advantage being, that character and item information are separated. Therefore a character list can be reused, either between groups (across the web, departments, whatever), or reused by the same researcher when producing a new taxonomy. <aside> I debated this structure in XDELTA, but plumbed for a single file comprising all information. I think I commented on this somewhere in the DTD. In my second iteration I'm going to split them into two files - actually I'm thinking that a modular approach may be useful : define a module for expressing character/state information and a module for expressing item information. I've tentatively identified a third module which comprising textual markup, links to additional resources, etc. Breaking the problem down into 3 sections should make tackling it a lot easier. The schema itself can then be modular as well. </aside> There is a standard for document linking - XLink - which allows two documents to be linked, and even portions of a document to be selected. So we should be able to use this kind of feature quite easily: e.g. <characters import="http://www.foo.org/characters.xml" /> or <characters import="c://my-project/characters.xml" /> (note these are not XLinks, I don't have the spec to hand for a clear example) > If this model works, it could perhaps also allow for > extensibility ie mixed lists comprising remote lexica and extra > document-defined characters. Yes. > A problem with such lexica would be data security - what would > happen if the lexicon's server crashes permanently - and how to allow for > evolving lexica without invalidating previous treatments. There are a number of issues: - how are character lists imported if a computer is not connected to the net? - what happens if a server is not available? - how are multiple imported character lists merged - which takes precedence (local or remote) - versioning of character lists - what if a remote character list is (or becomes) invalid (related to versioning) - how are a item list, and its associated character list (and any additional supporting files) packaged together for either delivery across the internet, or on traditional media like CDs. The good news is that these issues are not isolated to this particular problem and we can benefit from solutions defined elsewhere. > Any good? I think so! Cheers, L.

1 0

Re: (LEX) Global but non-universal lexicons
by Robert A. (Bob) Morris 15 Jan '00

15 Jan '00

Oh, no! Leigh gave the Long Answer. But it was more worth reading than mine was going to be. :-) Bob

1 0

(LEX) Global but non-universal lexicons
by Robert A. (Bob) Morris 15 Jan '00

15 Jan '00

Kevin Thiele writes: > Date: Sat, 15 Jan 2000 17:10:42 +1100 > From: Kevin Thiele <kevin.thiele(a)PI.CSIRO.AU> > > Question for XMLHeads - can an XML DTD refer to a remote XML document as a > resource? There's a short answer, a long answer, and a quibble about this first question. Like most quibbles, the quibble is the longest. The short answer is: Yes, this is routinely done. The long answer: There's little point in putting the long answer into this group. The quibble: As Leigh and others have suggested, there is substantial motivation to specify data structuring in XML-Schema instead of the older DTD formalisms, even if the result is then translated into a DTD so that more of current tools can use it. This is especially true here, because the mechanisms for importing definitions are less clunky in Schema than in DTD (largely because DTD arises from SGML, which predates the web by 20 years). If nothing else, the vocabulary of Schema allows more definitive descriptions of data types and so forms a better reference standard than a DTD. Although I count myself more an XMLTongue or XMLNose, I think it fair to say that the XMLHeads are probably thinking: "whenever the TaxonHead's say DTD, we should privately understand Schema" > ... > > If it can, could we solve the lexicon problem (which is, that some people > like the idea of a lexicon and others don't) like this? > > A lexicographer puts onto the web an XML resource that provides a > character/state list for a given universe of interest (seed plants/all > life/the flagellated sporozoans of deepsea hydrothermal vents, whatever). If > other workers like the character list and think they can live with it, then > they include a URL to the lexicon in their document, and the lexicon is used > as their character list. But if no available lexicon works for a given user, > then they can define their own character list. This is pretty much one of the things that XML is designed to make feasible. > That way, perhaps we'll all be happy. This is pretty much one of the things that XML is designed to make feasible, too. :-) > If this model works, it could perhaps also allow for extensibility ie mixed > lists comprising remote lexica and extra document-defined characters. > > A problem with such lexica would be data security - what would happen if the > lexicon's server crashes permanently - and how to allow for evolving lexica > without invalidating previous treatments. Definition evolution and resource location problems are so pervasive that they are an important part of the XML specifiers' work (and that of W3C in general). FWIW it's not that hard to make an application use a local copy of a resource if it can't find the network copy. Backward compatibility of course rests not so much on mechanisms enabling it as with the authors actually accomplishing it. The task is fraught with the risk of thinkos. (A thinko is to your brain as a typo is to your fingers. The term was coined by David Fuchs, one of the architects of FrameMaker and before that an implementor at Stanford of the portable version of TeX, which had originally been written in the SAIL language and only ran to a handful of PDP-10's.) > > Any good? Smells good to me... > > Cheers - k Bob Morris

1 0

Re: XDELTA and LucID
by Kevin Thiele 10 Jan '00

10 Jan '00

At 12:54 07/01/00 -0500, Joe Kirkbride wrote: >>>From a strictly practical point of view, multistate characters must be >allowed to function with just a single state. In an ongoing study, a >character such as corolla pubescence is included in the character list. >After collecting data from one, or just a few taxa, you may only have a >single character state known. As the research progresses, additional >character states will be discovered and added to the character list. In >the preliminary stages of any taxonomic study the number of known >character states is usually very low, and must be allowed for! Good point, and you're right. >My major concern with this process and its discussions is that the results >will be something that works and is practical. The most computionally >perfect system may well turn out to be impractical and unworkable. I sincerely >hope that it is kept firmly in mind that the results have to work in the >real world, not the ideal world. I couldn't agree more. But we also need to use this process to move ahead. And we need to make sure that the taxonomists on the list keep an eagle eye on the computer jocks to make sure they don't go off to Saturn. Cheers - k

1 0

Re: XDELTA and LucID
by Kevin Thiele 10 Jan '00

10 Jan '00

Leigh >> 2. Character state notes >> >> DELTA stores all notes as plain text, sometimes marked up with its own >> typesetting marks, but pretty plain nonetheless. LucID allows builders of >> keys to associate a formatted file (txt, rtf, html) with a >> character state, >> to hold explanatory notes. One option would be to include xml markup as >> comment (would this be messy if the notes included embedded >> images, hotlinks >> etc). Another would be to treat state notes as you do images, with an href >> pointer to a file. LucID also allows sound files to be associated with a >> state (imagine a key to cicadas or frogs with calls as characters). > >How about if the text allowed (X)HTML markup to me used. This would >allow the full range of HTML expression (tables, bold, images, etc). >However it would mean tht processors (e.g. a stylesheet) would need to >understand HTML. I suggest that a subset of the available markup >would be enough in most cases. A pointer to additional document(s) >would also be useful. Yes, I think we'd need to allow for both - embedded (X)HTML and pointers to external documents, from which an application could fetch stuff if necessary. I have a key with 800 states each with html notes, images etc. Embedding all this would make for one hell of a document. >> 3. Taxon multimedia >> >> LucID allows any number of multimedia files (images, videos, sounds, txt, >> rtf, html). > >Ideally an XML format would allow this as well. A generic 'link' to >additional information could be used. It then depends on the application >to decide whether it can fetch, and handle the particular >multimedia format. Ditto as for state notes >> 6. Taxon subkeys >> >> In LucID keys one key may be linked as a child to another. A >> taxon thus has the name of a subkey associated with it, so that a user on >> reaching the taxon can ask the program to automatically drop to the subkey >to >> continue the identification. > >If I understand this correctly you mean that the subkey provides additional >information regarding the taxon. To take a trivial example, the first >key may identify a plant to the Family level, whilst the subkey had >additional information - e.g. to the species? > >If so, then this is related to the document linking problem below... Yes it's a linking issue. By allowing links from one document to another you can build hierarchies of documents e.g. I have a key to Families of Flowering Plants of Australia: the family Solanaceae has a link to a separate key to genera and species of that family, and some of these taxa have further subkey links. The hierarchy of keys-subkeys is being built up in an ad hoc fashion. >> 8. General Information >> >> Not all information in a key or other treatment can be associated with >> character states or taxa. Some is associated with the treatment as a whole >> e.g. introductory notes, notes on using the key, general notes for the >> parent taxon of the key (e.g. a key to genera of Grossulariaceae >> may have a description of the family) etc. In LucID this information may >> comprise html files, images etc just as with taxa and states. This could >be >> easily dealt with by expanding your DTD. A special case is an "About this >key" >> file. This needs much more than is allowed in the document description: > >This is one area that I don't think we've touched upon to a great extent >(I could be wrong). Heres a question: > >- is it enough to just have an 'additional information' element >so that any required explanatory notes, etc can be added there. > >Heres my answer (and I'd welcome comment): > >No its not. It would be useful to try to quantify this additional >information, >so that a generic catch-all placeholder isn't used. This sidesteps issues >such as that shown by the comment marker in DELTA which ends up being >used for all sorts of other uses. Its this kind of information which is >highly useful both to the user, and the application developer: the user >has a defined place to add his/her information, the developer has explicit >cues on how particular sections of information should be used. > >So to start a running list: > - introduction > - about this key? > - parent taxon / related keys > - usage guidelines > - other meta information? > (see above - copyright, funding body, author, change history, etc). > >Anything else? I wouldn't like to try to force people into boxes here. It may be that there are some "General Info" topics that are fairly common, but you can't predict how someone will want to structure this. For instance, someone writing a key to insects might want to have a topic covering collecting techniques, another would want general info on biological control. A user here who has published a key to eucalypts of Australia has included a synoptic classification. Best option again I think would be to be very general and allow flexible links to external files with topic names. >You've commented that you're just getting to grips with DTDs, is >it going to cause a great problem if I design the next version of >XDELTA using XML Schemas [1,2]? Schemas provide a great deal more >flexibility >when defining document structures, as well as validation (not >that I expect the latter issue to matter to a huge degree in this >context). The are some structures in the XDELTA DTD which I'm not >particularly happy with, but are imposed by the limitations of >DTD syntax. I'm hoping that Schemas will remove some of these constraints. > >When I get chance to attempt another iteration of XDELTA (hopefully >quite shortly), I can fully document the Schema to bring people >up to speed - i.e. use XDELTA as a tutorial piece. Great, and you're a bloody marvel! >I also don't want to appear to be leading everyone by the nose >towards an XML based format, or one which I've produced myself. >So please consider XDELTA, as I've always maintained, as a discussion >point. I agree it's still a discussion point but it's the best we have so far to start bringing this discussion towards a product, doing which will bring out extra points, flaws, limitations etc. If you're willing to do the work on it, then I'm sure we're willing to comment and pick the holes. Cheers - k

1 0

Re: XDELTA and LucID
by Leigh Dodds 07 Jan '00

07 Jan '00

Oops, forgot my references: > You've commented that you're just getting to grips with DTDs, is > it going to cause a great problem if I design the next version of > XDELTA using XML Schemas [1,2]? Schemas provide a great deal more > flexibility [1] Schema Structures - http://www.w3.org/TR/xmlschema-1/ [2] Schema Datatypes - http://www.w3.org/TR/xmlschema-2/ Cheers, L.

1 0

Re: XDELTA and LucID
by Leigh Dodds 07 Jan '00

07 Jan '00

> I've been looking through your XDELTA DTD. The group has already commented > that it's "merely" a literal mapping of DELTA, but it's a very > useful start :-) Thanks I was hoping it would be a discussion point. > 1. Character groups > > I assume that character groups here are equivalent to character sets in > LucID - groupings of characters so that you can e.g. display only leaf > characters in a random-access key. In LucID there is no requirement that > characters in a set are consecutive in the character list. Further, any > character may belong to more than one set - this is very useful > so that e.g. > you can have the sets "leaves", "flowers", "petals", "simple" in which > "petal" characters also belong to the "flowers" set, and the "simple" set > comprises a selection of characters from all the other sets. Yes you've interpreted correctly, and XDELTA is limited in that characters can only be within a single group, and must be grouped together. This did concern me when I was initially writing the DTD, but based on the fact that DELTA had the same limitation I thought it would be good enough for a first cut. LucID is obviously more flexible. > 2. Character state notes > > DELTA stores all notes as plain text, sometimes marked up with its own > typesetting marks, but pretty plain nonetheless. LucID allows builders of > keys to associate a formatted file (txt, rtf, html) with a > character state, > to hold explanatory notes. One option would be to include xml markup as > comment (would this be messy if the notes included embedded > images, hotlinks > etc). Another would be to treat state notes as you do images, with an href > pointer to a file. LucID also allows sound files to be associated with a > state (imagine a key to cicadas or frogs with calls as characters). How about if the text allowed (X)HTML markup to me used. This would allow the full range of HTML expression (tables, bold, images, etc). However it would mean tht processors (e.g. a stylesheet) would need to understand HTML. I suggest that a subset of the available markup would be enough in most cases. A pointer to additional document(s) would also be useful. > 3. Taxon multimedia > > LucID allows any number of multimedia files (images, videos, sounds, txt, > rtf, html). Ideally an XML format would allow this as well. A generic 'link' to additional information could be used. It then depends on the application to decide whether it can fetch, and handle the particular multimedia format. > 4. Images > It's useful to allow an image to have both a title and associated > text e.g. > notes, credits, copyright notice. Defining the title separately from the > text allows an application to retrieve and display just this > without the rest. More generally I think that (from other comments on the list), it would be useful to attach this kind of information : notes, credit, copyright, origin, etc in multiple places in the format - including textual descriptions. > 5. Character values > > LucID allows a richer assignation of character state values to > taxa. Thus, a taxon may be scored as having state 1 as its normal state, > state 2 rarely, state 3 by misinterpretation, and be unknown for state 4. The > full list is: > > normally present > rarely present > uncertain > present by misinterpretation > rarely present by misinterpretation > > These are important. Note that the proposed extensions to DELTA > incorporate some or all of these. Gregor has already proposed a system of > modifiers to allow this. This sounds like a strong requirement then. > > 6. Taxon subkeys > > In LucID keys one key may be linked as a child to another. A > taxon thus has the name of a subkey associated with it, so that a user on > reaching the taxon can ask the program to automatically drop to the subkey to > continue the identification. If I understand this correctly you mean that the subkey provides additional information regarding the taxon. To take a trivial example, the first key may identify a plant to the Family level, whilst the subkey had additional information - e.g. to the species? If so, then this is related to the document linking problem below... > 7. A small point: > > > A multistate character must contain at least one state. > > Surely a multistate character must contain at least two states. Yes, You're quite right! :) > 8. General Information > > Not all information in a key or other treatment can be associated with > character states or taxa. Some is associated with the treatment as a whole > e.g. introductory notes, notes on using the key, general notes for the > parent taxon of the key (e.g. a key to genera of Grossulariaceae > may have a description of the family) etc. In LucID this information may > comprise html files, images etc just as with taxa and states. This could be > easily dealt with by expanding your DTD. A special case is an "About this key" > file. This needs much more than is allowed in the document description: This is one area that I don't think we've touched upon to a great extent (I could be wrong). Heres a question: - is it enough to just have an 'additional information' element so that any required explanatory notes, etc can be added there. Heres my answer (and I'd welcome comment): No its not. It would be useful to try to quantify this additional information, so that a generic catch-all placeholder isn't used. This sidesteps issues such as that shown by the comment marker in DELTA which ends up being used for all sorts of other uses. Its this kind of information which is highly useful both to the user, and the application developer: the user has a defined place to add his/her information, the developer has explicit cues on how particular sections of information should be used. So to start a running list: - introduction - about this key? - parent taxon / related keys - usage guidelines - other meta information? (see above - copyright, funding body, author, change history, etc). Anything else? > It will sometimes happen that several collaborators will share a common > character list and score different taxa, as you describe. But it will also > happen that the collaborators share a taxon list and score different > characters. This may be simple, but the same issues (interdocument > references, conflicts etc) need to be addressed. I'd hadn't thought of this - its the reverse of what I'd originally expected. I expect the referencing and conflict resolution mechanism to be tricky to solve, and I'm hoping we can delay some of those decisions until such time as a document structure has been determined - mainly because not all the issues will be obvious until that point. ------------------------------ You've commented that you're just getting to grips with DTDs, is it going to cause a great problem if I design the next version of XDELTA using XML Schemas [1,2]? Schemas provide a great deal more flexibility when defining document structures, as well as validation (not that I expect the latter issue to matter to a huge degree in this context). The are some structures in the XDELTA DTD which I'm not particularly happy with, but are imposed by the limitations of DTD syntax. I'm hoping that Schemas will remove some of these constraints. When I get chance to attempt another iteration of XDELTA (hopefully quite shortly), I can fully document the Schema to bring people up to speed - i.e. use XDELTA as a tutorial piece. As an aside - if anyone wishes additional explanation of the current DTD then I'm more than happy to oblige. I also don't want to appear to be leading everyone by the nose towards an XML based format, or one which I've produced myself. So please consider XDELTA, as I've always maintained, as a discussion point. L.

1 0