- tdwg-content - lists.tdwg.org

Re: XML gateways
by Jim Croft 20 Mar '01

20 Mar '01

>Virtually all major relational dbms now output XML, if crude, and >there is not really much reason people shouldn't be doing so on some >agreed upon alternative http port, say 8883. I would advise against this... many institutional firewalls prohibit all packets except through port 80... this would effectively prevent all of our staff from playing with the outside world... bummer, but them's the rules... > It's not critical to wait >for an agreed upon uniform schema to serve XML. it's even possible to >infer a schema or DTD from data instances, and anyway a schema or DTD >is not necessary to do much of what XML is good for. but it is nice to be able to look at them to see what people think they are doing... :) jim

1 0

Re: XML gateways
by Jones, Tim (CMR, Hobart) 20 Mar '01

20 Mar '01

Nice work!! I am wondering though, if you are federating heterogenous XML databases why do you need to scrape the returned HTML but instead operate on the XML or are they not XML enabled? Tim Jones --------------------------------------------------------- Database Manager Centre for Research on Introduced Marine Pests GPO Box 1538, Hobart Tas 7000, Australia. Phone : (03) 62325222 (switch), (03) 62325213 (direct) Mobile: 0411 560057 Fax : (03) 62325485 E-mail: tim.jones(a)marine.csiro.au --------------------------------------------------------- > -----Original Message----- > From: Robert A. (Bob) Morris [mailto:ram@CS.UMB.EDU] > Sent: Tuesday, 20 March 2001 15:18 > To: TDWG-SDD(a)USOBI.ORG > Subject: XML gateways > > > Here's some candy: we've built simple sample gateways to the USDA > Plants site and the Simthsonian Mammal Species of the World > site. Server-side perl makes up cgi requests to the site, parses > the returned html and emits XML. Of course, it is a very > unstable solution since > changes to the site's html will break it. But we did it because we > experiment with federating heterogeneous XML biodiversity databases. > > You can nibble at > http://www.cs.umb.edu/efg/xml/junwan/DEVELOP/gateway/gateway2.html > > > Bob Morris > Jun Wan >

1 0

Re: XML gateways
by Dave Vieglais 20 Mar '01

20 Mar '01

>-----Original Message----- >From: TDWG - Structure of Descriptive Data [mailto:TDWG-SDD@USOBI.ORG]On >Behalf Of Robert A. (Bob) Morris >Sent: Tuesday, March 20, 2001 10:07 AM >To: TDWG-SDD(a)USOBI.ORG >Subject: Re: XML gateways > > >Jim Croft writes: > > Date: Wed, 21 Mar 2001 00:25:09 +1100 > > From: Jim Croft <jrc(a)anbg.gov.au> > > To: TDWG-SDD(a)usobi.org > > Subject: Re: XML gateways > > ... > >Sad, but true. We should kick around other ideas to solve this >problem, though a conventional port may be the easiest for data >sources to implement. How about: > >- A convention whereby if > http://<host>/<filePath>/<cgiquery> >yields HTML then > http://<host>/<filePath>/xml/<cgiquery> >yields XML > >-A convention whereby people set up virtual hosts---pretty easy in >most web servers---so that if > http://<host>/<filePath>/<cgiquery> >yields HTML then something like > http://xml-<host>/<filePath>/<cgiquery> >yields XML > >Hopefully all of this is temporary, since a well crafted GBIF should >provide for discovery of URL, query syntax, and return schema. My real >point is that current versions of Oracle, SQL-Server, FileMaker, and >Access(?) can already emit XML without much(?) effort on the part of >the data source operators, and doing so would let people proceed to >build interesting distributed applications. There are initiatives to resolve each of the issues brought up here that are being worked on in several industry sectors: 1. Service discovery is addressed by UDDI (Universal Description, Description and Integration). 2. Service description by WSDL (Web Service Description Language) which is an XML document format for describing methods and data exposed by a web services. 3. IGIR (Interface for Generic Information Retrieval) provides a generic format for information retrieval via HTTP GET or POST requests that is easy to implement. The queries are sent as a structure easily translated into a number of common query syntaxes (SQL for example). Results are contained within a consistently formatted document. IGIR is optionally stateful (it's up to the server to decide)- so a query that generates a very large result set need not be sent back to the client all in one hit. IGIR also supports a SOAP interface (that's what it is originally designed for), so programmatic access to information resources becomes pretty simple. A search and retrieval scenario would work something like this: A client hits on the UDDI directory, looking for services that support IGIR and are "biologically relevant". A list of services supporting IGIR are returned, and the client broadcasts a query to them and waits for results to start coming back. Since the records are all formatted in a similar manner, it is a simple process for the client to merge the results and present to the user a nicely formatted set of results. Anyway, that's how things should work. To support a broadcast query, obviously the targets must understand the contex of the query terms- so groups within a particular domain (such as biological collections for example) will need a set of "well known" access points (searchable fields). Similarly, to simplify the merging of results that come back from such a query, it would be convenient if there was an agreed record structure that was shared between data providers. The xml output by most database vendors is a hack designed to allow easy transport of database records using http. Things are getting better with xsd and so forth, but the structure of the resulting xml documents will still reflect the structure of the databases rather than any agreed upon common format. Which brings us back to the "biological collections profile" concept that was raised at the last TDWG meeting. Would anybody like to volunteer a set of access points (searchable fields) that are relevant for biological collections? Similarly for the record structure? The DarwinCore elements start to address these issues, but are somewhat deficient in several respects. They might make a reasonable starting place though- The description with a comment section (add your gripes etc) is available at: http://tsadev.speciesanalyst.net/DarwinCore/Darwin_core.asp Most of you know that the DarwinCore was originally developed as a Z39.50 profile, hence some of the language may seem a little odd. But the main thing to take a look at is the list of access points- which are useful from a biological collections point of view, and what's missing? There are obviously other domains of biological databases beside just collection databases. Any suggestions for commonality between them all? Cheers, Dave V. ========================= David A. Vieglais University of Kansas Natural History Museum & Biodiversity Research Center http://www.nhm.ukans.edu

1 0

Re: XML gateways
by Robert A. (Bob) Morris 20 Mar '01

20 Mar '01

Jim Croft writes: > Date: Wed, 21 Mar 2001 00:25:09 +1100 > From: Jim Croft <jrc(a)anbg.gov.au> > To: TDWG-SDD(a)usobi.org > Subject: Re: XML gateways > > >Virtually all major relational dbms now output XML, if crude, and > >there is not really much reason people shouldn't be doing so on some > >agreed upon alternative http port, say 8883. > > I would advise against this... many institutional firewalls prohibit all > packets except through port 80... this would effectively prevent all of our > staff from playing with the outside world... bummer, but them's the rules... Sad, but true. We should kick around other ideas to solve this problem, though a conventional port may be the easiest for data sources to implement. How about: - A convention whereby if http://<host>/<filePath>/<cgiquery> yields HTML then http://<host>/<filePath>/xml/<cgiquery> yields XML -A convention whereby people set up virtual hosts---pretty easy in most web servers---so that if http://<host>/<filePath>/<cgiquery> yields HTML then something like http://xml-<host>/<filePath>/<cgiquery> yields XML Hopefully all of this is temporary, since a well crafted GBIF should provide for discovery of URL, query syntax, and return schema. My real point is that current versions of Oracle, SQL-Server, FileMaker, and Access(?) can already emit XML without much(?) effort on the part of the data source operators, and doing so would let people proceed to build interesting distributed applications. >...

1 0

Re: XML gateways
by Robert A. (Bob) Morris 19 Mar '01

19 Mar '01

They are not XML enabled. We wish they were since both are pretty interesting databases. Virtually all major relational dbms now output XML, if crude, and there is not really much reason people shouldn't be doing so on some agreed upon alternative http port, say 8883. It's not critical to wait for an agreed upon uniform schema to serve XML. it's even possible to infer a schema or DTD from data instances, and anyway a schema or DTD is not necessary to do much of what XML is good for. We hope people will pay more attention to getting out their "raw" data in XML and publishing their /query/ interfaces so that applications can use them. When we do something like our gateway, we have to put a lot of time into reverse engineering the query syntax because so many sites believe that the only use of their data is through the forms they provide, or some import of their entire dataset by a prior arrangement. Bob Jones, Tim (CMR, Hobart) writes: > Date: Tue, 20 Mar 2001 15:29:55 +1100 > From: "Jones, Tim (CMR, Hobart)" <tim.jones(a)MARINE.CSIRO.AU> > To: TDWG-SDD(a)usobi.org > Subject: Re: XML gateways > > Nice work!! > > I am wondering though, if you are federating heterogenous XML databases why > do you need to scrape the returned HTML but instead operate on the XML or > are they not XML enabled? > > Tim Jones > > --------------------------------------------------------- > Database Manager > Centre for Research on Introduced Marine Pests > GPO Box 1538, Hobart Tas 7000, Australia. > Phone : (03) 62325222 (switch), (03) 62325213 (direct) > Mobile: 0411 560057 > Fax : (03) 62325485 > E-mail: tim.jones(a)marine.csiro.au > --------------------------------------------------------- > > > > > > -----Original Message----- > > From: Robert A. (Bob) Morris [mailto:ram@CS.UMB.EDU] > > Sent: Tuesday, 20 March 2001 15:18 > > To: TDWG-SDD(a)USOBI.ORG > > Subject: XML gateways > > > > > > Here's some candy: we've built simple sample gateways to the USDA > > Plants site and the Simthsonian Mammal Species of the World > > site. Server-side perl makes up cgi requests to the site, parses > > the returned html and emits XML. Of course, it is a very > > unstable solution since > > changes to the site's html will break it. But we did it because we > > experiment with federating heterogeneous XML biodiversity databases. > > > > You can nibble at > > http://www.cs.umb.edu/efg/xml/junwan/DEVELOP/gateway/gateway2.html > > > > > > Bob Morris > > Jun Wan > > >

1 0

XML gateways
by Robert A. (Bob) Morris 19 Mar '01

19 Mar '01

Here's some candy: we've built simple sample gateways to the USDA Plants site and the Simthsonian Mammal Species of the World site. Server-side perl makes up cgi requests to the site, parses the returned html and emits XML. Of course, it is a very unstable solution since changes to the site's html will break it. But we did it because we experiment with federating heterogeneous XML biodiversity databases. You can nibble at http://www.cs.umb.edu/efg/xml/junwan/DEVELOP/gateway/gateway2.html Bob Morris Jun Wan

1 0

Re: Report, please -> apologies
by Gregor Hagedorn 27 Nov '00

27 Nov '00

Dear colleagues, My apologies for posting the report on the TDWG meeting in Frankfurt only after a 2 weeks delay. I did not manage to finish the report earlier, because I had too much work preparing for an annual project meeting last Friday. Naturally, I remember better what I have said than what others said, although I made a real effort to adequately take notes of other peoples contributions. Any additions or corrections from those present at the discussion in Frankfurt are most welcome! I will then repost a corrected report, which will also go to the TDWG newsletter. Hoping for your additions and corrections! Gregor ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

TDWG Frankfurt: SDD subgroup meeting report
by Gregor Hagedorn 27 Nov '00

27 Nov '00

TDWG subgroup: Structure of Descriptive Data, subgroup session at the TDWG meeting in Frankfurt, 12. Nov. 2000 ## Version 1 ## == Participants == Stan Blum, Jim Croft, Gregor Hagedorn, Nicholas Lander, Bob Morris, Jörg Ochsmann, Richard Pankhurst, Jean-Marc Vanel, Mark Watson, Greg Whitbread, and others == Discussion topic: What is this group about? == Bob Morris: Group is not interested in resource development. Nicholas Lander: Not on Standard Characters lists. There was a TDWG subgroup on Standard Characters for plants (Richard Pankhurst was convener), which did not succeed to draw up a standard character list. Richard Pankhurst thinks we should put it off for further 5 years until Delta-like technology is widely used and interest is sufficient. == Discussion topic: SDD future and presentation == Convener: Gregor Hagedorn apologized for his lack of initiatives during the year, being overwhelmed with work in the starting phase of the GLOPP project. He is willing to continue as convener, but also willing to step aside if somebody else wants to fill the position and bring more initiatives into the group. It was agreed that there should be a SDD website to point to information resources concerning the activities of the group. Nicholas Lander will ask Ben Richardson at Perth, alternatively if Gregor Hagedorn continues as convener he can host the site on the www.DiversityCampus.net server of the GLOPP project. Most participants agreed that a true SDD workshop should be organized, lasting at least full 2 days, to clarify the issues brought up during the email discussions and reach real understanding. It was agreed that the email discussions are extremely good and useful, but that it is difficult to fully assess where a consensus was reached, and where the discussion simply tapered off. Also, certain concepts are easier discussed with physical presence, using graphical presentations. Such a workshop was considered to be desirable in the near future, latest immediately before or after the next TDWG meeting. It should be organized in conjunction with some other event, to minimize traveling expenses in terms of time and money. Another time could be before or after a workshop which is planned by the TDWG subgroup on Accession Data in the spring of 2001. The more technical discussions regarding implementation choices in XML should be clearly separated on the discussion list by the string XML. Perhaps a sub-subgroup might be formed to resolve questions in this area? == Discussion topic: Resource discovery == Bob Morris: We need to distinguish between questions that hinge on resource cost, and questions that hinge on biological problems. Resource cost: does it pay to try a given information source whether it holds information. It should be easy to know in advance whether a server may hold the information I am looking for. New important development reported by Bob Morris: Uniform data description interface (UDDI). Nothing about resource cost has been discussed so far, but group may address this issue. Some information about this should be in the header of a new XML files standard. It was agreed that this question is secondary to the interest Information that should be present in the header of the files: Taxonomic scope: What are the taxa I hold data on, either list, or query definition (query -> sorry I hold no data) Do I have descriptions? (e.g. unparsed natural language) Do I have structured descriptions? Which kind? e.g. DELTA/NEXUS/XML? or: If so, point me to where the structure of description is defined Do I have media (images)? Do I have keys or other means of identification? Do I have interactive keys? == Discussion topic: Who is interested in a new standard for descriptive data? == We distinguished between 3 user/provider interests: 1) The pure user of electronic field guides, applying the descriptions e.g. to identification work, 2) The pure data provider, who holds large amounts of descriptive information that is historical and has the status of an unchanging publication 3) The data developing scientist, who creates and analyses/uses his or her data. In contrast to case 2, any data here must be referenced to individual sources and any information is under contention of being false. The case of 2 occurs e.g. in projects where legacy information like huge conventional flora or fauna works are digitized and shall be accessed in a structured way. In cases 1 and 2 the data can be transformed in ways that may lead to some loss of information (e.g. concerning the source of individual assertions: "leaves are glandulous when young: observed by Author1, 1980"). However, the case 3 is interested in rigid structure to allow knowledge management and data validation. The case 3 is the case assumed in programs like the CSIRO Delta programs, Pankey/Pandora, or DeltaAccess which see themselves as working tools for scientists. == Discussion topic: XML markup as proposed by Kevin Thiele == Bob Morris raised the question whether Kevin Thieles proposal would be enough to start with, to which additional structural information can be added. Greg Whitbread thinks yes, but character types should be added. Gregor Hagedorn remarked that, without having objections against Kevins principal argument about the advantages of providing a very simple system that could be used to markup existing information, it was difficult to say whether the system would be compatible with a more structured approach, as long as the structured approach is yet very vague. However, it may be wise to take Kevins approach ahead, risking however to redefine it later. It was agreed that it is desirable to have a common standard which defines levels of structure. Gregor Hagedorn remarked that a software may require at least a certain structural level, e.g. a structured database may require coded markup of character and feature, referring to a full character schema. Levels should be clearly labeled, so that an application can easily detect whether there is sufficient structure for its needs. However, levels should be known not only to the software, but also to users so that scientists can communicate I can give you this, is this ok for you? Problems like those with versions of the DELTA standard (format changes, but version not recognizable for importing software), or in the graphics area the TIF-problem (the standard defines an envelope, which may contain any kind of information, including proprietary formats unreadable for other software) should be avoided. Possible levels could be: >> Level 0: The description is marked up as a block referring to a certain taxon. No markup of structures, methodology, or features. >> Level 1: Level 0 plus markup (not necessarily complete) of structures (leaf, flower, ) or method (naked eye, hand lens, light microscope, scanning electron microscope) >> Level 2: Level 1 plus markup of characters (i.e. structure/methodology/feature), but not character states >> Level 3: Level 2 plus markup of states >> Level 4: Level 3, fully coded markup referring to separate character definition/schema Are more levels needed? More orthogonal scheme, with complete/incomplete markup noted separately? Gregor Hagedorn brought up the question whether the simpler, character schema-free forms of XML markup are able to cope with queries and reports in multiple languages. It seems to him that the words of the language stand for only the English understanding, without any definition being available elsewhere. This is a contrast to the DELTA method of defining characters and states, and using codes that can easily be output in multiple languages concurrently. Nicholas Lander: There is a file format standard, the "Star file" format (20 yrs old but dynamic CODATA standard, now also in xml) in chemistry supplies means to define core character lists with supplements == Discussion topic: Discussion of future of DELTA format == Nicholas Lander: We need more rigorous system. It was agreed to put efforts into the original idea of developing a system that goes beyond DELTA and Nexus, but encompassed the functionality of NEXUS, DELTA/New DELTA, and adds the additional requirements identified by LucID or DeltaAccess. Richard Pankhurst warned about the generalized system fallacy: Any system that tries to fulfill too many requirements will become very complex and inherently difficult to analyze and maintain. There is a danger of creating a monster-structure nobody will actually use. @@ Question to participants: something was discussed about the Free Delta system, but I missed that in my note. Anybody can fill in here? == Discussion topic: Standard character lists == Several members stressed the need for Standard character lists in the future. It was (who said that??) proposed to start with core schemata that can be expanded as time goes on. Gregor Hagedorn proposed that standards should not necessarily be seen as concentric rings (e.g. a single standard with successive versions or levels), but perhaps rather modular blocks that can stand side-by-side. For example, it may be wise to develop and maintain standards for different methodologies (field observation, light microscope, SEM characters, chemical compounds) by different standard bodies. A given description could choose from the standard modules as necessary for the observations or studies made. Gregor Hagedorn: Standard character schema should be developed like scientific publications, so that they can be developed and improved by scientists in the course of several years. Only after the contour of competing schemata have become clear, standardization efforts should begin. == Discussion topic: Character vs. Structure / feature == The following discussion happened after a break necessary due to the collision of Bob Morris presentation with the discussion. Many participants were absent during this part. Jean-Marc Vanel presented his views about structural analysis of character data. It was first that it is preferable to use the general term structure rather than the term organ proposed by Jean- Marc Vanel. Vanel proposed that structures can be hierarchical or primary/secondary. We found, however, that structures have not necessarily a clear hierarchy. If the same type of hairs exist on both the stem and the leaf, it is not sufficient to place hairs outside the group containing both stem and leaf. It is possible that characters and structures have overlapping hierarchies, that can not be resolved into a simple tree. For example, the same hairs may occur on many different structures, which can not be grouped hierarchically. A Ref-ID mechanism is necessary to document the relations between structures, substructures, and properties. Vanel: XML is like a Christmas tree: basic tree with decoration connecting branches. Richard Pankhurst uses relational adjectives (= linguistic term): part-of relationship and kind-of relationship. Examples: Leaflet is a part of a leaf. Basal leaf is a kind of leaf; glume is a kind of leaf, but in more special sense. Features have qualifiers. Richard Pankhurst further discussed restrictions of context: plant is young/old, ivy leaves: when young lobed, when old almost round. Certain conditions are called epitopic child birth is possible only in female, and only in female that is pregnant and of appropriate age. Gregor Hagedorn stressed the importance of methods or methodology in addition to the structure/basic property analysis performed by Diederich et al. The same character may have different states (or values/results) for different methodologies (e.g. surface rough in SEM, but smooth with hand lens). Methodology can further be split into observation method (type of apparatus used) and condition environmental or experimental method (soil, climate, culture media, substrate, etc.). In some cases a character is implicitly only possible to observe using a certain methodology. ---------------------------------------------------------- Gregor Hagedorn (G.Hagedorn(a)bba.de) Institute for Plant Virology, Microbiology, and Biosafety Federal Research Center for Agriculture and Forestry (BBA) Koenigin-Luise-Str. 19 Tel: +49-30-8304-2220 14195 Berlin, Germany Fax: +49-30-8304-2203 Often wrong but never in doubt!

1 0

Re: Report; correction
by Robert A. (Bob) Morris 27 Nov '00

27 Nov '00

Although I am the perpetrator of the mistake, UDDI does not stand for Universial Data Description Interface, but rather Universial Description, Discovery and Integration. See www.uddi.org. UDDI is not about describing data, but about how you can find out where and how to invoke services, e.g. databases, on the web. Designed originally for Business-to-Business service discovery, we are looking at it as a registry facility for biodiversity data sources. Bob

1 0

Report, please
by Kevin Thiele 21 Nov '00

21 Nov '00

Will someone who went to Frankfurt please report on the meeting and discussions re the SDD - Bob or Jim? - at your leisure

1 0