March 2001 - tdwg-content - lists.tdwg.org

Re: XML gateways
by Rob Stevenson 25 Mar '01

25 Mar '01

Dave, I tried looking at http://tsadev.speciesanalyst.net/DarwinCore/Darwin_core.asp but I was unsuccessful. Is the address wrong? Must I be registered? Rob Stevenson but I was unsuccessful Dave Vieglais wrote: > >-----Original Message----- > >From: TDWG - Structure of Descriptive Data [mailto:TDWG-SDD@USOBI.ORG]On > >Behalf Of Robert A. (Bob) Morris > >Sent: Tuesday, March 20, 2001 10:07 AM > >To: TDWG-SDD(a)USOBI.ORG > >Subject: Re: XML gateways > > > > > >Jim Croft writes: > > > Date: Wed, 21 Mar 2001 00:25:09 +1100 > > > From: Jim Croft <jrc(a)anbg.gov.au> > > > To: TDWG-SDD(a)usobi.org > > > Subject: Re: XML gateways > > > > ... > > > > >Sad, but true. We should kick around other ideas to solve this > >problem, though a conventional port may be the easiest for data > >sources to implement. How about: > > > >- A convention whereby if > > http://<host>/<filePath>/<cgiquery> > >yields HTML then > > http://<host>/<filePath>/xml/<cgiquery> > >yields XML > > > >-A convention whereby people set up virtual hosts---pretty easy in > >most web servers---so that if > > http://<host>/<filePath>/<cgiquery> > >yields HTML then something like > > http://xml-<host>/<filePath>/<cgiquery> > >yields XML > > > >Hopefully all of this is temporary, since a well crafted GBIF should > >provide for discovery of URL, query syntax, and return schema. My real > >point is that current versions of Oracle, SQL-Server, FileMaker, and > >Access(?) can already emit XML without much(?) effort on the part of > >the data source operators, and doing so would let people proceed to > >build interesting distributed applications. > > There are initiatives to resolve each of the issues brought up here that are > being worked on in several industry sectors: > 1. Service discovery is addressed by UDDI (Universal Description, > Description and Integration). > 2. Service description by WSDL (Web Service Description Language) which is > an XML document format for describing methods and data exposed by a web > services. > 3. IGIR (Interface for Generic Information Retrieval) provides a generic > format for information retrieval via HTTP GET or POST requests that is easy > to implement. The queries are sent as a structure easily translated into a > number of common query syntaxes (SQL for example). Results are contained > within a consistently formatted document. IGIR is optionally stateful (it's > up to the server to decide)- so a query that generates a very large result > set need not be sent back to the client all in one hit. IGIR also supports > a SOAP interface (that's what it is originally designed for), so > programmatic access to information resources becomes pretty simple. > > A search and retrieval scenario would work something like this: > A client hits on the UDDI directory, looking for services that support IGIR > and are "biologically relevant". A list of services supporting IGIR are > returned, and the client broadcasts a query to them and waits for results to > start coming back. Since the records are all formatted in a similar manner, > it is a simple process for the client to merge the results and present to > the user a nicely formatted set of results. > > Anyway, that's how things should work. To support a broadcast query, > obviously the targets must understand the contex of the query terms- so > groups within a particular domain (such as biological collections for > example) will need a set of "well known" access points (searchable fields). > Similarly, to simplify the merging of results that come back from such a > query, it would be convenient if there was an agreed record structure that > was shared between data providers. The xml output by most database vendors > is a hack designed to allow easy transport of database records using http. > Things are getting better with xsd and so forth, but the structure of the > resulting xml documents will still reflect the structure of the databases > rather than any agreed upon common format. > > Which brings us back to the "biological collections profile" concept that > was raised at the last TDWG meeting. Would anybody like to volunteer a set > of access points (searchable fields) that are relevant for biological > collections? Similarly for the record structure? The DarwinCore elements > start to address these issues, but are somewhat deficient in several > respects. They might make a reasonable starting place though- The > description with a comment section (add your gripes etc) is available at: > http://tsadev.speciesanalyst.net/DarwinCore/Darwin_core.asp > > Most of you know that the DarwinCore was originally developed as a Z39.50 > profile, hence some of the language may seem a little odd. But the main > thing to take a look at is the list of access points- which are useful from > a biological collections point of view, and what's missing? > > There are obviously other domains of biological databases beside just > collection databases. Any suggestions for commonality between them all? > > Cheers, > Dave V. > > ========================= > David A. Vieglais > University of Kansas > Natural History Museum & > Biodiversity Research Center > http://www.nhm.ukans.edu

1 0

Re: XML gateways
by Jim Croft 20 Mar '01

20 Mar '01

>Virtually all major relational dbms now output XML, if crude, and >there is not really much reason people shouldn't be doing so on some >agreed upon alternative http port, say 8883. I would advise against this... many institutional firewalls prohibit all packets except through port 80... this would effectively prevent all of our staff from playing with the outside world... bummer, but them's the rules... > It's not critical to wait >for an agreed upon uniform schema to serve XML. it's even possible to >infer a schema or DTD from data instances, and anyway a schema or DTD >is not necessary to do much of what XML is good for. but it is nice to be able to look at them to see what people think they are doing... :) jim

1 0

Re: XML gateways
by Jones, Tim (CMR, Hobart) 20 Mar '01

20 Mar '01

Nice work!! I am wondering though, if you are federating heterogenous XML databases why do you need to scrape the returned HTML but instead operate on the XML or are they not XML enabled? Tim Jones --------------------------------------------------------- Database Manager Centre for Research on Introduced Marine Pests GPO Box 1538, Hobart Tas 7000, Australia. Phone : (03) 62325222 (switch), (03) 62325213 (direct) Mobile: 0411 560057 Fax : (03) 62325485 E-mail: tim.jones(a)marine.csiro.au --------------------------------------------------------- > -----Original Message----- > From: Robert A. (Bob) Morris [mailto:ram@CS.UMB.EDU] > Sent: Tuesday, 20 March 2001 15:18 > To: TDWG-SDD(a)USOBI.ORG > Subject: XML gateways > > > Here's some candy: we've built simple sample gateways to the USDA > Plants site and the Simthsonian Mammal Species of the World > site. Server-side perl makes up cgi requests to the site, parses > the returned html and emits XML. Of course, it is a very > unstable solution since > changes to the site's html will break it. But we did it because we > experiment with federating heterogeneous XML biodiversity databases. > > You can nibble at > http://www.cs.umb.edu/efg/xml/junwan/DEVELOP/gateway/gateway2.html > > > Bob Morris > Jun Wan >

1 0

Re: XML gateways
by Dave Vieglais 20 Mar '01

20 Mar '01

>-----Original Message----- >From: TDWG - Structure of Descriptive Data [mailto:TDWG-SDD@USOBI.ORG]On >Behalf Of Robert A. (Bob) Morris >Sent: Tuesday, March 20, 2001 10:07 AM >To: TDWG-SDD(a)USOBI.ORG >Subject: Re: XML gateways > > >Jim Croft writes: > > Date: Wed, 21 Mar 2001 00:25:09 +1100 > > From: Jim Croft <jrc(a)anbg.gov.au> > > To: TDWG-SDD(a)usobi.org > > Subject: Re: XML gateways > > ... > >Sad, but true. We should kick around other ideas to solve this >problem, though a conventional port may be the easiest for data >sources to implement. How about: > >- A convention whereby if > http://<host>/<filePath>/<cgiquery> >yields HTML then > http://<host>/<filePath>/xml/<cgiquery> >yields XML > >-A convention whereby people set up virtual hosts---pretty easy in >most web servers---so that if > http://<host>/<filePath>/<cgiquery> >yields HTML then something like > http://xml-<host>/<filePath>/<cgiquery> >yields XML > >Hopefully all of this is temporary, since a well crafted GBIF should >provide for discovery of URL, query syntax, and return schema. My real >point is that current versions of Oracle, SQL-Server, FileMaker, and >Access(?) can already emit XML without much(?) effort on the part of >the data source operators, and doing so would let people proceed to >build interesting distributed applications. There are initiatives to resolve each of the issues brought up here that are being worked on in several industry sectors: 1. Service discovery is addressed by UDDI (Universal Description, Description and Integration). 2. Service description by WSDL (Web Service Description Language) which is an XML document format for describing methods and data exposed by a web services. 3. IGIR (Interface for Generic Information Retrieval) provides a generic format for information retrieval via HTTP GET or POST requests that is easy to implement. The queries are sent as a structure easily translated into a number of common query syntaxes (SQL for example). Results are contained within a consistently formatted document. IGIR is optionally stateful (it's up to the server to decide)- so a query that generates a very large result set need not be sent back to the client all in one hit. IGIR also supports a SOAP interface (that's what it is originally designed for), so programmatic access to information resources becomes pretty simple. A search and retrieval scenario would work something like this: A client hits on the UDDI directory, looking for services that support IGIR and are "biologically relevant". A list of services supporting IGIR are returned, and the client broadcasts a query to them and waits for results to start coming back. Since the records are all formatted in a similar manner, it is a simple process for the client to merge the results and present to the user a nicely formatted set of results. Anyway, that's how things should work. To support a broadcast query, obviously the targets must understand the contex of the query terms- so groups within a particular domain (such as biological collections for example) will need a set of "well known" access points (searchable fields). Similarly, to simplify the merging of results that come back from such a query, it would be convenient if there was an agreed record structure that was shared between data providers. The xml output by most database vendors is a hack designed to allow easy transport of database records using http. Things are getting better with xsd and so forth, but the structure of the resulting xml documents will still reflect the structure of the databases rather than any agreed upon common format. Which brings us back to the "biological collections profile" concept that was raised at the last TDWG meeting. Would anybody like to volunteer a set of access points (searchable fields) that are relevant for biological collections? Similarly for the record structure? The DarwinCore elements start to address these issues, but are somewhat deficient in several respects. They might make a reasonable starting place though- The description with a comment section (add your gripes etc) is available at: http://tsadev.speciesanalyst.net/DarwinCore/Darwin_core.asp Most of you know that the DarwinCore was originally developed as a Z39.50 profile, hence some of the language may seem a little odd. But the main thing to take a look at is the list of access points- which are useful from a biological collections point of view, and what's missing? There are obviously other domains of biological databases beside just collection databases. Any suggestions for commonality between them all? Cheers, Dave V. ========================= David A. Vieglais University of Kansas Natural History Museum & Biodiversity Research Center http://www.nhm.ukans.edu

1 0

Re: XML gateways
by Robert A. (Bob) Morris 20 Mar '01

20 Mar '01

Jim Croft writes: > Date: Wed, 21 Mar 2001 00:25:09 +1100 > From: Jim Croft <jrc(a)anbg.gov.au> > To: TDWG-SDD(a)usobi.org > Subject: Re: XML gateways > > >Virtually all major relational dbms now output XML, if crude, and > >there is not really much reason people shouldn't be doing so on some > >agreed upon alternative http port, say 8883. > > I would advise against this... many institutional firewalls prohibit all > packets except through port 80... this would effectively prevent all of our > staff from playing with the outside world... bummer, but them's the rules... Sad, but true. We should kick around other ideas to solve this problem, though a conventional port may be the easiest for data sources to implement. How about: - A convention whereby if http://<host>/<filePath>/<cgiquery> yields HTML then http://<host>/<filePath>/xml/<cgiquery> yields XML -A convention whereby people set up virtual hosts---pretty easy in most web servers---so that if http://<host>/<filePath>/<cgiquery> yields HTML then something like http://xml-<host>/<filePath>/<cgiquery> yields XML Hopefully all of this is temporary, since a well crafted GBIF should provide for discovery of URL, query syntax, and return schema. My real point is that current versions of Oracle, SQL-Server, FileMaker, and Access(?) can already emit XML without much(?) effort on the part of the data source operators, and doing so would let people proceed to build interesting distributed applications. >...

1 0

Re: XML gateways
by Robert A. (Bob) Morris 19 Mar '01

19 Mar '01

They are not XML enabled. We wish they were since both are pretty interesting databases. Virtually all major relational dbms now output XML, if crude, and there is not really much reason people shouldn't be doing so on some agreed upon alternative http port, say 8883. It's not critical to wait for an agreed upon uniform schema to serve XML. it's even possible to infer a schema or DTD from data instances, and anyway a schema or DTD is not necessary to do much of what XML is good for. We hope people will pay more attention to getting out their "raw" data in XML and publishing their /query/ interfaces so that applications can use them. When we do something like our gateway, we have to put a lot of time into reverse engineering the query syntax because so many sites believe that the only use of their data is through the forms they provide, or some import of their entire dataset by a prior arrangement. Bob Jones, Tim (CMR, Hobart) writes: > Date: Tue, 20 Mar 2001 15:29:55 +1100 > From: "Jones, Tim (CMR, Hobart)" <tim.jones(a)MARINE.CSIRO.AU> > To: TDWG-SDD(a)usobi.org > Subject: Re: XML gateways > > Nice work!! > > I am wondering though, if you are federating heterogenous XML databases why > do you need to scrape the returned HTML but instead operate on the XML or > are they not XML enabled? > > Tim Jones > > --------------------------------------------------------- > Database Manager > Centre for Research on Introduced Marine Pests > GPO Box 1538, Hobart Tas 7000, Australia. > Phone : (03) 62325222 (switch), (03) 62325213 (direct) > Mobile: 0411 560057 > Fax : (03) 62325485 > E-mail: tim.jones(a)marine.csiro.au > --------------------------------------------------------- > > > > > > -----Original Message----- > > From: Robert A. (Bob) Morris [mailto:ram@CS.UMB.EDU] > > Sent: Tuesday, 20 March 2001 15:18 > > To: TDWG-SDD(a)USOBI.ORG > > Subject: XML gateways > > > > > > Here's some candy: we've built simple sample gateways to the USDA > > Plants site and the Simthsonian Mammal Species of the World > > site. Server-side perl makes up cgi requests to the site, parses > > the returned html and emits XML. Of course, it is a very > > unstable solution since > > changes to the site's html will break it. But we did it because we > > experiment with federating heterogeneous XML biodiversity databases. > > > > You can nibble at > > http://www.cs.umb.edu/efg/xml/junwan/DEVELOP/gateway/gateway2.html > > > > > > Bob Morris > > Jun Wan > > >

1 0

XML gateways
by Robert A. (Bob) Morris 19 Mar '01

19 Mar '01

Here's some candy: we've built simple sample gateways to the USDA Plants site and the Simthsonian Mammal Species of the World site. Server-side perl makes up cgi requests to the site, parses the returned html and emits XML. Of course, it is a very unstable solution since changes to the site's html will break it. But we did it because we experiment with federating heterogeneous XML biodiversity databases. You can nibble at http://www.cs.umb.edu/efg/xml/junwan/DEVELOP/gateway/gateway2.html Bob Morris Jun Wan

1 0