[tdwg-tapir] Conversation with people from the Geoserver software.
Hi all, I do not know if I should post this here but I felt this may be of interest of everybody working in the protocol. I was doing some questions about the configuration of Geoserver to provide a WMS and WFS service that I am creating, I think you all know about it. The mapping configuration between the database and the GML for the WFS service inside Geoserver is quite limited, but there are some people working on ways to improve it. I introduced a little bit what we are doing and they answer all this... I hope it is of your interest... if anybody want to join the discussion go and subscribe to geoserver-devel@lists.sourceforge.net Javi. Begin forwarded message:
From: Chris Holmes <cholmes@openplans.org> Date: 23 de noviembre de 2005 20:55:29 GMT+01:00 To: Javier de la Torre <jatorre@gmail.com> Cc: geoserver-devel@lists.sourceforge.net, rob@socialchange.net.au Subject: Re: [Geoserver-devel] Re: [Geoserver-users] schema.xml file
Quoting Javier de la Torre <jatorre@gmail.com>:
Hi Chris,
Interesting discussion!
In our mapping language (we call it CMF, concept mapping file) comparable to schema.xml file we allow complicate mappings with filters, conditional mappings and things like that. I suppose is pretty much the same thing you are thinking. Interesting. It could be interesting to try to sync up the work we're both doing. I don't like everyone reinventing the wheel for mapping files, but I don't know of anyone else really doing them. Perhaps we could try to use the same file format between our projects. The 'filters' that we make use of our primarily OGC Filters - their representation of a where statement.
Short (rough) overview here: http://udig.refractions.net/confluence/display/GOWS/Querying+data +using+%27Filter%27 and full spec is here: http://portal.opengeospatial.org/files/?artifact_id=8340 (pdf)
Well, Ok, let's go in parts. What we are working is in an HTTP-XML protocol to allow dynamic queries to heterogeneous databases and get results in XML compliant to whatever XML schema they have mapped their databases... (puff this may not be the best description of it...). In our community (TDWG, Taxonomic Database Working Group, a very little OGC like standardization group) there are two protocols for doing that, Digir and bioCASe, and now we are merging them into a new one called TAPIR. There will be two implementations of this protocol. The protocol, very much like the WFS protocol, is still under discussion in a wiki page http://ww3.bgbm.org/protocolwiki/FrontPage.
Actually the WFS protocol was reviewed when creating this new protocol because it was considered a possible solution to our problems. If you take a look at our capability operation is very likely to WFS one, and the filtering in a query is also very similar to the one you pointed me (because we also get inspired by it!).
For this protocol we are creating there will be two reference implementations, pywrapper in python, and Digir 2 in Java and RDF based. Interesting. Looking at the schemas it actually looks a bit closer to catalogue than WFS: http://portal.opengeospatial.org/files/?artifact_id=5929&version=2 Unfortunately it's not as solid of a spec, and there aren't any good open source reference implementations.
I wonder if it'd be possible to get some of your specs to line up more with OGC ones, and vice-versa. Because it'd be really nice to be able to share code and implementations. Like if it was possible to do just a few extensions to WFS for your needs, then you could just contribute to GeoServer, and the core would be stronger for all. And people could use the same parsing tools on both GML and your schemas, and the same clients to communicate back and forth. I'm not sure how practical any of this actually is, but I _hate_ reduplication of work, and it just seems at the core we're doing a lot of the same stuff.
The Catalogue spec has a good bit on a common query language, if you were to support that, and the same XML filters for example, then we could share code with GeoTools.
We already had an implementation of them in GeoTools, since we're
OGC
focused, so they made logical sense to do the mappings (correct me if I'm wrong here Gabriel). A nice thing about them is that we already have most of the code written to 'unpack' them, that is if the back end data format can perform the filter, then we turn it into the SQL (or whatever query language it might use). So we should be able to get good performance out of the mappings, since where possible it's passed to the backend. But the nice thing is that we _can_ do it on anything, since there's Java code for it all, so Shapefile supports all filters as well.
Uhm... I do not understand this totally... You mean that implementing a more intelligent mapping will let you create dynamically SQL statements that you can use for filtering based on the query filter? I feel I may be mixing everything...
In our case we describe the database structure and we map the database to the different schemas that we want to share the data in. When someone do a query the wrapper read this configuration file and figure out the where clause to filter the data in the SQL statement. Then depending on the xml schema that is requested as a response it selects the necessary information from the database to populate an XML instance of this XML schema. Yeah, we do a similar thing, I guess the point of difference is that we're not always dealing with databases, so we're not always able to turn the filter into SQL, so we do so in Java code.
Well, in reality it is a little bit more complicate to give the mapping of the database-schema the necessary information to create correct xml documents... Unfortunately an XML schema can validate many different XML documents that looks completely different...
I am working also for a European project called Synthesys and I was planning to get
in
contact with the INSPIRE project in the near soon when I get more OGC knowledge :) Excellent. If you want some contacts I know at least one good one.
Yes! that would be great... I feel is a shame I am not in contact with them when I am aiming at providing GIS services in a european level...
Ok, I'll let you know.
And the thing I don't quite understand, how does the geographic stuff relate to the work you're doing on the spec? Like are you planning on geographic stuff as part of it? Or would you set up a database with both a TAPIR interface and a WFS interface?
What you are planning to do with complex mappings looks quite
similar
to the problem that we are trying to solve too. As I described we have a wrapper tool to make possible to connect heterogenous databases to a network where we can query them using a standard protocol (TAPIR and BioCASe before), the results from the providers come in XML following a certain "conceptual schema" (what you call application schema). Maybe we should have a talk and I will show you how are we doing things and you may get some ideas... Yeah, that could be good. I don't know how far in to things you are, but note that a WFS doesn't _have_ to do geometric objects. Like it could theoretically not return any GML at all, just XML. I don't know if that's the route to go, but it might weirdly be worthwhile to just sync your effort up with GeoTools/GeoServer, since we would both benefit from more users and use cases. But yeah, at the very least it sounds like we're working on very similar problems, and can learn what works and what doesn't, even if it doesn't make sense to share at the coding level.
As I said before when we were discussing our protocol we took a look at available protocols and one of those was WFS. I was not resposible for that and I do not know the exact reason why it was discard (it should be in the protocol review uhmm). But... is it possible with WFS to do paging? I mean... in our case someone retrieve ten "features" and then want to retrieve the next 10...
No WFS does not currently do paging, but I actually don't think that not currently meeting all your capabilities is a good enough reason to reject a spec. Because it wouldn't be very hard to add a 'paging' capability to a WFS. I mean, make a schema that extends WFS a bit, for paging, make a reference implementation, and make it your official thing, and could even try to get it rolled back into the OGC specs. I quote I quite like about standards is this:
'This tinkering with the [standardized] protocol, however, is not an empirical fact showing the limits of standardization in practice. We do not point at these instances in order to demonstrate the ‘resistance’ of actors to domination. Rather, we argue that the ongoing subordination and (re)articulation of the [standardized] protocol to meet the primary goals of the actors involved is a sine qua non for the functioning of the [standardized] protocol in the first place.' Timmermans and Berg (1997: p. 291.)
The point is you _should_ take a standard and improve it to your needs, the point of standards is a basis to meet further needs. And if you start with a standard, it's much, much easier to build upon lots of other work that's gone before. Like if it's super similar to WFS, then there could easily be both open source and commercial implementations of your spec made.
I think Catalogue actually does have a concept of paging. It's concepts could probably be applied to WFS. At the GeoTools level Jody is doing some work now for a Random Access API, which could make paging quite easy.
But it looks like you're already too far along to really switch paths, and I'm not sure if the OGC specs would hold up to all your needs, even expanding them where appropriate.
At the end seems everybody is working on the same problem, the mapping of relational databases to XML schemas... Yeah, there's a lot out there, and a lot of different approaches, and it feels silly to roll so much of your own. But many are lacking in different ways. We're hoping with our approach to eventually be able to sync up with other more generic things. We looked into using Hibernate to do the object-relational mapping, but Rob needed to be able to control the SQL select statements, so you could derive your mapping from the results of a select statement, which hibernate just doesn't let you. And we wrote a straight SAX producer to stream out GML, since at the time there were no good, fast object -> XML producers that allowed you real flexibility, since we have to deal with arbitrary Features, of arbitrary complexity, can't define the schema beforehand. There are solutions now, and I think our next generation parser/producer makes of JAXB. But yeah, working with Geographic Information we hit a lot of the big problems early, since the datasets so quickly get huge.
Yes, in our case this is an issue... we have to be able to page trough datasets because even the fastest wrapper in the world will not be able to retrieve 1 million features full of properties at once... that's why we have paging. One could thing we are trying to implement, for example, is that the user is able to send the structure, the "complex_type", in an XML schema fragment specifying the result formats he want the data to be responded. There is a two step-mapping going here but I do not want to get long now...
That's really interesting, I could see that being very useful from a WFS/GML perspective as well:
In OGC there's this concept of a Feature Portrayl Service, that takes a WFS and an SLD, which is an XML language to specify how to 'style' a map, what the image should look like, and returns an image. It's similar to a WMS, but sort of fully virtual, since nothing really sits on the server.
It'd be very interesting to see such a thing for full GML, it would take a WFS and a Mapping file as arguments, and perform the mapping, returning the result as GML, a sort of virtual WFS. So if someone else put their data up as WFS, but didn't bother to get it into a well known schema, then you could just do it yourself.
When we get the Complex Feature stuff sorted out in GeoServer this type of thing shouldn't be all that hard. We already support WFS as a DataSource (in addition to a number of databases and file formats), and then we'd just need to define a mapping file.
It seems like it'd be _really_ nice to have a common spec for the 'mapping file' we're talking about, how you apply filters and whatnot to go from one format/schema to another. Rob, any thoughts about that? We're just kind of rolling our own this round, right? It could be a cool win to have clients be able to choose the schema they want to view data in, and just connect to a WFS supplying their chosen mapping.
The project web page of one of this implementation is at http:// ww3.bgbm.org/tapir and if you want to see a data provider in action you can go to http://ww3.bgbm.org/tapirwrapper/
Looks cool.
best regards,
Chris
Best regards,
Javier.
best regards,
Chris
Javier.
On 23/11/2005, at 8:23, Chris Holmes wrote:
Actually, Javier, you're getting into the bleeding edge of GeoServer, and you actually understand how WFS and GML _should_ work perfectly. Rob has been pushing these types of things for awhile, and just recently managed to get a lot of the ideas in GeoServer as
working
code with Gabriel. The schema.xml files were not about actually mapping your GML to different types, they were just kind of surface level ways of renaming things. The problem of actually getting names to map back and forth, to perform filters on them, and more, is much more complex.
The problem is the code is on a branch, like Gabriel just announced it last week, and there will still be bugs, and there's lots of documentation to write. But we would love to have you as a user testing it out. Indeed your intended schema is pretty simple compared to the complexity of what we're hoping to support.
See the 'proto-documentation' (really just a place holder) here:
http://docs.codehaus.org/display/GEOSDOC/Complex+Feature +Documentation
As for MySQL, if you're using the MySQL module you need to have
your
points stored as Geometries, see: http://dev.mysql.com/doc/refman/5.0/en/spatial-extensions-in- mysql.html (it's available in 4.1 as well). As for improvements from 4.1 to 5.0, I haven't tried out 5.0, but looking at the docs there appears to be nothing new, the support overall is pretty poor, I'd recommend upgrading to PostGIS asap.
If you really want to define your MySQL table as just x,y coordinates, Rob made a Geometryless module that will work, but we don't currently include it with GeoServer. I believe Gabriel upgraded it though in his latest work.
best regards,
Chris
Quoting Justin Deoliveira <jdeolive@unknown-road.com>:
Hi Javier,
So what you need is the wfs to serve up gml that conforms to your application schema. After you add your data store and feature type to geoserver (via the config gui), your feature type will contain attributes with the default types. You can customize the types with the feature type editor. There are some docs on how to do this here.
http://docs.codehaus.org/display/GEOSDOC/FeatureType+Config
Note the part about changing the type of an attribute to an xml fragment.
Unfortunatley I dont know much about mysql, and even less about its spatial extension. However I am sure someone else will be able to help with that. And questions about mysql could probably be best answered on the geotools devel and user lists. I would try repositing those questions there.
If you have any more questions dont hesitate to ask.
Justin
Javier de la Torre wrote: > Hi Justin, > > Thanks for your answer. > Some background. I am using MySQL right now as a DataStore (but also > planning to go for PostGIS). In my database I have a table where I have > records of specimens (plants) with information like their name, who > identified them, when they were collected, etc. Also I have in the same > table coordinates of the location where these plants where collected. > > My idea is to create a WFS service to provide the points where the > specimens where collected together with the other information. We have > some kind of standards about how all this information must be > described. The idea is to create a GML application schema where we > describe the specimens using our own "concepts" (like CatalogNumber and > things like that) and the location of the specimen using a > gml:PointPropertyType. > > If I let Geoserver create automatically the schema for me I get > something like this: > > ---------- > <xs:complexType name = "specimens_Type" > > <xs:complexContent> > <xs:extension base = "gml:AbstractFeatureType" > > <xs:sequence> > <xs:element type = "xs:string" minOccurs = "1" name = "SpecimenGUID" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:string" minOccurs = "0" name = "ProjectTitle" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:string" minOccurs = "0" name = "BiotopeText" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:dateTime" minOccurs = "0" name = > "CollectingISODate" nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:string" minOccurs = "0" name = "LocalityText" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:string" minOccurs = "0" name = "CountryName" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:string" minOccurs = "0" name = "Notes" nillable > = "true" maxOccurs = "1" /> > <xs:element type = "xs:int" minOccurs = "0" name = "AltitudeLowerValue" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "xs:string" minOccurs = "0" name = "coordinates" > nillable = "true" maxOccurs = "1" /> > </xs:sequence> > </xs:extension> > </xs:complexContent> > </xs:complexType> > ------------- > > But I would prefer to get something like: > > ----------- > <xs:complexType name = "specimens_Type" > > <xs:complexContent> > <xs:extension base = "gml:AbstractFeatureType" > > <xs:sequence> > <xs:element type = "my_ns:GUID" minOccurs = "1" name = "SpecimenGUID" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns:ProjectTitle" minOccurs = "0" name = > "ProjectTitle" nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns: BiotopeText" minOccurs = "0" name = > "BiotopeText" nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns:dateTime" minOccurs = "0" name = > "CollectingISODate" nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns:string" minOccurs = "0" name = "LocalityText" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns:string" minOccurs = "0" name = "CountryName" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns:string" minOccurs = "0" name = "Notes" > nillable = "true" maxOccurs = "1" /> > <xs:element type = "my_ns:int" minOccurs = "0" name = > "AltitudeLowerValue" nillable = "true" maxOccurs = "1" /> > <xs:element type = "gml:PointPropertyType" minOccurs = "0" name = > "coordinates" nillable = "true" maxOccurs = "1" /> > </xs:sequence> > </xs:extension> > </xs:complexContent> > </xs:complexType> > ---------------- > > And by the way... how do I have to define the coordinates in my MySQL > table so that Geoserver understand them as Points in gml? In the case > of PostGIS is this done automatically? > > A final question... I do not find any documentation about the spatial > capabilities of MySQL 5, do you know if there is any improvement from > what it was available already in 4.1? > > I am new to OGC standards so i am still a little bit confused on how to > create this GML application schemas so sorry if I am asking no- sense... > > Thanks. > > Javier. > > > On 22/11/2005, at 20:29, Justin Deoliveira wrote: > >> Hi Javier, >> >> I suspect that the GML writer ignores the information that you have >> supplied in the schema.xml file, and generates the gml from the >> actual features / feature types themselves. >> >> Since the DescribeFeatureType logic takes place in GeoServer itself, >> it can be smart and delegate to the schema.xml file. However the GML >> writing logic takes place in the geotools library, and it has no >> notion of this file. >> >> Can you tell us a bit more about how you are trying to customize your >> schema. Perhaps there is another way around this problem. >> >> Justin >> >> Javier de la Torre wrote: >> >>> Hi all, >>> I am starting to play with Geoserver and until now I have to say I >>> am impressed by how easy is to install and configure, >>> congratulations!! >>> But when trying to edit a schema.xml file to adapt the output of a >>> WFS service I get into problems. In the documentation it says: >>> "The last file is not always present, it is the schema.xml file. >>> This is a tricky file - if present it is used directly for a >>> DescribeFeatureType response, but configuration information is also >>> read from it. If it is absent then the DescribeFeatureType response >>> is automatically generated by GeoServer with the default >>> AttributeTypes of the DataStore (such as the columns of the table)." >>> But if I edit manually this file Geoserver does not take care of it >>> for producing the GML of the features, but it does for the >>> DescribeFeatureType.... >>> And the Graphical interface automatically overwrite to the default >>> one... >>> Do you know what is going on? >>> Thanks in advance. >>> Javier. >>> ------------------------------------------------------- >>> This SF.Net email is sponsored by the JBoss Inc. Get Certified Today >>> Register for a JBoss Training Course. Free Certification Exam >>> for All Training Attendees Through End of 2005. For more info visit: >>> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click >>> _______________________________________________ >>> Geoserver-users mailing list >>> Geoserver-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/geoserver-users >> >> >> >> -- >> Justin Deoliveira >> The Open Planning Project >> http://topp.openplans.org > > >
-- Justin Deoliveira The Open Planning Project http://topp.openplans.org
------------------------------------------------------- This SF.Net email is sponsored by the JBoss Inc. Get Certified Today Register for a JBoss Training Course. Free Certification Exam for All Training Attendees Through End of 2005. For more info visit: http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click _______________________________________________ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
---------------------------------------------------------- This mail sent through IMP: https://webmail.limegroup.com/
---------------------------------------------------------- This mail sent through IMP: https://webmail.limegroup.com/
------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
---------------------------------------------------------- This mail sent through IMP: https://webmail.limegroup.com/
participants (1)
-
Javier de la Torre