Re: [tdwg-tapir] Updates and plans
Dear all, I would like to discuss the optional bits of TAPIR capabilities again in the light of service interoperability, especially when dealing with TAPIR lite. Do we really need to have so many optional capabilities for TAPIR? I do firmly think this is causing more problems than benefits and would prefer more mandatory capabilities. In particular I think TAPIR model support should be the lowest common denominator and not only templates as it is currently for TAPIR lite.
Do the following capabilities really need to optional?
- the inventory and search operation is optional. Is there a need for services not supporting them? - any need for inventory operations not to support anyConcept? - search operation should mandate at least 1 output model support - cant concept, literal and parameter expressions in filters be mandatory? - likewise cant logical operators and equals comparisons be mandatory filter parts? what about greaterThan and lessThan?
regards, Markus
On Oct 6, 2008, at 16:14, Markus Döring (GBIF) wrote:
Renato, all proposals sound valuable and good to me. Working on my new TAPIR implementation I would like to propose another change. What do you think of removing the TAPIR lite option altogether and mandate tapir model support via KVP as the lowest option for compliant TAPIR services? I would at least have a discussion around this subject, as it effects TAPIR interoperability in networks a lot. And I think you didn't feel too comfortable when I said GBIF will implement TAPIRlite only. I am thinking of implementing model based TAPIR now because I see too many interoperability problems.
what do you think? can we put this on the agenda for discussion at least? Markus
On Oct 6, 2008, at 15:14, Renato De Giovanni wrote:
Dear all,
As mentioned in the last message, I created a new page in the Wiki to document possible changes in the protocol:
http://wiki.tdwg.org/twiki/bin/view/TAPIR/PossibleChanges
Please feel free to add your comments or suggest other changes. Use the Wiki or the mailing list.
If we decide to make any changes, they will be the last ones before submitting TAPIR to the TDWG standards track. I would also like to suggest October 31st as the deadline to decide about this.
Best Regards,
Renato
Dear all,
I finally managed to update the TAPIR specification to reflect the last issues discussed in this mailing list:
http://www.tdwg.org/dav/subgroups/tapir/1.0/docs/TAPIRSpecification_2008-09-...
I also updated the network builders' guide to include references to the new resources (TapirTester and TapirBuilder) and to recommend the new CNS encoding in XML instead of the old text file:
http://www.tdwg.org/dav/subgroups/tapir/1.0/docs/TAPIRNetworkBuildersGuide_2...
The official links are already redirecting to these documents.
I think it's time to consider submitting TAPIR to the TDWG standards track, but my feeling is that we should first wait for the annual meeting since we're close to it anyway. There's no subgroup meeting planned for TAPIR this year, but the TDWG annual meeting is always an opportunity for most people here to interact and to see if there are any specific improvements we should still make in the protocol.
I've been taking notes of additional changes that we could possibly make before finishing version 1.0. I'll try to organize this in the Wiki during the next days and I'll give you the link. We can then revise the list of possible changes and decide together what we should do.
Please let me know if you have any ideas about this. I think we should definitely finish TAPIR 1.0 this year and try to make it a TDWG standard.
Best Regards,
Renato
tdwg-tapir mailing list tdwg-tapir@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tapir
Hi Markus,
Sorry about the delay - I've just returned from a workshop.
If I understood your suggestion correctly, all TAPIR providers would need to parse filters, understand search requests with the model parameter, and support inventories in any concepts. This means that query templates would not be necessary anymore, right?
I think it's quite a big change at this point and it will impact existing provider implementations that follow the current TAPIRLite approach.
If there's any serious interoperability problem now, we need to investigate it carefully before making this kind of change. I understand that raising the minimum service profile level can in principle make things easier for clients. But if this is going to make things too difficult for new provider implementations, one of the possible consequences is to see other protocols or even unofficial variations of TAPIR being used, which can be worse than having a single protocol with different service profiles.
What I do think we're missing is guidelines for different types of providers and networks. Simple documents, 1 or 2 pages, with clear recommendations such as what conceptual schemas, output models and query templates to use.
Some comments about your specific suggestions:
the inventory and search operation is optional. Is there a need for
services not supporting them?
We were probably too flexible when we made them optional. However, now that we're considering a change that will allow providers to advertise dump files, I'm just thinking if this can be acceptable: "here's my metadata, here's the conceptual schemas that I mapped and here's a dump file, but I still don't support inventories and searches". If we don't want to allow this, then at least the search operation could be mandatory. Or maybe both search and inventory, as you suggested.
any need for inventory operations not to support anyConcept?
I don't know if there's any TAPIR provider not supporting them (anyone here?). If not, perhaps we can change. Anyway, I think we still need to allow inventory templates for providers that won't support filters.
search operation should mandate at least 1 output model support
This is related with dropping TAPIRLite. I'm not convinced that this would be a good change.
cant concept, literal and parameter expressions in filters be mandatory? likewise cant logical operators and equals comparisons be mandatory
filter parts? what about greaterThan and lessThan?
I didn't understand this. Are you talking about capabilities or about the filter definition in the XML Schema? Can you give some example?
Best Regards, -- Renato
PS: I'm traveling next week, so I may not be able to respond promptly. By the way, we can also discuss this during TDWG.
Dear all, I would like to discuss the optional bits of TAPIR capabilities again in the light of service interoperability, especially when dealing with TAPIR lite. Do we really need to have so many optional capabilities for TAPIR? I do firmly think this is causing more problems than benefits and would prefer more mandatory capabilities. In particular I think TAPIR model support should be the lowest common denominator and not only templates as it is currently for TAPIR lite.
Do the following capabilities really need to optional?
- the inventory and search operation is optional. Is there a need for
services not supporting them?
- any need for inventory operations not to support anyConcept?
- search operation should mandate at least 1 output model support
- cant concept, literal and parameter expressions in filters be
mandatory?
- likewise cant logical operators and equals comparisons be mandatory
filter parts? what about greaterThan and lessThan?
regards, Markus
Hi Renato, Markus, Kevin et al
Should a dump file really have anything to do with TAPIR? I would have expected to see it external to TAPIR spec - not to say that the wrapper software can't be a TAPIR wrapper plus a 'dump file' wrapper at the same time.
Similarly with TAPIRLite - maybe I don't get it right, or perhaps it was the early adopters, but one can be a TAPIRLite provider while only answering one search URL pointing at a custom query template with say a couple of GET params passed in, only support a custom response schema, yet claim TAPIR compliance - is this technically correct?
What say you guys? Looking forward to discussing with you all.
Cheers from Singapore Airport,
Tim
Hi Markus,
Sorry about the delay - I've just returned from a workshop.
If I understood your suggestion correctly, all TAPIR providers would need to parse filters, understand search requests with the model parameter, and support inventories in any concepts. This means that query templates would not be necessary anymore, right?
I think it's quite a big change at this point and it will impact existing provider implementations that follow the current TAPIRLite approach.
If there's any serious interoperability problem now, we need to investigate it carefully before making this kind of change. I understand that raising the minimum service profile level can in principle make things easier for clients. But if this is going to make things too difficult for new provider implementations, one of the possible consequences is to see other protocols or even unofficial variations of TAPIR being used, which can be worse than having a single protocol with different service profiles.
What I do think we're missing is guidelines for different types of providers and networks. Simple documents, 1 or 2 pages, with clear recommendations such as what conceptual schemas, output models and query templates to use.
Some comments about your specific suggestions:
the inventory and search operation is optional. Is there a need for
services not supporting them?
We were probably too flexible when we made them optional. However, now that we're considering a change that will allow providers to advertise dump files, I'm just thinking if this can be acceptable: "here's my metadata, here's the conceptual schemas that I mapped and here's a dump file, but I still don't support inventories and searches". If we don't want to allow this, then at least the search operation could be mandatory. Or maybe both search and inventory, as you suggested.
any need for inventory operations not to support anyConcept?
I don't know if there's any TAPIR provider not supporting them (anyone here?). If not, perhaps we can change. Anyway, I think we still need to allow inventory templates for providers that won't support filters.
search operation should mandate at least 1 output model support
This is related with dropping TAPIRLite. I'm not convinced that this would be a good change.
cant concept, literal and parameter expressions in filters be mandatory? likewise cant logical operators and equals comparisons be mandatory
filter parts? what about greaterThan and lessThan?
I didn't understand this. Are you talking about capabilities or about the filter definition in the XML Schema? Can you give some example?
Best Regards,
Renato
PS: I'm traveling next week, so I may not be able to respond promptly. By the way, we can also discuss this during TDWG.
Dear all, I would like to discuss the optional bits of TAPIR capabilities again in the light of service interoperability, especially when dealing with TAPIR lite. Do we really need to have so many optional capabilities for TAPIR? I do firmly think this is causing more problems than benefits and would prefer more mandatory capabilities. In particular I think TAPIR model support should be the lowest common denominator and not only templates as it is currently for TAPIR lite.
Do the following capabilities really need to optional?
- the inventory and search operation is optional. Is there a need for
services not supporting them?
- any need for inventory operations not to support anyConcept?
- search operation should mandate at least 1 output model support
- cant concept, literal and parameter expressions in filters be
mandatory?
- likewise cant logical operators and equals comparisons be mandatory
filter parts? what about greaterThan and lessThan?
regards, Markus
tdwg-tapir mailing list tdwg-tapir@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tapir
____________________________________________________________ Tim Robertson Systems Architect Global Biodiversity Information Facility Secretariat (GBIF) Universitetsparken 15, 2100 Copenhagen Ø, Denmark http://www.gbif.org trobertson@gbif.org Phone: +45 35 32 14 87 (Office) Fax: +45 35 32 14 80 ____________________________________________________________
Hi Tim,
If the provider only has a dump file and is not planning to offer any TAPIR operation in the future, I certainly agree there's nothing much to do with TAPIR. However, we all know that data harvesters may benefit if real TAPIR providers do offer some sort of dump file to help in the initial load. In this case, isn't it better to know from the capabilities response that the provider has a dump file in a certain format available in a certain place? If we don't include this information as part of a capabilities response, how would you know about it? I'm sure you'll prefer not to receive this additional information by e-mail from each provider...
Regarding TAPIRLite, the example you gave will only be technically correct if the service is also able to respond the 3 (currently) mandatory operations (metadata, capabilities and ping) and if the query template is defined according to the TAPIR spec.
Best Regards, -- Renato
Hi Renato, Markus, Kevin et al
Should a dump file really have anything to do with TAPIR? I would have expected to see it external to TAPIR spec - not to say that the wrapper software can't be a TAPIR wrapper plus a 'dump file' wrapper at the same time.
Similarly with TAPIRLite - maybe I don't get it right, or perhaps it was the early adopters, but one can be a TAPIRLite provider while only answering one search URL pointing at a custom query template with say a couple of GET params passed in, only support a custom response schema, yet claim TAPIR compliance - is this technically correct?
What say you guys? Looking forward to discussing with you all.
Cheers from Singapore Airport,
Tim
Hi Renato,
Hmmm... I get what you are saying but would the wrapper tool be better to just register the existence of the dump file (or the meta file describing the dump) in a registry? E.g. A new service binding for a UDDI registration. If I bundled WFS with a TAPIR implementation I wouldn't advertise the existence of that in the capabilities, or would I? I foresee wrapper tools with WFS, WMS, TAPIR, locally generated index files (so called dumps), OAI-PMH, TCS complete dataset files, EML in a single installation - would the capabilities be flexible enough to allow for these additional protocols?
TAPIRLite - I think then to write a client you would have to be able to understand the templated request (E.g. deconstruct the filters) and work out the parameters required in order to construct the URL. To me it seems backwards since it is the provider saying to the client "this is the one query I support" which is more like just advertising a custom webservice - no? We must be misusing it, or shying away from writing complex client code at GBIFS as we only support 2 templates, which effectively says we only support 2 URL types and one output document format. So we don't even issue capabilities requests to these providers since they are known to be coded against the template... misuse?
Really looking forward to picking your brains Renato!
Cheers
Tim
On 13 Oct 2008, at 21:04, Renato De Giovanni wrote:
Hi Tim,
If the provider only has a dump file and is not planning to offer any TAPIR operation in the future, I certainly agree there's nothing much to do with TAPIR. However, we all know that data harvesters may benefit if real TAPIR providers do offer some sort of dump file to help in the initial load. In this case, isn't it better to know from the capabilities response that the provider has a dump file in a certain format available in a certain place? If we don't include this information as part of a capabilities response, how would you know about it? I'm sure you'll prefer not to receive this additional information by e-mail from each provider...
Regarding TAPIRLite, the example you gave will only be technically correct if the service is also able to respond the 3 (currently) mandatory operations (metadata, capabilities and ping) and if the query template is defined according to the TAPIR spec.
Best Regards,
Renato
Hi Renato, Markus, Kevin et al
Should a dump file really have anything to do with TAPIR? I would have expected to see it external to TAPIR spec - not to say that the wrapper software can't be a TAPIR wrapper plus a 'dump file' wrapper at the same time.
Similarly with TAPIRLite - maybe I don't get it right, or perhaps it was the early adopters, but one can be a TAPIRLite provider while only answering one search URL pointing at a custom query template with say a couple of GET params passed in, only support a custom response schema, yet claim TAPIR compliance - is this technically correct?
What say you guys? Looking forward to discussing with you all.
Cheers from Singapore Airport,
Tim
tdwg-tapir mailing list tdwg-tapir@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tapir
Hi Tim,
OK, now I think I can see from your point of view. You're considering a multi-protocol wrapper software on top of the same database, and it's strange to see the dump file being advertised by one of the protocols when it could be useful to any client. From a registry perspective, assuming that non-TAPIR protocols are unlikely to ever give an indication about dump files, this means that besides advertising all endpoints (each protocol with a separate endpoint, I suppose) you will also want to register the dump file as a separate link. A new service binding as you said.
Anyway, regardless the protocol being used by the client it also seems strange to use a specific protocol to interact with the provider but get the information about dump files somewhere else. Since we have the chance to still make adjustments in TAPIR, why not make things easier and more consistent at least for TAPIR clients? If we do this, the multi protocol wrapper could easily advertise the dump file twice: as an independent link (if you wish) and as part of TAPIR capabilities.
Regarding the TapirLite example, it's not exactly a misuse, but it's certainly good practice to periodically check the provider capabilities. In your case, given any TAPIR endpoint, a generic TAPIR client should ideally send a capabilities request first. If the two query templates are supported, you can proceed to the search requests. If not, you can still check if the provider supports the necessary output models. If they are supported, you can proceed to the search requests. If not, you can still check if the provider supports any output model and if the necessary concepts are mapped. If the answer is yes, you can send exactly the same search requests. If not, then the provider won't understand your search requests. This would be the way to cover the 3 TAPIR profile levels. The logic requires some understanding about TAPIR, but it's not complex. However, it definitely helps if you use a generic TAPIR client library to parse the capabilities for you and offer a few methods to return what you need.
Best Regards, -- Renato
Hi Renato,
Hmmm... I get what you are saying but would the wrapper tool be better to just register the existence of the dump file (or the meta file describing the dump) in a registry? E.g. A new service binding for a UDDI registration. If I bundled WFS with a TAPIR implementation I wouldn't advertise the existence of that in the capabilities, or would I? I foresee wrapper tools with WFS, WMS, TAPIR, locally generated index files (so called dumps), OAI-PMH, TCS complete dataset files, EML in a single installation - would the capabilities be flexible enough to allow for these additional protocols?
TAPIRLite - I think then to write a client you would have to be able to understand the templated request (E.g. deconstruct the filters) and work out the parameters required in order to construct the URL. To me it seems backwards since it is the provider saying to the client "this is the one query I support" which is more like just advertising a custom webservice - no? We must be misusing it, or shying away from writing complex client code at GBIFS as we only support 2 templates, which effectively says we only support 2 URL types and one output document format. So we don't even issue capabilities requests to these providers since they are known to be coded against the template... misuse?
Really looking forward to picking your brains Renato!
Cheers
Tim
participants (4)
-
"Markus Döring (GBIF)" -
Renato De Giovanni -
Tim Robertson -
Tim Robertson (GBIF)