January 2006 - tdwg-tag - lists.tdwg.org

Re: RDF query and inference in a distributed environment
by Richard Pyle 04 Jan '06

04 Jan '06

Thanks, Arthur. > It has taken many years to bring the biological collection > community this far (and still many institutions are reluctant to > make their data available via GBIF) - to even propose that ALL > their data be warehoused in mirror sites around the world would, > I fear, cause many to withdraw even the data they are making > available now. As desirable as I believe it is to make as much of > these data freely available as possible, we have to be careful or > we may lose much of which we have already gained. It is a > political/social issue, and as such is a very difficult one to negotiate. ...which brings me back to my original post on this: "...perhaps it would be useful for other shared data as well (geographic localities, people/agents, publications/references, etc.)..." Most of the political/social issues revolve around specimen/observation records, and are less applicable to these "shared" data categories. I believe the "distributed warehouse" approach can also apply to specimen data (as I indicated in the original post), but maybe to help focus the current discussion -- and to re-orient it back to the GUID context -- we should concentrate discussions about such "distributed warehouses" to only globally shared datasets (taxonomy, geography, publications, people), where there is ambiguous "ownership" of the data objects (if not the myriad digital records representing those data objects). Aloha, Rich

1 0

Re: RDF query and inference in a distributed environment
by Richard Pyle 04 Jan '06

04 Jan '06

Kevin Richards wrote: > Coming from an IT background rather than a taxonomic background, > I have never understood the strong "ownership" of data that > people/scientists have for their data. This seems rather short > sited to me - people with these concerns must have some thought > about how to maintain/expose/use "their" data in the long term > future? I can understand their concerns, but there must be a > solution, otherwise their data will be no-ones concern in 50 > years time when it disappears from existence. Philosophically, I agree with you 100%. As a scientist myself, I strongly believe that scientific data (ESPECIALLY from government-funded research) belongs in the public domain. But there are socio-political realities that must be dealt with -- and, as pointed out by Chuck, Patricia, and others, dealt with carefully and with sensitivity. It's really the main reason we're not further along than we currently are. > Another thought I had about data caching systems. Say you > want to search the cached/centralised copy of the data (eg a > GBIF cache). A list of results is returned, then you decide > you want to view more details of one of the results, so you > follow a link off to the associated data (this would > theoretically be by using the GUID system we are discussing). > This would result in viewing the details of the selected > record at the location where the GUID resolves to - this > would always be the same location as a GUID only resolves > to a single location. I'm confused. I thought that a GUID resolves to a single data record. The location of that data record seems to me to be an issue of resolution, not (neccessarily) an intrinsic component of the GUID<->data relationship per se. I don't understand why -- from a purely technological perspective -- a GUID must be resolved through a single designated resolution service, as opposed to any of a multitude of resolution services that conform to a standard protocol. The main issue would be the confidence/certainty that the same GUID would be resolved by any one of the services to the exact same data -- which comes back to the robustness/reliability of the automated synchronization protocols. > Is this correct, or would the intention here be to view > the cached details of the selected record (which would > require an separate ID for all the cached records)? I guess it comes back to what the GUID is attached to (i.e., defined as representing). I view the GUID as a universally adopted surrogate representation of a specified collection of data (information). A more technically rigorous interpretation of a GUID would be as a data-record *instance* identifier. In the former case, the taxon-name data object "Centropyge boylei Pyle & Randall 2001 [etc...]" and the specimen data object "BPBM 35041 [etc...]" would each be representd by a single GUID, regardless of how many instances of the bit-equivalent data records existed on various servers around the world (e.g., my laptop computer, the Catalog of Fishes, GBIF, ITIS, FishBase, Species2000, etc.). In the latter case, each of these two data objects (the taxon name and the specimen) would have a different GUID for each database/server that a copy of the record for the data objects lived on. Aloha, Rich

1 0

AW: RDF query and inference in a distributed environment
by Robert Huber 04 Jan '06

04 Jan '06

Dear all, > Not if robust automated replication/synchronization protocols are > in place. E.g. the Open Archive Initiatives Protocol (OAI) http://openarchives.org. Could also be used to sync e.g. ABDC metadata.

1 0

Re: RDF query and inference in a distributed environment
by Ricardo Scachetti Pereira 04 Jan '06

04 Jan '06

Hi Rich, Richard Pyle wrote: >With hard drive sizes spiraling skyward, and $/GB ($/TB) spiraling >downward.... I'm wondering whether or not the "distributed" system that >serves us best might be "distributeded mirror copies", rather than >distributed complementary data. > Instead of mutually exclusive, both approaches you mention are complementary. The "distributed complementary data" aproach is a fundamental part of the infrastructure necessary to build the "distributed mirror copies" you propose. That (the "distributed complementary data" approach) is essentially what we have in place now with DiGIR/BioCase/Tapir as the harvesting protocol and GBIF and other institutions as harvesters. The only missing piece of software we need to really have "an automated and robust synchronization protocol", in my opinion is some kind of push mechanism to trigger updates in the caches. However, I think it is not very useful to try to standardize how the distributed mirror copies should be built and organized. I also don't think that having exact copies of all metadata available is very helpful. Considering a RDF approach to metadata, it may not be feasible or desirable to harvest every single triple there is. A (socially) descentralized schema would work better in this case: data providers would make available data they create and that is under their direct custody and individual harvesters would be free to look at the metadata being served and create their own caches, selectively harvesting only the information that is relevant to the services they intend to provide. One important point in the GUID discussion is to make sure that IPR is in the metadata and ownership is unanbiguous, regardless of whether the harvester left that information intact or not. As long as the aggregator leave a reference to the original record (using its GUID), any of the GUID technologies under evaluation lets you get back to the ownership information. Regards, Ricardo

1 0

Re: RDF query and inference in a distributed environm ent
by Chuck Miller 04 Jan '06

04 Jan '06

Querying a central database is certainly technically easier and more responsive than distributed data. Mirroring servers is mature technology. And, disk drives get cheaper every day. But, in addition to the social, technical and financial factors that limit the ability to implement distributed caches or full mirrors there is also the political factor, which GBIF has spent great effort successfully addressing. As a global entity, GBIF cannot exist without the political support from its members that leads in turn to financial support. Most of the members of GBIF are sovereign nations who seek to preserve and protect their country's assets. Some protect more than others. To cache or mirror data in multiple locations, the dilemma lies in the simplest question: "Where is my data?" A cache or mirror from a technologist's perspective is just a technical trick, identical to the original, dependent upon the master version, no big deal. But, simplistically, the data may be actually located in another country, no matter how it got there, and some politicians could misconstrue the whole thing, if not properly negotiated and agreed upon in advance. In my experience, negotiating such agreements is more work than the technical development. I think the distributed nature of DiGIR was critical to selling it at the start of GBIF. The original design assured providers that their source data would "stay" in their country and not be wholesale copied somewhere else. It's hard to say what the political effect of creating mirrors would be. Chuck Miller Missouri Botanical Garden _____ From: Patricia Mergen [mailto:p_mergen@YAHOO.COM] Sent: Wednesday, January 04, 2006 5:38 AM To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: [TDWG-GUID] RDF query and inference in a distributed environment Dear Rich Richard Pyle <deepreef(a)BISHOPMUSEUM.ORG> wrote: Hi Patricia, Many thanks for the feedback (and thanks also to Bob -- who I neglected to thank in my previous post). What do you reckon would be the limiting social and financial factors for full mirrors? In social terms, if I'm going to expose my data to the world anyway (e.g., via DiGIR), then I don't see why I would be socially reluctant to allow others to mirror the data (provided robust syncronization protocols are in place -- see my previous response to Bob; and provided data "ownership" credentials are embedded within the core metadata). I agree with you about the logic in this. However accoding to my daily experience with potential dataproviders there is a lot of teaching and conviencing needed to make this logic accepted that this does not result in the loss of control over own data. I agree that to be conviencing a robust syncronization is needed. As for financial, I prefaced my original post with the observation of ever decreasing $/GB for storage space. I suspect that, before TDWG nails down the GUID protocols, entry-level web servers (of the sort that even the most modest DiGIR provider would need to establish) will come with nearly a TB of disk storage space by default. Perhaps the cost of bandwidth will be a limiting factor? Or maybe DB software capable of managing such large datasets? I agree that for machines and storage it is not that expensive. I was more referring to the human ressources needed to manage the mirror. Smaller institutions do not have necessary the funds or cannot justify to their hierachy that staff is devoting time to maintain a full miror containing mainly "references" to information coming from other institutions, but it is easier to justify the time spent to contribute to the whole with the part concerning directly the institution ... As for IPR -- well, ultimately that applies mostly to specimens. And again, assuming that "ownership" metadata remains intact, I see no basis for increased apprehension about allowing mirrored copies of data records (as GBIF already does, for example) over and above exposing them in the first place. Yes I agree with you here too, but as said before this need teaching and convincing ... Personally, I don't think the social, legal, or financial barriers are significantly greater for a mass-mirror paradigm than they are for distributed complementary data sets. I suspect the major barriers will be more technical (i.e., those aforementioned "robust syncronization protocols"). Yes I agree with you that robust syncronization will be needed but as my IT colleague always remind me, I guess we must not forget that setting up an IT infrastructure is most of the time 10 % technical issues to be solved and 90% of the time solving "human problems and barriers" to make it work and accepted ... Pat Aloha, Rich -----Original Message----- From: Taxonomic Databases Working Group GUID Project [mailto:TDWG-GUID@LISTSERV.NHM.KU.EDU]On Behalf Of Patricia Mergen Sent: Tuesday, January 03, 2006 10:31 PM To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: RDF query and inference in a distributed environment Dear Richard I agree with you that several mirror copies will and are needed, preferably well spread geographically as back-ups. This is exactely the approach of GBIF, as they are now in the process to mirror their services. However as highlighted by Bob Morris their is are social, but also financial barriers to have all contributing institutions run a "full" mirror. In order to insure the participation of all those who are willing to, I believe that a distributed system where each provider can participate with his part should be kept. Those who have the ressources could of course set up full mirrors if this match their needs and if this is allowed by the providers (there are also IPRs issues which may be raise here by some institutions). Patricia Richard Pyle wrote: > Long term what I think might happen is that users have their own triple > stores, and as they do queries the results get added to their own > triple store and they can make inferences locally that they are > interested in. MIT's Piggy bank project > (http://simile.mit.edu/piggy-bank/) is an example of this sort of > approach. With hard drive sizes spiraling skyward, and $/GB ($/TB) spiraling downward.... I'm wondering whether or not the "distributed" system that serves us best might be "distributeded mirror copies", rather than distributed complementary data. I've been pushing this approach for taxonomic data for a while, but perhaps it would be useful for other shared data as well (geographic localities, people/agents, publications/references, etc.) Even for specimen data -- where "ownership" is unambiguous -- it seems that as long as the ownership is clearly embedded in the core metadata, there are more fundamental advantages in storing and serving data from multiple data resources, rather than serving it from only one single data resource. One way to look at it would be "robust caching", with automated update capabilities. The main benefits would be: 1) Large-scale distributed backup of the world's biodata (ensuring perpetuity across a changing technological landscape); 2) Performance and reliability enhancement for local data authority needs; 4) Essentially 100% data availability (like DNS), regardless of which servers are up or down at any given moment; 3) Maximization of distributed work/effort for data "maintenance and repair". The point is, the technology discussions would focus less on issues of distributed queries, and more on issues of replication/synchronization and data edit authorization protocols. Perhaps this would be reaching too far, too soon. But on the other han d, I don't see why implementing a "distributed mirror" system would be any more technically, financially, or socially challenging than implementing a distributed query system for distributed data. Aloha, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever. _____ Yahoo! Photos Ring in the New Year with Photo <http://us.rd.yahoo.com/mail_us/taglines/photos/*http:/pg.photos.yahoo.com/p h/page?.file=calendar_splash.html&.dir=> Calendars. Add photos, events, holidays, whatever.

1 0

Re: RDF query and inference in a distributed environment
by Richard Pyle 04 Jan '06

04 Jan '06

Wow! Lots of resposes this morning. Many thanks to all who responded! Some replies: Patricia wrote: > I agree with you about the logic in this. However accoding to my daily > experience with potential dataproviders there is a lot of teaching and > convincing needed to make this logic accepted that this does not result > in the loss of control over own data. I agree that to be conviencing > a robust syncronization is needed. I understand the psychology, and agree that the barrier for "expose your data online so that everyone can access it" strikes a more palatable chord than "expose your data online so that everyone can mirror it". > I agree that for machines and storage it is not that expensive. I was > more referring to the human ressources needed to manage the > mirror. Smaller institutions do not have necessary the funds or cannot > justify to their hierachy that staff is devoting time to maint ain a > full miror containing mainly "references" to information coming from > other institutions, but it is easier to justify the time spent to > contribute to the whole with the part concerning directly the institution ... Agreed -- but I wonder really how much more time would be involved in setting up an automated mirror, compared with, e.g., a DiGIR provider. In my earlier post when I said that barriers would likely be technical, I didn't mean that technology doesn't exist (it does). Rather, I meant that in order to be successful, setting up and maintaining a mirror would have to be no more technically challenging than establishing a DiGIR provider. I'm not sure that's possible (yet). > Yes I agree with you that rob ust syncronization will be needed but > as my IT colleague always remind me, I guess we must not forget that > setting up an IT infrastructure is most of the time 10 % technical > issues to be solved and 90% of the time solving "human problems > and barriers" to make it work and accepted ... I agree -- but part of overcoming the human problems and barriers is to make setting up a mirror site relatively easy for a non-specialist IT person. And this is really a technical challenge. The other human problems & barriers (IPR, allowing access to hard-earned data, confidence that data edit authorization rules will be enforced, etc.) also have some technical foundation, and also exist for the distributed/complementary approach. Ricardo wrote: > Instead of mutually exclusive, both approaches you mention are > complementary. The "distributed complementary data" aproach is a > fundamental part of the infrastructure necessary to build the > "distributed mirror copies" you propose. That (the "distributed > complementary data" approach) is essentially what we have in place now > with DiGIR/BioCase/Tapir as the harvesting protocol and GBIF and other > institutions as harvesters. The only missing piece of software we need > to really have "an automated and robust synchronization protocol", in my > opinion is some kind of push mechanism to trigger updates in the caches. Agreed on all points. > However, I think it is not very useful to try to standardize how the > distributed mirror copies should be built and organized. Here's where I disagree somewhat. Standardized mirror protocols would be a fundamental necessity of the paradigm I described. The paradigm would simply not work without such standardization. Non-standard means non-interactive, and therefore no automated synchronization. What makes it useful is that every participating organization has local (=high performance) access to a full biodiversity dataset. Another thing that makes it useful is that, rather than needing *all* data providers to be fully functional at any given moment for 100% data retrival, only one mirror site needs to be functional. I understand this is the role that GBIF already serves -- and perhaps that's all that's really needed. > A (socially) > descentralized schema would work better in this case: data providers > would make available data they create and that is under their direct > custody and individual harvesters would be free to look at the metadata > being served and create their own caches, selectively harvesting only > the information that is relevant to the services they intend to provide. There really should be no difference, as long as data edit authorization protocols are as robust as the synchronization protocols. If I'm exposing my specimen data for the world to access (e.g., via DiGIR/BioCase/Tapir), then it shouldn't bother me that GBIF is mirroring it -- as long as GBIF includes metadata about data ownership, and as long as the only people who can edit the data are people who I authorize to do so. And if GBIF is mirroring it, what difference is there if there are 100, or 1000, or 10,000 other servers mirroring it (same assumptions). The two things that are important to me are: 1) corrections to data get propagated quickly to mirrors; and 2) only people who I authorize to edit my data may do so. Other than that, it doesn't really matter if the data are only on my server's hard drive, or are replicated on GBIF's hard drive, or are replicated on 10,000 server hard drives. But the 10,000 hard drives option gives me two advantages: 1) confidence that the data will survive even continental-scale catastrophies; and 2) it means that my server's hard drive (one of the 10,000) has local and immediate access to everyone else's data (for use as authorities, etc.) Rod Page wrote: [an excellent and thought-proviking post, making important, if uncomfortable, observations] > What is present in bioinformatics are large numbers of "value added" > databases that take GenBank, PubMed, etc. and do neat things with > them. This is possible because you can download the entire database. Exactly! The real value of portals to end users are the value-added "neat things". I think we ought to encourage as many enterprising biology/data nerds as possible to build such things. > Each one of these value added databases does need to deal with the > issue of what happens when GenBank (say) changes, but because GenBank > has well defined releases, essentially they can grab a new copy of the > data, update their local copy, and regenerate their database. Wouldn't it be so much nicer if the protocols were in place to automaticaly propagate updates to the value-added sites in real time, so that the aggregated databases were effectively self-maintaining? I guess the fundamental question of this whole discussion ultimately boils down to "push" vs. "pull" paradigms. > In summary, I think the issue raised by Rich is important, but is one > to be addressed by whoever takes on the task of assembling a data > warehouse from the individual providers. Of course, once providers make > their data available, anybody can do this... Yes, but aybody (everybody) has to rebuild it themselves from scratch. What I'm suggesting is to use the TDWG/GBIF/GUID standard community to come up with standard protocols to lower the technological bar of becomming an aggregator -- so much so that it becomes the real foundation of data distribution, dissemination, and access (and makes it feasible for the enterprising biology/data nerds to make good use of their late night hours....) In a later post, Rod wrote: > I think there is a continuum of possibilities: > > 1. Pure distributed - query always sent to remote sources, no data ever > held locally > > 2. Distributed with cache - results from a user query are cached > (either for a limited time, or until source sends a message saying its > data has been updated). > > 3. Distributed with partial local copy - some information harvested > from sources stored locally (e.g., metadata), detailed information only > held by sources. > > 4. Not distributed (data warehouse) - all data from distributed sources > held locally (harvested), with periodic updates. I would add to this list: 5. Distributed Warehouses -- all data is mirrored across many warehouses, automatically kept in synchronization in real time. Queries are sent to any one warehouse. Also, Rod's reference to "The Cathedral and the Bazaar" is excellent and highly relevant. The example of the Hawaii Biological Survey is a good example of this in our community. Chuck Miller wrote: > To cache or mirror data in multiple locations, the dilemma lies > in the simplest question: Where is my data? A cache or mirror > from a technologists perspective is just a technical trick, identical > to the original, dependent upon the master version, no big deal. > But, simplistically, the data may be actually located in another > country, no matter how it got there, and some politicians could m > isconstrue the whole thing, if not properly negotiated and agreed > upon in advance. In my experience, negotiating such agreements > is more work than the technical development. I agree the political issues may be show-stoppers -- but mostly because of unwarranted paranoia on the part of the data owners (unfortunate, but certainly real). But the reality is, any data exposed on a distributed provider (indeed, any data published in any form -- electronic or paper) is already "out there", and located in many different countries. Data contained in a printed publication simulatnously exist in many countries. The only difference with the distributed warehouse approach is that the data are much, MUCH more powerful. Perhaps the real political barrier is putting so much power into so many hands. Sort of like the World Wide Web itself. > I think the distributed nature of DiGIR was critical to selling > it at the start of GBIF. The original design assured providers that > their source data would stay in their country and not be wholesale > copied somewhere else. Its hard to say what the political effect > of creating mirrors would be. Agreed on all points. There was a time when the world wasn't even even ready for the DiGIR approach. Perhaps it's too soon to push it to the next level. Bear in mind, though, that all these IPR/political issues relate mostly to specimen data. The real value of the "distributed warehouses" approach to the warehouses providers themselves are the things that would traditionally be though of as "authorities" (taxon names, reference citations, geographic gazetteers, etc.) Maybe the distributed warehouse approach will work well for these kinds of data, but not for specimen data? O.K., I should probably stop here and get to work.... Aloha, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html

1 0

Re: RDF query and inference in a distributed environm ent
by Patricia Mergen 04 Jan '06

04 Jan '06

Dear Colleagues Yes I agree with Chuck that we have to be carefull here. For example providers have agreed to make their data public by providing it to GBIF, but are not aware what this really implies. Most of our providers are for example very pleased to have now their georeferenced data so easy and nice in google earth others are not because of the lack of precision or the fact that apparently some center of areas are shown fasely as too precise sampling points. The same reactions of lost of control or "where does my data show up and how" will certainly come with mirrors, caching or "central" GUID attributions. Before suggesting anything official as a GUID and GUID system, I agree with Chuck that we should be aware of political implications and maybe misinterpretation of our purely technical purposes. I guess there will be some "legal" documents needed to negociate and agree in advance in form maybe as an extension to GBIF data users and providers agreements ? Patricia Chuck Miller <Chuck.Miller(a)MOBOT.ORG> wrote: v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);} st1\:*{behavior:url(#default#ieooui) } Querying a central database is certainly technically easier and more responsive than distributed data. Mirroring servers is mature technology. And, disk drives get cheaper every day. But, in addition to the social, technical and financial factors that limit the ability to implement distributed caches or full mirrors there is also the political factor, which GBIF has spent great effort successfully addressing. As a global entity, GBIF cannot exist without the political support from its members that leads in turn to financial support. Most of the members of GBIF are sovereign nations who seek to preserve and protect their country’s assets. Some protect more than others. To cache or mirror data in multiple locations, the dilemma lies in the simplest question: “Where is my data?” A cache or mirror from a technologist’s perspective is just a technical trick, identical to the original, dependent upon the master version, no big deal. But, simplistically, the data may be actually located in another country, no matter how it got there, and some politicians could misconstrue the whole thing, if not properly negotiated and agreed upon in advance. In my experience, negotiating such agreements is more work than the technical development. I think the distributed nature of DiGIR was critical to selling it at the start of GBIF. The original design assured providers that their source data would “stay” in their country and not be wholesale copied somewhere else. It’s hard to say what the political effect of creating mirrors would be. Chuck Miller Missouri Botanical Garden --------------------------------- From: Patricia Mergen [mailto:p_mergen@YAHOO.COM] Sent: Wednesday, January 04, 2006 5:38 AM To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: [TDWG-GUID] RDF query and inference in a distributed environment Dear Rich Richard Pyle <deepreef(a)BISHOPMUSEUM.ORG> wrote: Hi Patricia, Many thanks for the feedback (and thanks also to Bob -- who I neglected to thank in my previous post). What do you reckon would be the limiting social and financial factors for full mirrors? In social terms, if I'm going to expose my data to the world anyway (e.g., via DiGIR), then I don't see why I would be socially reluctant to allow others to mirror the data (provided robust syncronization protocols are in place -- see my previous response to Bob; and provided data "ownership" credentials are embedded within the core metadata). I agree with you about the logic in this. However accoding to my daily experience with potential dataproviders there is a lot of teaching and conviencing needed to make this logic accepted that this does not result in the loss of control over own data. I agree that to be conviencing a robust syncronization is needed. As for financial, I prefaced my original post with the observation of ever decreasing $/GB for storage space. I suspect that, before TDWG nails down the GUID protocols, entry-level web servers (of the sort that even the most modest DiGIR provider would need to establish) will come with nearly a TB of disk storage space by default. Perhaps the cost of bandwidth will be a limiting factor? Or maybe DB software capable of managing such large datasets? I agree that for machines and storage it is not that expensive. I was more referring to the human ressources needed to manage the mirror. Smaller institutions do not have necessary the funds or cannot justify to their hierachy that staff is devoting time to maintain a full miror containing mainly "references" to information coming from other institutions, but it is easier to justify the time spent to contribute to the whole with the part concerning directly the institution ... As for IPR -- well, ultimately that applies mostly to specimens. And again, assuming that "ownership" metadata remains intact, I see no basis for increased apprehension about allowing mirrored copies of data records (as GBIF already does, for example) over and above exposing them in the first place. Yes I agree with you here too, but as said before this need teaching and convincing ... Personally, I don't think the social, legal, or financial barriers are significantly greater for a mass-mirror paradigm than they are for distributed complementary data sets. I suspect the major barriers will be more technical (i.e., those aforementioned "robust syncronization protocols"). Yes I agree with you that robust syncronization will be needed but as my IT colleague always remind me, I guess we must not forget that setting up an IT infrastructure is most of the time 10 % technical issues to be solved and 90% of the time solving "human problems and barriers" to make it work and accepted ... Pat Aloha, Rich -----Original Message----- From: Taxonomic Databases Working Group GUID Project [mailto:TDWG-GUID@LISTSERV.NHM.KU.EDU]On Behalf Of Patricia Mergen Sent: Tuesday, January 03, 2006 10:31 PM To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: RDF query and inference in a distributed environment Dear Richard I agree with you that several mirror copies will and are needed, preferably well spread geographically as back-ups. This is exactely the approach of GBIF, as they are now in the process to mirror their services. However as highlighted by Bob Morris their is are social, but also financial barriers to have all contributing institutions run a "full" mirror. In order to insure the participation of all those who are willing to, I believe that a distributed system where each provider can participate with his part should be kept. Those who have the ressources could of course set up full mirrors if this match their needs and if this is allowed by the providers (there are also IPRs issues which may be raise here by some institutions). Patricia Richard Pyle wrote: > Long term what I think might happen is that users have their own triple > stores, and as they do queries the results get added to their own > triple store and they can make inferences locally that they are > interested in. MIT's Piggy bank project > (http://simile.mit.edu/piggy-bank/) is an example of this sort of > approach. With hard drive sizes spiraling skyward, and $/GB ($/TB) spiraling downward.... I'm wondering whether or not the "distributed" system that serves us best might be "distributeded mirror copies", rather than distributed complementary data. I've been pushing this approach for taxonomic data for a while, but perhaps it would be useful for other shared data as well (geographic localities, people/agents, publications/references, etc.) Even for specimen data -- where "ownership" is unambiguous -- it seems that as long as the ownership is clearly embedded in the core metadata, there are more fundamental advantages in storing and serving data from multiple data resources, rather than serving it from only one single data resource. One way to look at it would be "robust caching", with automated update capabilities. The main benefits would be: 1) Large-scale distributed backup of the world's biodata (ensuring perpetuity across a changing technological landscape); 2) Performance and reliability enhancement for local data authority needs; 4) Essentially 100% data availability (like DNS), regardless of which servers are up or down at any given moment; 3) Maximization of distributed work/effort for data "maintenance and repair". The point is, the technology discussions would focus less on issues of distributed queries, and more on issues of replication/synchronization and data edit authorization protocols. Perhaps this would be reaching too far, too soon. But on the other han d, I don't see why implementing a "distributed mirror" system would be any more technically, financially, or socially challenging than implementing a distributed query system for distributed data. Aloha, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever. --------------------------------- Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever. --------------------------------- Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever.

1 0

Re: RDF query and inference in a distributed environment
by Patricia Mergen 03 Jan '06

03 Jan '06

Dear Rich Richard Pyle <deepreef(a)BISHOPMUSEUM.ORG> wrote: Hi Patricia, Many thanks for the feedback (and thanks also to Bob -- who I neglected to thank in my previous post). What do you reckon would be the limiting social and financial factors for full mirrors? In social terms, if I'm going to expose my data to the world anyway (e.g., via DiGIR), then I don't see why I would be socially reluctant to allow others to mirror the data (provided robust syncronization protocols are in place -- see my previous response to Bob; and provided data "ownership" credentials are embedded within the core metadata). I agree with you about the logic in this. However accoding to my daily experience with potential dataproviders there is a lot of teaching and conviencing needed to make this logic accepted that this does not result in the loss of control over own data. I agree that to be conviencing a robust syncronization is needed. As for financial, I prefaced my original post with the observation of ever decreasing $/GB for storage space. I suspect that, before TDWG nails down the GUID protocols, entry-level web servers (of the sort that even the most modest DiGIR provider would need to establish) will come with nearly a TB of disk storage space by default. Perhaps the cost of bandwidth will be a limiting factor? Or maybe DB software capable of managing such large datasets? I agree that for machines and storage it is not that expensive. I was more referring to the human ressources needed to manage the mirror. Smaller institutions do not have necessary the funds or cannot justify to their hierachy that staff is devoting time to maintain a full miror containing mainly "references" to information coming from other institutions, but it is easier to justify the time spent to contribute to the whole with the part concerning directly the institution ... As for IPR -- well, ultimately that applies mostly to specimens. And again, assuming that "ownership" metadata remains intact, I see no basis for increased apprehension about allowing mirrored copies of data records (as GBIF already does, for example) over and above exposing them in the first place. Yes I agree with you here too, but as said before this need teaching and convincing ... Personally, I don't think the social, legal, or financial barriers are significantly greater for a mass-mirror paradigm than they are for distributed complementary data sets. I suspect the major barriers will be more technical (i.e., those aforementioned "robust syncronization protocols"). Yes I agree with you that robust syncronization will be needed but as my IT colleague always remind me, I guess we must not forget that setting up an IT infrastructure is most of the time 10 % technical issues to be solved and 90% of the time solving "human problems and barriers" to make it work and accepted ... Pat Aloha, Rich -----Original Message----- From: Taxonomic Databases Working Group GUID Project [mailto:TDWG-GUID@LISTSERV.NHM.KU.EDU]On Behalf Of Patricia Mergen Sent: Tuesday, January 03, 2006 10:31 PM To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: RDF query and inference in a distributed environment Dear Richard I agree with you that several mirror copies will and are needed, preferably well spread geographically as back-ups. This is exactely the approach of GBIF, as they are now in the process to mirror their services. However as highlighted by Bob Morris their is are social, but also financial barriers to have all contributing institutions run a "full" mirror. In order to insure the participation of all those who are willing to, I believe that a distributed system where each provider can participate with his part should be kept. Those who have the ressources could of course set up full mirrors if this match their needs and if this is allowed by the providers (there are also IPRs issues which may be raise here by some institutions). Patricia Richard Pyle wrote: > Long term what I think might happen is that users have their own triple > stores, and as they do queries the results get added to their own > triple store and they can make inferences locally that they are > interested in. MIT's Piggy bank project > (http://simile.mit.edu/piggy-bank/) is an example of this sort of > approach. With hard drive sizes spiraling skyward, and $/GB ($/TB) spiraling downward.... I'm wondering whether or not the "distributed" system that serves us best might be "distributeded mirror copies", rather than distributed complementary data. I've been pushing this approach for taxonomic data for a while, but perhaps it would be useful for other shared data as well (geographic localities, people/agents, publications/references, etc.) Even for specimen data -- where "ownership" is unambiguous -- it seems that as long as the ownership is clearly embedded in the core metadata, there are more fundamental advantages in storing and serving data from multiple data resources, rather than serving it from only one single data resource. One way to look at it would be "robust caching", with automated update capabilities. The main benefits would be: 1) Large-scale distributed backup of the world's biodata (ensuring perpetuity across a changing technological landscape); 2) Performance and reliability enhancement for local data authority needs; 4) Essentially 100% data availability (like DNS), regardless of which servers are up or down at any given moment; 3) Maximization of distributed work/effort for data "maintenance and repair". The point is, the technology discussions would focus less on issues of distributed queries, and more on issues of replication/synchronization and data edit authorization protocols. Perhaps this would be reaching too far, too soon. But on the other han d, I don't see why implementing a "distributed mirror" system would be any more technically, financially, or socially challenging than implementing a distributed query system for distributed data. Aloha, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever. --------------------------------- Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever.

1 0

Re: RDF query and inference in a distributed environment
by Patricia Mergen 03 Jan '06

03 Jan '06

Dear Richard I agree with you that several mirror copies will and are needed, preferably well spread geographically as back-ups. This is exactely the approach of GBIF, as they are now in the process to mirror their services. However as highlighted by Bob Morris their is are social, but also financial barriers to have all contributing institutions run a "full" mirror. In order to insure the participation of all those who are willing to, I believe that a distributed system where each provider can participate with his part should be kept. Those who have the ressources could of course set up full mirrors if this match their needs and if this is allowed by the providers (there are also IPRs issues which may be raise here by some institutions). Patricia Richard Pyle <deepreef(a)BISHOPMUSEUM.ORG> wrote: > Long term what I think might happen is that users have their own triple > stores, and as they do queries the results get added to their own > triple store and they can make inferences locally that they are > interested in. MIT's Piggy bank project > (http://simile.mit.edu/piggy-bank/) is an example of this sort of > approach. With hard drive sizes spiraling skyward, and $/GB ($/TB) spiraling downward.... I'm wondering whether or not the "distributed" system that serves us best might be "distributeded mirror copies", rather than distributed complementary data. I've been pushing this approach for taxonomic data for a while, but perhaps it would be useful for other shared data as well (geographic localities, people/agents, publications/references, etc.) Even for specimen data -- where "ownership" is unambiguous -- it seems that as long as the ownership is clearly embedded in the core metadata, there are more fundamental advantages in storing and serving data from multiple data resources, rather than serving it from only one single data resource. One way to look at it would be "robust caching", with automated update capabilities. The main benefits would be: 1) Large-scale distributed backup of the world's biodata (ensuring perpetuity across a changing technological landscape); 2) Performance and reliability enhancement for local data authority needs; 4) Essentially 100% data availability (like DNS), regardless of which servers are up or down at any given moment; 3) Maximization of distributed work/effort for data "maintenance and repair". The point is, the technology discussions would focus less on issues of distributed queries, and more on issues of replication/synchronization and data edit authorization protocols. Perhaps this would be reaching too far, too soon. But on the other hand, I don't see why implementing a "distributed mirror" system would be any more technically, financially, or socially challenging than implementing a distributed query system for distributed data. Aloha, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html --------------------------------- Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever.

1 0

Re: RDF query and inference in a distributed environment
by Richard Pyle 03 Jan '06

03 Jan '06

Hi Patricia, Many thanks for the feedback (and thanks also to Bob -- who I neglected to thank in my previous post). What do you reckon would be the limiting social and financial factors for full mirrors? In social terms, if I'm going to expose my data to the world anyway (e.g., via DiGIR), then I don't see why I would be socially reluctant to allow others to mirror the data (provided robust syncronization protocols are in place -- see my previous response to Bob; and provided data "ownership" credentials are embedded within the core metadata). As for financial, I prefaced my original post with the observation of ever decreasing $/GB for storage space. I suspect that, before TDWG nails down the GUID protocols, entry-level web servers (of the sort that even the most modest DiGIR provider would need to establish) will come with nearly a TB of disk storage space by default. Perhaps the cost of bandwidth will be a limiting factor? Or maybe DB software capable of managing such large datasets? As for IPR -- well, ultimately that applies mostly to specimens. And again, assuming that "ownership" metadata remains intact, I see no basis for increased apprehension about allowing mirrored copies of data records (as GBIF already does, for example) over and above exposing them in the first place. Personally, I don't think the social, legal, or financial barriers are significantly greater for a mass-mirror paradigm than they are for distributed complementary data sets. I suspect the major barriers will be more technical (i.e., those aforementioned "robust syncronization protocols"). Aloha, Rich -----Original Message----- From: Taxonomic Databases Working Group GUID Project [mailto:TDWG-GUID@LISTSERV.NHM.KU.EDU]On Behalf Of Patricia Mergen Sent: Tuesday, January 03, 2006 10:31 PM To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: RDF query and inference in a distributed environment Dear Richard I agree with you that several mirror copies will and are needed, preferably well spread geographically as back-ups. This is exactely the approach of GBIF, as they are now in the process to mirror their services. However as highlighted by Bob Morris their is are social, but also financial barriers to have all contributing institutions run a "full" mirror. In order to insure the participation of all those who are willing to, I believe that a distributed system where each provider can participate with his part should be kept. Those who have the ressources could of course set up full mirrors if this match their needs and if this is allowed by the providers (there are also IPRs issues which may be raise here by some institutions). Patricia Richard Pyle <deepreef(a)BISHOPMUSEUM.ORG> wrote: > Long term what I think might happen is that users have their own triple > stores, and as they do queries the results get added to their own > triple store and they can make inferences locally that they are > interested in. MIT's Piggy bank project > (http://simile.mit.edu/piggy-bank/) is an example of this sort of > approach. With hard drive sizes spiraling skyward, and $/GB ($/TB) spiraling downward.... I'm wondering whether or not the "distributed" system that serves us best might be "distributeded mirror copies", rather than distributed complementary data. I've been pushing this approach for taxonomic data for a while, but perhaps it would be useful for other shared data as well (geographic localities, people/agents, publications/references, etc.) Even for specimen data -- where "ownership" is unambiguous -- it seems that as long as the ownership is clearly embedded in the core metadata, there are more fundamental advantages in storing and serving data from multiple data resources, rather than serving it from only one single data resource. One way to look at it would be "robust caching", with automated update capabilities. The main benefits would be: 1) Large-scale distributed backup of the world's biodata (ensuring perpetuity across a changing technological landscape); 2) Performance and reliability enhancement for local data authority needs; 4) Essentially 100% data availability (like DNS), regardless of which servers are up or down at any given moment; 3) Maximization of distributed work/effort for data "maintenance and repair". The point is, the technology discussions would focus less on issues of distributed queries, and more on issues of replication/synchronization and data edit authorization protocols. Perhaps this would be reaching too far, too soon. But on the other han d, I don't see why implementing a "distributed mirror" system would be any more technically, financially, or socially challenging than implementing a distributed query system for distributed data. Aloha, Rich Richard L. Pyle, PhD Database Coordinator for Natural Sciences and Associate Zoologist in Ichthyology Department of Natural Sciences, Bishop Museum 1525 Bernice St., Honolulu, HI 96817 Ph: (808)848-4115, Fax: (808)847-8252 email: deepreef(a)bishopmuseum.org http://hbs.bishopmuseum.org/staff/pylerichard.html Yahoo! Photos Ring in the New Year with Photo Calendars. Add photos, events, holidays, whatever.

1 0