January 2006 - tdwg-tag - lists.tdwg.org

Re: RDF query and inference in a distributed environment
by Roderic Page 04 Jan '06

04 Jan '06

Without wishing to labour the point, the latest issue of Nature has a nice piece by Declan Butler on "mashups", which mentions my iSpecies toy, and some more serious efforts by Donat Agosti and Norman Johnson, as well as Declan's own work on display avian flu records in Google Earth. I'm quoted as gushing: 'If you could pool data from every museum or lab in the world, "you could do amazing things", says Page.' This is what happens when being interviewed by Nature goes to your head, as well as being distracted by trying to keep my four year old daughter occupied while on the phone to Declan ("Dad, I want a drink NOW"). Still, I stand by the sentiment. As an example, take a look at what Declan himslef managed to do with data on avian flu (http://declanbutler.info/blog/?p=16). Regards Rod ------------------------------------------------------------------------ ---------------------------------------- Professor Roderic D. M. Page Editor, Systematic Biology DEEB, IBLS Graham Kerr Building University of Glasgow Glasgow G12 8QP United Kingdom Phone: +44 141 330 4778 Fax: +44 141 330 2792 email: r.page(a)bio.gla.ac.uk web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html Subscribe to Systematic Biology through the Society of Systematic Biologists Website: http://systematicbiology.org Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ Find out what we know about a species at http://ispecies.org

1 0

Re: RDF query and inference in a distributed environm ent
by Roderic Page 04 Jan '06

04 Jan '06

> Yes I agree with Chuck that we have to be carefull here. > > For example providers have agreed to make their data public by > providing it to GBIF, but are not aware what this really implies. Most > of our providers are for example very pleased to have now their > georeferenced data so easy and nice in google earth others are not > because of the lack of precision or the fact that apparently some > center of areas are shown fasely as too precise sampling points. > Conversely, having poorly georeferenced data so readily visible could > be seen as a Good Thing (tm). Eric Raymond's phrase "Given enough eyeballs, all bugs are shallow" springs to mind, the idea being that in open software development, having many people involved tends to find bugs fast, and fix them quickly. This phrase comes from his article "The Cathedral and the Bazaar" (http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/), which I'd recommend. The idea is that some people think computer software is best done by a small number of experts (the cathedral), whereas the lesson from open source is that a bunch of people, seemingly working in chaos (the bazaar), can yield better results. The culture of taxonomy is driven by reliance on a few experts, whereas I suggest we'd be better off involving as many people as possible. This means making our data available, being prepared to expose "bugs" (such as bad georeferencing), and developing mechanisms to fix mistakes. If we rely solely on experts, I think we're doomed. Here's one example. When playing with my iSpecies.org toy, I entered "Hoplobatrachus tigerinus", which it turns out is an Asian frog. If I go to GBIF and search on this name (http://www.secretariat.gbif.net:80/portal/ecat_browser.jsp? taxonKey=358519&countryKey=0&resourceKey=0), I discover that ITIS lists Bufo typhonius as a synonym. From that link, GBIF gives me lots of records from America. Now, the history of these two frog names is a bit messy, but it's clear from Amphibian Species of the World (http://research.amnh.org/herpetology/amphibia/references.php?id=16455 and http://research.amnh.org/herpetology/amphibia/references.php?id=1578) that the records from the Americas that GBIF lists have nothing to do with Hoplobatrachus tigerinus. Now, I know nothing about frogs, but just by playing I've found a problem with records served by several major museums. I wouldn't mind betting that the museums themselves are unaware of the problem with this record -- given enough eyeballs, all bugs are shallow. Regards Rod ------------------------------------------------------------------------ ---------------------------------------- Professor Roderic D. M. Page Editor, Systematic Biology DEEB, IBLS Graham Kerr Building University of Glasgow Glasgow G12 8QP United Kingdom Phone: +44 141 330 4778 Fax: +44 141 330 2792 email: r.page(a)bio.gla.ac.uk web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html Subscribe to Systematic Biology through the Society of Systematic Biologists Website: http://systematicbiology.org Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ Find out what we know about a species at http://ispecies.org

1 0

Re: RDF query and inference in a distributed environment
by Roderic Page 04 Jan '06

04 Jan '06

On 4 Jan 2006, at 13:23, Sally Hinchcliffe wrote: > Species 2000 uses a federated data model. As to whether that counts as > 'large scale' or not I don't know. I believe that even getting > permission for a data cache from the various providers was a painful > exercise. > Sally I think there is a continuum of possibilities: 1. Pure distributed - query always sent to remote sources, no data ever held locally 2. Distributed with cache - results from a user query are cached (either for a limited time, or until source sends a message saying its data has been updated). 3. Distributed with partial local copy - some information harvested from sources stored locally (e.g., metadata), detailed information only held by sources. 4. Not distributed (data warehouse) - all data from distributed sources held locally (harvested), with periodic updates. My own Taxonomic Search Engine belongs in category 2 (it has a 24 hour cache for queries made by users). I'm guessing GBIF is category 3 (or is it 4, I can get full details from GBIF as well as from remote source). I'm not totally sure where Species 2000's SPICE fits in, they certainly use a cache, and my feeling is they cache more than just queries made by individual users (if the speed of response is anything to go by). From 1 -4 we trade off several things. 1 gives us the freshest data, at the cost of speed (querying multiple sources takes time, sources can go offline, etc.), 4 gives us maximum power (we can do queries that are hard if not impossible in a distributed environment, such as find how many names are in common to two data sources) at the cost of possibly being out of date. My comment was really about the lack of large scale federated systems was really about systems of type 1 and 2, where essentially every search query is live. If the search is done on a local copy of the data, or on a persistent cache, then it's not really distributed (IMHO). Regards Rod ------------------------------------------------------------------------ ---------------------------------------- Professor Roderic D. M. Page Editor, Systematic Biology DEEB, IBLS Graham Kerr Building University of Glasgow Glasgow G12 8QP United Kingdom Phone: +44 141 330 4778 Fax: +44 141 330 2792 email: r.page(a)bio.gla.ac.uk web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html Subscribe to Systematic Biology through the Society of Systematic Biologists Website: http://systematicbiology.org Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ Find out what we know about a species at http://ispecies.org

1 0

Re: RDF query and inference in a distributed environment
by Robert Huber 04 Jan '06

04 Jan '06

Dear all, In my (as a non taxonomist) point of view, the distributed GBIF model is not the real problem but the protocol. DiGIR etc try to offer 'quasi SQL' query possibilities (The 'SOAP' approach) in a distributed environment. However, even GBIF apparently does not use the protocol as it was intended, it uses a 'Harvesting approach' and offers access to the metadata via its cache (please correct me if Im wrong) probably because of some provider problems. As I mentioned in an earlier mesage,we favor here the harvesting approach. Especially in a highly heterogenous metaddata store which offers eg. oceonographic, biogeographic, taxonomic and literature data the protocol needs to be almost independent from the metadata. We use OAI to harvest most of these third party data sources. Depending on the portal purpose we (will) offer the appropriate query possibilities. It would be very interesting to me whether OAI ever was discussed in the TDWG or for use in GBIF? best regards, Robert > -----Ursprungliche Nachricht----- > Von: Taxonomic Databases Working Group GUID Project > [mailto:TDWG-GUID@LISTSERV.NHM.KU.EDU]Im Auftrag von Roderic Page > Gesendet: Mittwoch, 4. Januar 2006 13:51 > An: TDWG-GUID(a)LISTSERV.NHM.KU.EDU > Betreff: Re: RDF query and inference in a distributed environment > > > I wonder whether at some point we need to think carefully about why we > have a distributed model in the first place. Is it the best choice? Did > we chose it, or has it emerged primarily because data providers > want/need to keep control over "their" data? > > In the context of bioinformatics, I'm not aware of any large scale > distributed environments that actually work (probably ignorance on my > part). The largest databases (GenBank, PubMed, EMBL, DDBJ) have all the > data stored centrally, and mirror at least some of it (e.g., sequences > are mirrored between GenBank, EMBL, and DDBJ). Hence, issues of > synchronisation are largely limited to these big databases, and hence > manageable. > > My sense is that there is a lot of computer science research on > federated databases, but few actual large-scale systems that people > make regular use of. > > What is present in bioinformatics are large numbers of "value added" > databases that take GenBank, PubMed, etc. and do neat things with > them. This is possible because you can download the entire database. > Each one of these value added databases does need to deal with the > issue of what happens when GenBank (say) changes, but because GenBank > has well defined releases, essentially they can grab a new copy of the > data, update their local copy, and regenerate their database. > > Having access to all the data makes all kinds of things possible which > are harder to do if the data is distributed. I'd argue that part of the > success of bioinformatics is because of data availability. > > Hence, my own view (at least today) is: > > 1. Individual data providers manage their own data, and also make > available their data in the following ways: > i) provide GUIDs and metadata (e.g., LSIDs) > ii) provide a basic, standard search web service > iii) provide their own web interface > iv) periodically provide complete dump of data > > 2. GBIF (or some equivalent) takes on the job of harvesting all > providers, building a warehouse, and making that data available through > i) web interface > ii) web services > iii) complete data dump > > 3. Researchers can decide how best to make use of this data. They may > wish to get a complete "GBIF" dump and install that locally, or query > GBIF, or they may wish to query the individual providers for the most > up to date information, or some mixture of this. > > Issues of synchronisation are dealt with by GBIF and its providers, > which I think essentially amounts to having versioning and release > numbers (but I'm probably being naive). > > Probably 1iv and 3iii are going to cause some issues, and this is off > topic, but if bioinformatics is anything to go by, if we don't make our > data available in bulk we are tying our own hands. However, this is > obviously something each individual provider will have to decide upon > themselves. > > My other feeling is that from the point of view of end users (and I > class myself as one) the real game will be services, especially search > (think "Google Life"). And my feeling is that this won't work if > queries are done in a distributed fashion -- the Web is supremely > distributed, but Google doesn't query the Web, it queries its local > copy. > > In summary, I think the issue raised by Rich is important, but is one > to be addressed by whoever takes on the task of assembling a data > warehouse from the individual providers. Of course, once providers make > their data available, anybody can do this... > > Regards > > Rod > > > ------------------------------------------------------------------------ > ---------------------------------------- > Professor Roderic D. M. Page > Editor, Systematic Biology > DEEB, IBLS > Graham Kerr Building > University of Glasgow > Glasgow G12 8QP > United Kingdom > > Phone: +44 141 330 4778 > Fax: +44 141 330 2792 > email: r.page(a)bio.gla.ac.uk > web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html > reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html > > Subscribe to Systematic Biology through the Society of Systematic > Biologists Website: http://systematicbiology.org > Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ > Find out what we know about a species at http://ispecies.org

1 0

Re: RDF query and inference in a distributed environment
by Donald Hobern 04 Jan '06

04 Jan '06

Rod, This is an excellent question. It is quite clear that much of what we want to do would be much easier, more efficient and more authoritative if we established a data-warehousing model for our data. To my mind, it should not be impossible to develop such a system while still giving data providers the right and ability to pull their data at any subsequent date. (Indeed it may be simpler to assure this possibility in a centralised model than it is with a distributed model in which users will inevitably try to make their own local caches.) The driver for the distributed model has been precisely that many data providers have stated the concern that they should maintain control over their data. The current GBIF data index is an attempt to balance the needs of users with these concerns from providers. I am sure that we will continue to review the situation, but for now my hope is that we will be able to follow just about exactly the path you describe below. I see this work on GUIDs as an important step in making it possible to do this much better than we have been able to do up to now. All of this will however critically require a careful review of the real expectations, fears and needs of our data providers. They have up to this point not given us any explicit mandate to offer complete data dumps. We need therefore to work with them to develop a legitimate and acceptable strategy. Donald --------------------------------------------------------------- Donald Hobern (dhobern(a)gbif.org) Programme Officer for Data Access and Database Interoperability Global Biodiversity Information Facility Secretariat Universitetsparken 15, DK-2100 Copenhagen, Denmark Tel: +45-35321483 Mobile: +45-28751483 Fax: +45-35321480 --------------------------------------------------------------- -----Original Message----- From: Taxonomic Databases Working Group GUID Project [mailto:TDWG-GUID@LISTSERV.NHM.KU.EDU] On Behalf Of Roderic Page Sent: 04 January 2006 13:51 To: TDWG-GUID(a)LISTSERV.NHM.KU.EDU Subject: Re: RDF query and inference in a distributed environment I wonder whether at some point we need to think carefully about why we have a distributed model in the first place. Is it the best choice? Did we chose it, or has it emerged primarily because data providers want/need to keep control over "their" data? In the context of bioinformatics, I'm not aware of any large scale distributed environments that actually work (probably ignorance on my part). The largest databases (GenBank, PubMed, EMBL, DDBJ) have all the data stored centrally, and mirror at least some of it (e.g., sequences are mirrored between GenBank, EMBL, and DDBJ). Hence, issues of synchronisation are largely limited to these big databases, and hence manageable. My sense is that there is a lot of computer science research on federated databases, but few actual large-scale systems that people make regular use of. What is present in bioinformatics are large numbers of "value added" databases that take GenBank, PubMed, etc. and do neat things with them. This is possible because you can download the entire database. Each one of these value added databases does need to deal with the issue of what happens when GenBank (say) changes, but because GenBank has well defined releases, essentially they can grab a new copy of the data, update their local copy, and regenerate their database. Having access to all the data makes all kinds of things possible which are harder to do if the data is distributed. I'd argue that part of the success of bioinformatics is because of data availability. Hence, my own view (at least today) is: 1. Individual data providers manage their own data, and also make available their data in the following ways: i) provide GUIDs and metadata (e.g., LSIDs) ii) provide a basic, standard search web service iii) provide their own web interface iv) periodically provide complete dump of data 2. GBIF (or some equivalent) takes on the job of harvesting all providers, building a warehouse, and making that data available through i) web interface ii) web services iii) complete data dump 3. Researchers can decide how best to make use of this data. They may wish to get a complete "GBIF" dump and install that locally, or query GBIF, or they may wish to query the individual providers for the most up to date information, or some mixture of this. Issues of synchronisation are dealt with by GBIF and its providers, which I think essentially amounts to having versioning and release numbers (but I'm probably being naive). Probably 1iv and 3iii are going to cause some issues, and this is off topic, but if bioinformatics is anything to go by, if we don't make our data available in bulk we are tying our own hands. However, this is obviously something each individual provider will have to decide upon themselves. My other feeling is that from the point of view of end users (and I class myself as one) the real game will be services, especially search (think "Google Life"). And my feeling is that this won't work if queries are done in a distributed fashion -- the Web is supremely distributed, but Google doesn't query the Web, it queries its local copy. In summary, I think the issue raised by Rich is important, but is one to be addressed by whoever takes on the task of assembling a data warehouse from the individual providers. Of course, once providers make their data available, anybody can do this... Regards Rod ------------------------------------------------------------------------ ---------------------------------------- Professor Roderic D. M. Page Editor, Systematic Biology DEEB, IBLS Graham Kerr Building University of Glasgow Glasgow G12 8QP United Kingdom Phone: +44 141 330 4778 Fax: +44 141 330 2792 email: r.page(a)bio.gla.ac.uk web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html Subscribe to Systematic Biology through the Society of Systematic Biologists Website: http://systematicbiology.org Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ Find out what we know about a species at http://ispecies.org

1 0

Re: RDF query and inference in a distributed environment
by Blum, Stan 04 Jan '06

04 Jan '06

Robert Huber wrote: It would be very interesting to me whether OAI ever was discussed in the TDWG or for use in GBIF? --------------------- Robert, Back in the 90's issues of "control" (integrity and availability, even down to classes of users) and attribution really dominated discussions. Many institutions wanted assurance that users would understand who created, or at least digitized, and maintained the data. Oposing schools of thought emerged, the "data hoarders" versus the "data anarchists". Given that context, several forays into data integration (Species Analyst, REMIB, AVH, etc.) prior to DiGIR and BioCASe all assumed a requirement for distributed queries (and later indexing), rather than harvesting and wholesale caching. We were aware of OAI by 2000, and it was even implemented in vPlants project http://www.vplants.org/, but the rest of us avoided that approach because we fully expected it to be much more difficult to sell "let us serve your data for you". The community is STILL very wary of empire building. Regarding Rod Page's contrast between the approaches in biodiversity- and (molecular) bio-informatics: I think one of the drivers behind the two approaches that is that (lab-)bench scientists tend to view sequence data as observation-events and relatively fixed once they are submitted; whereas collections people tend to view specimens (and maybe even classifications) as evolving entities that must be updated. We have a longer view of the specimens and the data about them. Having said that, there is no doubt that building large warehouses is a simpler and better performing architecture for integration. We just need to address data replication explicitly and deliberately, in a heterogeneous environment where "subscribers" can go offline (i.e., a database transaction oriented approach would be too stringent). A GUID system that incorporates versioning should help support data replication. Open and standards-based protocols appear to be very effective in calming fears about empire buiding. -Stan Stanley D. Blum, Ph.D. Research Information Manager California Academy of Sciences 875 Howard St. San Francisco, CA +1 (415) 321-8183

1 0

Re: RDF query and inference in a distributed environment
by Roger Hyam 04 Jan '06

04 Jan '06

Dear all, There have been some interesting ideas put forward in this thread. The future is bright but nearly all the proposed mechanisms rely on some basic things we don't have yet. Even if we are only considering collecting data from a hand full of data sources and caching them till we go for a coffee break: 1. Heterogeneous data from multiple sources must fit into a single data model (our cache's data model). 2. New data that we have never come across before needs to fit in the same model. 3. Data from different but overlapping domains needs to be linked semantically. 4. Real world objects need GUIDs so we can link them. To achieve this pick 'n' mix approach to data we need it all to follow a single low level schema. An RDF approach, backed up with a ontology in OWL or some other technology, is the strongest contender for being able to do this. We haven't done this yet so the only way to really join data together is in a big data warehouse like GBIF where Donald and his team have to work hard keeping track of the different data standards and mapping them onto their own schema that we can then search. Rod's recent post also gives a good example of the work involved with doing this kind of thing for TreeBASE. I am not saying warehouses will go away with a more RDF like approach but it will become less onerous to collect pools of it to work with. Roger Roderic Page wrote: > I wonder whether at some point we need to think carefully about why we > have a distributed model in the first place. Is it the best choice? Did > we chose it, or has it emerged primarily because data providers > want/need to keep control over "their" data? > > In the context of bioinformatics, I'm not aware of any large scale > distributed environments that actually work (probably ignorance on my > part). The largest databases (GenBank, PubMed, EMBL, DDBJ) have all the > data stored centrally, and mirror at least some of it (e.g., sequences > are mirrored between GenBank, EMBL, and DDBJ). Hence, issues of > synchronisation are largely limited to these big databases, and hence > manageable. > > My sense is that there is a lot of computer science research on > federated databases, but few actual large-scale systems that people > make regular use of. > > What is present in bioinformatics are large numbers of "value added" > databases that take GenBank, PubMed, etc. and do neat things with > them. This is possible because you can download the entire database. > Each one of these value added databases does need to deal with the > issue of what happens when GenBank (say) changes, but because GenBank > has well defined releases, essentially they can grab a new copy of the > data, update their local copy, and regenerate their database. > > Having access to all the data makes all kinds of things possible which > are harder to do if the data is distributed. I'd argue that part of the > success of bioinformatics is because of data availability. > > Hence, my own view (at least today) is: > > 1. Individual data providers manage their own data, and also make > available their data in the following ways: > i) provide GUIDs and metadata (e.g., LSIDs) > ii) provide a basic, standard search web service > iii) provide their own web interface > iv) periodically provide complete dump of data > > 2. GBIF (or some equivalent) takes on the job of harvesting all > providers, building a warehouse, and making that data available through > i) web interface > ii) web services > iii) complete data dump > > 3. Researchers can decide how best to make use of this data. They may > wish to get a complete "GBIF" dump and install that locally, or query > GBIF, or they may wish to query the individual providers for the most > up to date information, or some mixture of this. > > Issues of synchronisation are dealt with by GBIF and its providers, > which I think essentially amounts to having versioning and release > numbers (but I'm probably being naive). > > Probably 1iv and 3iii are going to cause some issues, and this is off > topic, but if bioinformatics is anything to go by, if we don't make our > data available in bulk we are tying our own hands. However, this is > obviously something each individual provider will have to decide upon > themselves. > > My other feeling is that from the point of view of end users (and I > class myself as one) the real game will be services, especially search > (think "Google Life"). And my feeling is that this won't work if > queries are done in a distributed fashion -- the Web is supremely > distributed, but Google doesn't query the Web, it queries its local > copy. > > In summary, I think the issue raised by Rich is important, but is one > to be addressed by whoever takes on the task of assembling a data > warehouse from the individual providers. Of course, once providers make > their data available, anybody can do this... > > Regards > > Rod > > > ------------------------------------------------------------------------ > ---------------------------------------- > Professor Roderic D. M. Page > Editor, Systematic Biology > DEEB, IBLS > Graham Kerr Building > University of Glasgow > Glasgow G12 8QP > United Kingdom > > Phone: +44 141 330 4778 > Fax: +44 141 330 2792 > email: r.page(a)bio.gla.ac.uk > web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html > reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html > > Subscribe to Systematic Biology through the Society of Systematic > Biologists Website: http://systematicbiology.org > Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ > Find out what we know about a species at http://ispecies.org > -- ------------------------------------- Roger Hyam Technical Architect Taxonomic Databases Working Group ------------------------------------- http://www.tdwg.org roger(a)tdwg.org +44 1578 722782 -------------------------------------

1 0

Re: RDF query and inference in a distributed environment
by Arthur Chapman 04 Jan '06

04 Jan '06

The problem with searching GBIF for the Index and then searching for each individual record makes it almost impossible to do any medium or large-scale data integration. The inclusion of more information within the "Index" in cached form, would allow for a greater range of studies to be conducted that involve large-scale data integration. For example species modelling can be done with name, latitude/longitude, etc., but a study of a particular collector could not be done if the collector's name were not included, etc. So the list of 4(5) methods for data query is not just that simple - as there are data that will fit into more than one category - some can be warehoused/mirrored, whereas other data may not be and only be searched via a second query to the source. One issue that needs to be discussed is what data needs/should to be included in the cache/mirrored warehouses etc. and what is not necessary to be so stored (at this stage). But we are getting off track on the discussion of technical issues wrt to GUIDs etc. Whatever method is used, we will need GUIDs of some sort and this discussion has been very interesting and informative. It has taken many years to bring the biological collection community this far (and still many institutions are reluctant to make their data available via GBIF) - to even propose that ALL their data be warehoused in mirror sites around the world would, I fear, cause many to withdraw even the data they are making available now. As desirable as I believe it is to make as much of these data freely available as possible, we have to be careful or we may lose much of which we have already gained. It is a political/social issue, and as such is a very difficult one to negotiate. Cheers Arthur D. Chapman >>>From Kevin Richards <RichardsK(a)LANDCARERESEARCH.CO.NZ> on 4 Jan 2006: > Coming from an IT background rather than a taxonomic background, I have > = > never understood the strong "ownership" of data that people/scientists > h= > ave for their data. This seems rather short sited to me - people with > t= > hese concerns must have some thought about how to maintain/expose/use > "t= > heir" data in the long term future? I can understand their concerns, > bu= > t there must be a solution, otherwise their data will be no-ones > concern= > in 50 years time when it disappears from existence. > > Another thought I had about data caching systems. Say you want to > searc= > h the cached/centralised copy of the data (eg a GBIF cache). A list of > = > results is returned, then you decide you want to view more details of > on= > e of the results, so you follow a link off to the associated data (this > = > would theoretically be by using the GUID system we are discussing). > Thi= > s would result in viewing the details of the selected record at the > loca= > tion where the GUID resolves to - this would always be the same > location= > as a GUID only resolves to a single location. Is this correct, or > woul= > d the intention here be to view the cached details of the selected > recor= > d (which would require an separate ID for all the cached records)? Its > = > this navigation through the caches/repositories that I am not quite > sure= > of how it will work? > > Kevin > > >>> deepreef(a)BISHOPMUSEUM.ORG 5/01/2006 8:22 a.m. >>> > > Wow! Lots of resposes this morning. Many thanks to all who responded! > S= > ome > replies: > > Patricia wrote: > > > I agree with you about the logic in this. However accoding to my daily > > experience with potential dataproviders there is a lot of teaching and > > convincing needed to make this logic accepted that this does not > resul= > t > > in the loss of control over own data. I agree that to be conviencing > > a robust syncronization is needed. > > I understand the psychology, and agree that the barrier for "expose your > data online so that everyone can access it" strikes a more palatable > cho= > rd > than "expose your data online so that everyone can mirror it". > > > I agree that for machines and storage it is not that expensive. I was > > more referring to the human ressources needed to manage the > > mirror. Smaller institutions do not have necessary the funds or cannot > > justify to their hierachy that staff is devoting time to maint ain a > > full miror containing mainly "references" to information coming from > > other institutions, but it is easier to justify the time spent to > > contribute to the whole with the part concerning directly the > institut= > ion > ... > > Agreed -- but I wonder really how much more time would be involved in > setting up an automated mirror, compared with, e.g., a DiGIR provider. > = > In > my earlier post when I said that barriers would likely be technical, I > didn't mean that technology doesn't exist (it does). Rather, I meant > th= > at > in order to be successful, setting up and maintaining a mirror would > hav= > e to > be no more technically challenging than establishing a DiGIR provider. > = > I'm > not sure that's possible (yet). > > > Yes I agree with you that rob ust syncronization will be needed but > > as my IT colleague always remind me, I guess we must not forget that > > setting up an IT infrastructure is most of the time 10 % technical > > issues to be solved and 90% of the time solving "human problems > > and barriers" to make it work and accepted ... > > I agree -- but part of overcoming the human problems and barriers is to > = > make > setting up a mirror site relatively easy for a non-specialist IT person. > And this is really a technical challenge. The other human problems & > barriers (IPR, allowing access to hard-earned data, confidence that data > edit authorization rules will be enforced, etc.) also have some > technica= > l > foundation, and also exist for the distributed/complementary approach. > > Ricardo wrote: > > > Instead of mutually exclusive, both approaches you mention are > > complementary. The "distributed complementary data" aproach is a > > fundamental part of the infrastructure necessary to build the > > "distributed mirror copies" you propose. That (the "distributed > > complementary data" approach) is essentially what we have in place now > > with DiGIR/BioCase/Tapir as the harvesting protocol and GBIF and other > > institutions as harvesters. The only missing piece of software we need > > to really have "an automated and robust synchronization protocol", in > = > my > > opinion is some kind of push mechanism to trigger updates in the > cache= > s. > > Agreed on all points. > > > However, I think it is not very useful to try to standardize how > t= > he > > distributed mirror copies should be built and organized. > === message truncated ===

1 0

Re: RDF query and inference in a distributed environment
by Sally Hinchcliffe 04 Jan '06

04 Jan '06

Species 2000 uses a federated data model. As to whether that counts as 'large scale' or not I don't know. I believe that even getting permission for a data cache from the various providers was a painful exercise. Sally > I wonder whether at some point we need to think carefully about why we > have a distributed model in the first place. Is it the best choice? Did > we chose it, or has it emerged primarily because data providers > want/need to keep control over "their" data? > > In the context of bioinformatics, I'm not aware of any large scale > distributed environments that actually work (probably ignorance on my > part). The largest databases (GenBank, PubMed, EMBL, DDBJ) have all the > data stored centrally, and mirror at least some of it (e.g., sequences > are mirrored between GenBank, EMBL, and DDBJ). Hence, issues of > synchronisation are largely limited to these big databases, and hence > manageable. > > My sense is that there is a lot of computer science research on > federated databases, but few actual large-scale systems that people > make regular use of. > > What is present in bioinformatics are large numbers of "value added" > databases that take GenBank, PubMed, etc. and do neat things with > them. This is possible because you can download the entire database. > Each one of these value added databases does need to deal with the > issue of what happens when GenBank (say) changes, but because GenBank > has well defined releases, essentially they can grab a new copy of the > data, update their local copy, and regenerate their database. > > Having access to all the data makes all kinds of things possible which > are harder to do if the data is distributed. I'd argue that part of the > success of bioinformatics is because of data availability. > > Hence, my own view (at least today) is: > > 1. Individual data providers manage their own data, and also make > available their data in the following ways: > i) provide GUIDs and metadata (e.g., LSIDs) > ii) provide a basic, standard search web service > iii) provide their own web interface > iv) periodically provide complete dump of data > > 2. GBIF (or some equivalent) takes on the job of harvesting all > providers, building a warehouse, and making that data available through > i) web interface > ii) web services > iii) complete data dump > > 3. Researchers can decide how best to make use of this data. They may > wish to get a complete "GBIF" dump and install that locally, or query > GBIF, or they may wish to query the individual providers for the most > up to date information, or some mixture of this. > > Issues of synchronisation are dealt with by GBIF and its providers, > which I think essentially amounts to having versioning and release > numbers (but I'm probably being naive). > > Probably 1iv and 3iii are going to cause some issues, and this is off > topic, but if bioinformatics is anything to go by, if we don't make our > data available in bulk we are tying our own hands. However, this is > obviously something each individual provider will have to decide upon > themselves. > > My other feeling is that from the point of view of end users (and I > class myself as one) the real game will be services, especially search > (think "Google Life"). And my feeling is that this won't work if > queries are done in a distributed fashion -- the Web is supremely > distributed, but Google doesn't query the Web, it queries its local > copy. > > In summary, I think the issue raised by Rich is important, but is one > to be addressed by whoever takes on the task of assembling a data > warehouse from the individual providers. Of course, once providers make > their data available, anybody can do this... > > Regards > > Rod > > > ------------------------------------------------------------------------ > ---------------------------------------- > Professor Roderic D. M. Page > Editor, Systematic Biology > DEEB, IBLS > Graham Kerr Building > University of Glasgow > Glasgow G12 8QP > United Kingdom > > Phone: +44 141 330 4778 > Fax: +44 141 330 2792 > email: r.page(a)bio.gla.ac.uk > web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html > reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html > > Subscribe to Systematic Biology through the Society of Systematic > Biologists Website: http://systematicbiology.org > Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ > Find out what we know about a species at http://ispecies.org *** Sally Hinchcliffe *** Computer section, Royal Botanic Gardens, Kew *** tel: +44 (0)20 8332 5708 *** S.Hinchcliffe(a)rbgkew.org.uk

1 0

Re: RDF query and inference in a distributed environment
by Roderic Page 04 Jan '06

04 Jan '06

I wonder whether at some point we need to think carefully about why we have a distributed model in the first place. Is it the best choice? Did we chose it, or has it emerged primarily because data providers want/need to keep control over "their" data? In the context of bioinformatics, I'm not aware of any large scale distributed environments that actually work (probably ignorance on my part). The largest databases (GenBank, PubMed, EMBL, DDBJ) have all the data stored centrally, and mirror at least some of it (e.g., sequences are mirrored between GenBank, EMBL, and DDBJ). Hence, issues of synchronisation are largely limited to these big databases, and hence manageable. My sense is that there is a lot of computer science research on federated databases, but few actual large-scale systems that people make regular use of. What is present in bioinformatics are large numbers of "value added" databases that take GenBank, PubMed, etc. and do neat things with them. This is possible because you can download the entire database. Each one of these value added databases does need to deal with the issue of what happens when GenBank (say) changes, but because GenBank has well defined releases, essentially they can grab a new copy of the data, update their local copy, and regenerate their database. Having access to all the data makes all kinds of things possible which are harder to do if the data is distributed. I'd argue that part of the success of bioinformatics is because of data availability. Hence, my own view (at least today) is: 1. Individual data providers manage their own data, and also make available their data in the following ways: i) provide GUIDs and metadata (e.g., LSIDs) ii) provide a basic, standard search web service iii) provide their own web interface iv) periodically provide complete dump of data 2. GBIF (or some equivalent) takes on the job of harvesting all providers, building a warehouse, and making that data available through i) web interface ii) web services iii) complete data dump 3. Researchers can decide how best to make use of this data. They may wish to get a complete "GBIF" dump and install that locally, or query GBIF, or they may wish to query the individual providers for the most up to date information, or some mixture of this. Issues of synchronisation are dealt with by GBIF and its providers, which I think essentially amounts to having versioning and release numbers (but I'm probably being naive). Probably 1iv and 3iii are going to cause some issues, and this is off topic, but if bioinformatics is anything to go by, if we don't make our data available in bulk we are tying our own hands. However, this is obviously something each individual provider will have to decide upon themselves. My other feeling is that from the point of view of end users (and I class myself as one) the real game will be services, especially search (think "Google Life"). And my feeling is that this won't work if queries are done in a distributed fashion -- the Web is supremely distributed, but Google doesn't query the Web, it queries its local copy. In summary, I think the issue raised by Rich is important, but is one to be addressed by whoever takes on the task of assembling a data warehouse from the individual providers. Of course, once providers make their data available, anybody can do this... Regards Rod ------------------------------------------------------------------------ ---------------------------------------- Professor Roderic D. M. Page Editor, Systematic Biology DEEB, IBLS Graham Kerr Building University of Glasgow Glasgow G12 8QP United Kingdom Phone: +44 141 330 4778 Fax: +44 141 330 2792 email: r.page(a)bio.gla.ac.uk web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html Subscribe to Systematic Biology through the Society of Systematic Biologists Website: http://systematicbiology.org Search for taxon names at http://darwin.zoology.gla.ac.uk/~rpage/portal/ Find out what we know about a species at http://ispecies.org

1 0