On 06/01/2011, at 1:36 PM, Peter DeVries wrote:Most groups now make their data available as an RDF dump which eliminated the need to crawl if you want to pull in a lot of data.We supply this functionality via an OAI-PMH service. The "Protocol for Metadata Harvesting" is a standard way of asking for a list of ids or a list of actual records, limited by an optional "set" specifier and by start/end modification dates. Essentially: "What records do you have?" and "What's changed since last thursday?"If you pass set specifier to our service, we treat it as the prefix of the LSIDs that you want to limit your search to, allowing data harvesting to be done in manageable chunks.Our implementation does not perform especially well (I think the XSLT engine is the bottleneck). But that aside, OAI-PMH seems to be a sensible way for data stores to maintain mirrors of each other's data. Even just the list of ids changed since X id useful - you can remove them from your cache, and load them via linked data when required.See:Please do not send ListIdentifiers or ListRecords requests to this service without specifying a set that includes at least one character of the objectid part of the lsid.It hasn't crashed for weeks now, and it would be nice to keep it that way for just a little longer <sob!>._______________________________________________If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. Please consider the environment before printing this email.