[Tdwg-guid] Throttling searches [ Scanned for viruses ]
Sally Hinchcliffe
S.Hinchcliffe at kew.org
Mon Jun 19 11:34:15 CEST 2006
How can we pass a token with an LSID?
>
> I think the only way to throttle in these situations is to have some
> notion of who the client is and the only way to do that is to have some
> kind of token exchange over HTTP saying who they are. Basically you have
> to have some kind of client registration system or you can never
> distinguish between a call from a new client and a repeat call. The use
> of IP address is a pain because so many people are now behind some kind
> of NAT gateway.
>
> How about this for a plan:
>
> You could give a degraded services to people who don't pass a token (a 5
> second delay perhaps) and offer a quicker service to registered users
> who pass a token (but then perhaps limit the number of calls they make).
> This would mean you could offer a universal service even to those with
> naive client software but a better service to those who play nicely. You
> could also get better stats on who is using the service.
>
> So there are ways that this could be done. I expect people will come up
> with a host of different ways. It is outside LSIDs though.
>
> Roger
>
> Sally Hinchcliffe wrote:
> > It's not an LSID issue per se, but LSIDs will make it harder to slow
> > searches down. For instance, Google restricts use of its spell
> > checker to 1000 a day by use of a key which is passed in with each
> > request. Obviously this can't be done with LSIDs as then they
> > wouldn't be the same for each user.
> > The other reason why it's relevant to LSIDs is simply that providing
> > a list of all relevant IPNI LSIDs (not necessary to the LSID
> > implementation but a nice to have for caching / lookups for other
> > systems using our LSIDs) also makes life easier for the datascrapers
> > to operate
> >
> > Also I thought ... here's a list full of clever people perhaps they
> > will have some suggestions
> >
> > Sally
> >
> >
> >> Is this an LSID issue? LSIDs essential provide a binding service between
> >> an name and one or more web services (we default to HTTP GET bindings).
> >> It isn't really up to the LSID authority to administer any policies
> >> regarding the web service but simply to point at it. It is up to the web
> >> service to do things like throttling, authentication and authorization.
> >>
> >> Imagine, for example, that the different services had different
> >> policies. It may be reasonable not to restrict the getMetadata() calls
> >> but to restrict the getData() calls.
> >>
> >> The use of LSIDs does not create any new problems that weren't there
> >> with web page scraping - or scraping of any other web service.
> >>
> >> Just my thoughts...
> >>
> >> Roger
> >>
> >>
> >> Ricardo Scachetti Pereira wrote:
> >>
> >>> Sally,
> >>>
> >>> You raised a really important issue that we had not really addressed
> >>> at the meeting. Thanks for that.
> >>>
> >>> I would say that we should not constrain the resolution of LSIDs if
> >>> we expect our LSID infrastructure to work. LSIDs will be the basis of
> >>> our architecture so we better have good support for that.
> >>>
> >>> However, that is sure a limiting factor. Also server efficiency will
> >>> likely vary quite a lot, depending on underlying system optimizations
> >>> and all.
> >>>
> >>> So I think that the solution for this problem is in caching LSID
> >>> responses on the server LSID stack. Basically, after resolving an LSID
> >>> once, your server should be able to resolve it again and again really
> >>> quickly, until something on the metadata that is related to that id changes.
> >>>
> >>> I haven't looked at this aspect of the LSID software stack, but
> >>> maybe others can say something about it. In any case I'll do some
> >>> research on it and get back to you.
> >>>
> >>> Again, thanks for bringing it up.
> >>>
> >>> Cheers,
> >>>
> >>> Ricardo
> >>>
> >>>
> >>> Sally Hinchcliffe wrote:
> >>>
> >>>
> >>>> There are enough discontinuities in IPNI ids that 1,2,3 would quickly
> >>>> run into the sand. I agree it's not a new problem - I just hate to
> >>>> think I'm making life easier for the data scrapers
> >>>> Sally
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> It can be a problem but I'm not sure if there is a simple solution ... and how different is the LSID crawler scenario from an http://www.ipni.org/ipni/plantsearch?id= 1,2,3,4,5 ... 9999999 scenario?
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: tdwg-guid-bounces at mailman.nhm.ku.edu
> >>>>> [mailto:tdwg-guid-bounces at mailman.nhm.ku.edu]On Behalf Of Sally
> >>>>> Hinchcliffe
> >>>>> Sent: 15 June 2006 12:08
> >>>>> To: tdwg-guid at mailman.nhm.ku.edu
> >>>>> Subject: [Tdwg-guid] Throttling searches [ Scanned for viruses ]
> >>>>>
> >>>>>
> >>>>> Hi all
> >>>>> another question that has come up here.
> >>>>>
> >>>>> As discussed at the meeting, we're thinking of providing a complete
> >>>>> download of all IPNI LSIDs plus a label (name and author, probably)
> >>>>> which will be available as an annually produced download
> >>>>>
> >>>>> Most people will play nice and just resolve one or two LSIDs as
> >>>>> required, but by providing a complete list, we're making it very easy
> >>>>> for someone to write a crawler that hits every LSID in turn and
> >>>>> basically brings our server to its knees
> >>>>>
> >>>>> Anybody know of a good way of enforcing more polite behaviour? We can
> >>>>> make the download only available under a data supply agreement that
> >>>>> includes a clause limiting hit rates, or we could limit by IP address
> >>>>> (but this would ultimately block out services like Rod's simple
> >>>>> resolver). I beleive Google's spell checker uses a key which has to
> >>>>> be passed in as part of the query - obviously we can't do that with
> >>>>> LSIDs
> >>>>>
> >>>>> Any thoughts? Anyone think this is a problem?
> >>>>>
> >>>>> Sally
> >>>>> *** Sally Hinchcliffe
> >>>>> *** Computer section, Royal Botanic Gardens, Kew
> >>>>> *** tel: +44 (0)20 8332 5708
> >>>>> *** S.Hinchcliffe at rbgkew.org.uk
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> TDWG-GUID mailing list
> >>>>> TDWG-GUID at mailman.nhm.ku.edu
> >>>>> http://mailman.nhm.ku.edu/mailman/listinfo/tdwg-guid
> >>>>>
> >>>>> _______________________________________________
> >>>>> TDWG-GUID mailing list
> >>>>> TDWG-GUID at mailman.nhm.ku.edu
> >>>>> http://mailman.nhm.ku.edu/mailman/listinfo/tdwg-guid
> >>>>>
> >>>>>
> >>>>>
> >>>> *** Sally Hinchcliffe
> >>>> *** Computer section, Royal Botanic Gardens, Kew
> >>>> *** tel: +44 (0)20 8332 5708
> >>>> *** S.Hinchcliffe at rbgkew.org.uk
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> TDWG-GUID mailing list
> >>>> TDWG-GUID at mailman.nhm.ku.edu
> >>>> http://mailman.nhm.ku.edu/mailman/listinfo/tdwg-guid
> >>>>
> >>>>
> >>>>
> >>>>
> >>> _______________________________________________
> >>> TDWG-GUID mailing list
> >>> TDWG-GUID at mailman.nhm.ku.edu
> >>> http://mailman.nhm.ku.edu/mailman/listinfo/tdwg-guid
> >>>
> >>>
> >>>
> >> --
> >>
> >> -------------------------------------
> >> Roger Hyam
> >> Technical Architect
> >> Taxonomic Databases Working Group
> >> -------------------------------------
> >> http://www.tdwg.org
> >> roger at tdwg.org
> >> +44 1578 722782
> >> -------------------------------------
> >>
> >>
> >>
> >
> > *** Sally Hinchcliffe
> > *** Computer section, Royal Botanic Gardens, Kew
> > *** tel: +44 (0)20 8332 5708
> > *** S.Hinchcliffe at rbgkew.org.uk
> >
> >
> >
>
>
> --
>
> -------------------------------------
> Roger Hyam
> Technical Architect
> Taxonomic Databases Working Group
> -------------------------------------
> http://www.tdwg.org
> roger at tdwg.org
> +44 1578 722782
> -------------------------------------
>
>
*** Sally Hinchcliffe
*** Computer section, Royal Botanic Gardens, Kew
*** tel: +44 (0)20 8332 5708
*** S.Hinchcliffe at rbgkew.org.uk
More information about the tdwg-tag
mailing list