Hi Everyone -
Javier and I chatted yesterday about unresolved technical issues. Our primary goal was simplicity. Here are the outcomes. Feel free to suggest something different, if either i) you're planning on developing around your suggestion; or ii) you believe that to not follow your suggestion would be a grave mistake.
NOTE: by "app" we mean anything that is writing to the table - could be a smartphone app, could be a web form, could be a screenscraper or tweet-parser.
1. All apps will write to the same table. Each app will need a Google account, which will be given write-access to the table.
2. The occurrence_id will be the row_id assigned by Fusion Tables. This is not seen in the web interface, but is available from the API.
3. To support crowdsourcing of image identification, alternative identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these columns will also exist in the Occurrences table. When the observation is first reported, any identification will be entered in the Occurrences table, with subsequent identifications in the Identifications table. (The main motivation for keeping the initial identification in the Occurrences table is to ease the process for app developers.)
4. Multiple multimedia URLs will be listed in a single column, comma separated.
5. The lat and long columns will be WGS84. If app developers wish, they are welcome to build support for additional datums into their apps, since the apps will have authority to create additional columns. If this doesn't happen, anyone planning to use another datum will need to announce themselves ahead of time, and we will create appropriate columns for them.
6. The table will contain Kingdom, Phylum, Class, Order, and Family columns. Javier will write software to resolve identifications to the Catalog of Life, and to populate the taxonomy columns. If someones want to resolve observations to a different classification, they are free to do so, and are encouraged to publish the results in Fusion Tables.
7. In addition to the identification table, there is scope for creating an Annotations table and front-end, should anyone wish to tackle this.
Here's a timetable that would be nice to adhere to:
Sept. 1: Final versions of the Occurrences and Identifications tables published to Fusion Tables. Write access will only be granted to developers. These will be the test tables. The day before the bioblitz, they will become the development tables, and test records will be expunged.
Sept. 3: Table documentation and sample code for writing to the tables.
Sept. 8: Taxanomic resolution, validation, assignment of taxon GUIDS/LSIDs.
Sept. 15: At least one Image identification framework in place.
Sept. 15 - Sept. 28: Testing and work on visualization and other data-oriented services (e.g. publication as DwC or Linked Data, etc.)
Best to all - Joel.
Joel,
Based on our discussion with David R. and the phone conference 2 weeks back you where thinking everyone can write to their own tables then we merge in the data into one master table. I wrote a script to do this but have not tested it out. It is something we can do if you have more then 2 sample tables with data. This way anyone that shares their table with userX will get harvested and that data will be loaded into the master readonly table that is reloaded over x minutes. And if there are custom tables that are not part of your default table standards we can possibly setup a unique harvester during the conference to transform to the correct column model. This way everyone is independent but at the same time able to view the master data every 20 minutes or whatever update time we set it to. We can do the same for different table types.
Any questions just let me know or contact me directly.
Michael Giddens Office: 225-238-1879 skype: mikegiddens
On 8/26/2010 11:45 AM, joel sachs wrote:
Hi Everyone -
Javier and I chatted yesterday about unresolved technical issues. Our primary goal was simplicity. Here are the outcomes. Feel free to suggest something different, if either i) you're planning on developing around your suggestion; or ii) you believe that to not follow your suggestion would be a grave mistake.
NOTE: by "app" we mean anything that is writing to the table - could be a smartphone app, could be a web form, could be a screenscraper or tweet-parser.
- All apps will write to the same table. Each app will need a Google
account, which will be given write-access to the table.
- The occurrence_id will be the row_id assigned by Fusion Tables. This is
not seen in the web interface, but is available from the API.
- To support crowdsourcing of image identification, alternative
identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these columns will also exist in the Occurrences table. When the observation is first reported, any identification will be entered in the Occurrences table, with subsequent identifications in the Identifications table. (The main motivation for keeping the initial identification in the Occurrences table is to ease the process for app developers.)
- Multiple multimedia URLs will be listed in a single column, comma
separated.
- The lat and long columns will be WGS84. If app developers wish, they
are welcome to build support for additional datums into their apps, since the apps will have authority to create additional columns. If this doesn't happen, anyone planning to use another datum will need to announce themselves ahead of time, and we will create appropriate columns for them.
- The table will contain Kingdom, Phylum, Class, Order, and Family
columns. Javier will write software to resolve identifications to the Catalog of Life, and to populate the taxonomy columns. If someones want to resolve observations to a different classification, they are free to do so, and are encouraged to publish the results in Fusion Tables.
- In addition to the identification table, there is scope for creating an
Annotations table and front-end, should anyone wish to tackle this.
Here's a timetable that would be nice to adhere to:
Sept. 1: Final versions of the Occurrences and Identifications tables published to Fusion Tables. Write access will only be granted to developers. These will be the test tables. The day before the bioblitz, they will become the development tables, and test records will be expunged.
Sept. 3: Table documentation and sample code for writing to the tables.
Sept. 8: Taxanomic resolution, validation, assignment of taxon GUIDS/LSIDs.
Sept. 15: At least one Image identification framework in place.
Sept. 15 - Sept. 28: Testing and work on visualization and other data-oriented services (e.g. publication as DwC or Linked Data, etc.)
Best to all - Joel.
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
Hi Michael,
The motivations for using one single table are to make things easier considering the little time we have to test things. That being said I see the following advantages
1) Simplicity. Not having to merge tables will reduce the problems pretty much, plus we will need less coordination asking people which tables we have to merge and so on.
2) No lack of flexibility, we are not thousands of developers. I dont see why everybody can not add whatever fields they want to the master table. Plus before adding if they ask, we might have it already and we reduce the work needed for doing mappings. We can have one single wiki page describing every field on the master table.
3) We get live data, no need for 20min refresh periods. We can create apps that visualize in real time how the data get entered. One of the interesting visualizations is to see things going on in real time.
But again, the biggest reason to me is to have to avoid merges of multiples tables, one per application. I prefer if we dont need to do integration with such little time ahead.
What do you see as the biggest problems for using one single table for everybody?
In any case. A combination of both can also work I have to say. Actually if someone uses his own fusion table that will be the same scenario as scrapping image from the EOL Flickr group.
Javier de la Torre www.vizzuality.com
On Aug 26, 2010, at 6:58 PM, Michael Giddens wrote:
Joel,
Based on our discussion with David R. and the phone conference 2 weeks back you where thinking everyone can write to their own tables then we merge in the data into one master table. I wrote a script to do this but have not tested it out. It is something we can do if you have more then 2 sample tables with data. This way anyone that shares their table with userX will get harvested and that data will be loaded into the master readonly table that is reloaded over x minutes. And if there are custom tables that are not part of your default table standards we can possibly setup a unique harvester during the conference to transform to the correct column model. This way everyone is independent but at the same time able to view the master data every 20 minutes or whatever update time we set it to. We can do the same for different table types.
Any questions just let me know or contact me directly.
Michael Giddens Office: 225-238-1879 skype: mikegiddens
On 8/26/2010 11:45 AM, joel sachs wrote:
Hi Everyone -
Javier and I chatted yesterday about unresolved technical issues. Our primary goal was simplicity. Here are the outcomes. Feel free to suggest something different, if either i) you're planning on developing around your suggestion; or ii) you believe that to not follow your suggestion would be a grave mistake.
NOTE: by "app" we mean anything that is writing to the table - could be a smartphone app, could be a web form, could be a screenscraper or tweet-parser.
- All apps will write to the same table. Each app will need a Google
account, which will be given write-access to the table.
- The occurrence_id will be the row_id assigned by Fusion Tables. This is
not seen in the web interface, but is available from the API.
- To support crowdsourcing of image identification, alternative
identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these columns will also exist in the Occurrences table. When the observation is first reported, any identification will be entered in the Occurrences table, with subsequent identifications in the Identifications table. (The main motivation for keeping the initial identification in the Occurrences table is to ease the process for app developers.)
- Multiple multimedia URLs will be listed in a single column, comma
separated.
- The lat and long columns will be WGS84. If app developers wish, they
are welcome to build support for additional datums into their apps, since the apps will have authority to create additional columns. If this doesn't happen, anyone planning to use another datum will need to announce themselves ahead of time, and we will create appropriate columns for them.
- The table will contain Kingdom, Phylum, Class, Order, and Family
columns. Javier will write software to resolve identifications to the Catalog of Life, and to populate the taxonomy columns. If someones want to resolve observations to a different classification, they are free to do so, and are encouraged to publish the results in Fusion Tables.
- In addition to the identification table, there is scope for creating an
Annotations table and front-end, should anyone wish to tackle this.
Here's a timetable that would be nice to adhere to:
Sept. 1: Final versions of the Occurrences and Identifications tables published to Fusion Tables. Write access will only be granted to developers. These will be the test tables. The day before the bioblitz, they will become the development tables, and test records will be expunged.
Sept. 3: Table documentation and sample code for writing to the tables.
Sept. 8: Taxanomic resolution, validation, assignment of taxon GUIDS/LSIDs.
Sept. 15: At least one Image identification framework in place.
Sept. 15 - Sept. 28: Testing and work on visualization and other data-oriented services (e.g. publication as DwC or Linked Data, etc.)
Best to all - Joel.
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
To clarify: My understanding of "everyone writes to their own table" doesn't mean "each app writes to its own table" (which gives us nothing), but, rather, "each instance of each app writes to its own table" which provides additional security to the data, and more reliable provenance.
Here's how this could work: i. Before a field session, the app says to the user "Do you want to create a new table? If so, enter your Google id and password." Then the app creates the table, and shares it with the Google account of the data_merge_script.
ii. The merged table, like the Identifications table, will have columns "table_id" and "row_id" for identifying observations.
If one of the apps can provide (i), and Michael can provide (ii), that would be great (and was my first choice). But given our aggressive deadline of everything working by 9/15, I think we should allow developers to write to a common table.
(Michael, to test your script can you just make a bunch of copies of the table, and populate them with data from the EOL Flickr group (using Javier's scraper) or with junk? Or do you need something more?)
A new twist: We're talking with the Fusion Table folks this afternoon, and may end up as beta-testers for new functionality that allows row-level authentication within a table. Stay posted.
Joel.
On Thu, 26 Aug 2010, Javier de la Torre wrote:
Hi Michael,
The motivations for using one single table are to make things easier considering the little time we have to test things. That being said I see the following advantages
Simplicity. Not having to merge tables will reduce the problems pretty much, plus we will need less coordination asking people which tables we have to merge and so on.
No lack of flexibility, we are not thousands of developers. I dont see why everybody can not add whatever fields they want to the master table. Plus before adding if they ask, we might have it already and we reduce the work needed for doing mappings. We can have one single wiki page describing every field on the master table.
We get live data, no need for 20min refresh periods. We can create apps that visualize in real time how the data get entered. One of the interesting visualizations is to see things going on in real time.
But again, the biggest reason to me is to have to avoid merges of multiples tables, one per application. I prefer if we dont need to do integration with such little time ahead.
What do you see as the biggest problems for using one single table for everybody?
In any case. A combination of both can also work I have to say. Actually if someone uses his own fusion table that will be the same scenario as scrapping image from the EOL Flickr group.
Javier de la Torre www.vizzuality.com
On Aug 26, 2010, at 6:58 PM, Michael Giddens wrote:
Joel,
Based on our discussion with David R. and the phone conference 2 weeks back you where thinking everyone can write to their own tables then we merge in the data into one master table. I wrote a script to do this but have not tested it out. It is something we can do if you have more then 2 sample tables with data. This way anyone that shares their table with userX will get harvested and that data will be loaded into the master readonly table that is reloaded over x minutes. And if there are custom tables that are not part of your default table standards we can possibly setup a unique harvester during the conference to transform to the correct column model. This way everyone is independent but at the same time able to view the master data every 20 minutes or whatever update time we set it to. We can do the same for different table types.
Any questions just let me know or contact me directly.
Michael Giddens Office: 225-238-1879 skype: mikegiddens
On 8/26/2010 11:45 AM, joel sachs wrote:
Hi Everyone -
Javier and I chatted yesterday about unresolved technical issues. Our primary goal was simplicity. Here are the outcomes. Feel free to suggest something different, if either i) you're planning on developing around your suggestion; or ii) you believe that to not follow your suggestion would be a grave mistake.
NOTE: by "app" we mean anything that is writing to the table - could be a smartphone app, could be a web form, could be a screenscraper or tweet-parser.
- All apps will write to the same table. Each app will need a Google
account, which will be given write-access to the table.
- The occurrence_id will be the row_id assigned by Fusion Tables. This is
not seen in the web interface, but is available from the API.
- To support crowdsourcing of image identification, alternative
identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these columns will also exist in the Occurrences table. When the observation is first reported, any identification will be entered in the Occurrences table, with subsequent identifications in the Identifications table. (The main motivation for keeping the initial identification in the Occurrences table is to ease the process for app developers.)
- Multiple multimedia URLs will be listed in a single column, comma
separated.
- The lat and long columns will be WGS84. If app developers wish, they
are welcome to build support for additional datums into their apps, since the apps will have authority to create additional columns. If this doesn't happen, anyone planning to use another datum will need to announce themselves ahead of time, and we will create appropriate columns for them.
- The table will contain Kingdom, Phylum, Class, Order, and Family
columns. Javier will write software to resolve identifications to the Catalog of Life, and to populate the taxonomy columns. If someones want to resolve observations to a different classification, they are free to do so, and are encouraged to publish the results in Fusion Tables.
- In addition to the identification table, there is scope for creating an
Annotations table and front-end, should anyone wish to tackle this.
Here's a timetable that would be nice to adhere to:
Sept. 1: Final versions of the Occurrences and Identifications tables published to Fusion Tables. Write access will only be granted to developers. These will be the test tables. The day before the bioblitz, they will become the development tables, and test records will be expunged.
Sept. 3: Table documentation and sample code for writing to the tables.
Sept. 8: Taxanomic resolution, validation, assignment of taxon GUIDS/LSIDs.
Sept. 15: At least one Image identification framework in place.
Sept. 15 - Sept. 28: Testing and work on visualization and other data-oriented services (e.g. publication as DwC or Linked Data, etc.)
Best to all - Joel.
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
Hi,
Well. I have expressed already my opinion so will not empathize more. But some issue I see with your described approach:
1) It is not possible, trough the API, to change the sharing permissions. So if you create a table with the user credentials from an app, it will not be possible to give access to a master program that collects data from multiple tables. In fact I think it is not possible to create a table public using the API. That would mean that you app will, apart of doing what you describe, tell the user to go to Fusion tables and use the UI to give access to someone. I think this is big problem and one reason enough to not go that route.
2) We will need kind of a catalog or a registry of all tables created from multiple apps in order to be able to merge them. And then the merging tool will need to be going potentially against 200 tables merging all of them every 20min, that might be too complicate or even against the fair terms of use of the API.
Javier.
On Aug 27, 2010, at 6:38 PM, joel sachs wrote:
To clarify: My understanding of "everyone writes to their own table" doesn't mean "each app writes to its own table" (which gives us nothing), but, rather, "each instance of each app writes to its own table" which provides additional security to the data, and more reliable provenance.
Here's how this could work: i. Before a field session, the app says to the user "Do you want to create a new table? If so, enter your Google id and password." Then the app creates the table, and shares it with the Google account of the data_merge_script.
ii. The merged table, like the Identifications table, will have columns "table_id" and "row_id" for identifying observations.
If one of the apps can provide (i), and Michael can provide (ii), that would be great (and was my first choice). But given our aggressive deadline of everything working by 9/15, I think we should allow developers to write to a common table.
(Michael, to test your script can you just make a bunch of copies of the table, and populate them with data from the EOL Flickr group (using Javier's scraper) or with junk? Or do you need something more?)
A new twist: We're talking with the Fusion Table folks this afternoon, and may end up as beta-testers for new functionality that allows row-level authentication within a table. Stay posted.
Joel.
On Thu, 26 Aug 2010, Javier de la Torre wrote:
Hi Michael,
The motivations for using one single table are to make things easier considering the little time we have to test things. That being said I see the following advantages
Simplicity. Not having to merge tables will reduce the problems pretty much, plus we will need less coordination asking people which tables we have to merge and so on.
No lack of flexibility, we are not thousands of developers. I dont see why everybody can not add whatever fields they want to the master table. Plus before adding if they ask, we might have it already and we reduce the work needed for doing mappings. We can have one single wiki page describing every field on the master table.
We get live data, no need for 20min refresh periods. We can create apps that visualize in real time how the data get entered. One of the interesting visualizations is to see things going on in real time.
But again, the biggest reason to me is to have to avoid merges of multiples tables, one per application. I prefer if we dont need to do integration with such little time ahead.
What do you see as the biggest problems for using one single table for everybody?
In any case. A combination of both can also work I have to say. Actually if someone uses his own fusion table that will be the same scenario as scrapping image from the EOL Flickr group.
Javier de la Torre www.vizzuality.com
On Aug 26, 2010, at 6:58 PM, Michael Giddens wrote:
Joel,
Based on our discussion with David R. and the phone conference 2 weeks back you where thinking everyone can write to their own tables then we merge in the data into one master table. I wrote a script to do this but have not tested it out. It is something we can do if you have more then 2 sample tables with data. This way anyone that shares their table with userX will get harvested and that data will be loaded into the master readonly table that is reloaded over x minutes. And if there are custom tables that are not part of your default table standards we can possibly setup a unique harvester during the conference to transform to the correct column model. This way everyone is independent but at the same time able to view the master data every 20 minutes or whatever update time we set it to. We can do the same for different table types.
Any questions just let me know or contact me directly.
Michael Giddens Office: 225-238-1879 skype: mikegiddens
On 8/26/2010 11:45 AM, joel sachs wrote:
Hi Everyone -
Javier and I chatted yesterday about unresolved technical issues. Our primary goal was simplicity. Here are the outcomes. Feel free to suggest something different, if either i) you're planning on developing around your suggestion; or ii) you believe that to not follow your suggestion would be a grave mistake.
NOTE: by "app" we mean anything that is writing to the table - could be a smartphone app, could be a web form, could be a screenscraper or tweet-parser.
- All apps will write to the same table. Each app will need a Google
account, which will be given write-access to the table.
- The occurrence_id will be the row_id assigned by Fusion Tables. This is
not seen in the web interface, but is available from the API.
- To support crowdsourcing of image identification, alternative
identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these columns will also exist in the Occurrences table. When the observation is first reported, any identification will be entered in the Occurrences table, with subsequent identifications in the Identifications table. (The main motivation for keeping the initial identification in the Occurrences table is to ease the process for app developers.)
- Multiple multimedia URLs will be listed in a single column, comma
separated.
- The lat and long columns will be WGS84. If app developers wish, they
are welcome to build support for additional datums into their apps, since the apps will have authority to create additional columns. If this doesn't happen, anyone planning to use another datum will need to announce themselves ahead of time, and we will create appropriate columns for them.
- The table will contain Kingdom, Phylum, Class, Order, and Family
columns. Javier will write software to resolve identifications to the Catalog of Life, and to populate the taxonomy columns. If someones want to resolve observations to a different classification, they are free to do so, and are encouraged to publish the results in Fusion Tables.
- In addition to the identification table, there is scope for creating an
Annotations table and front-end, should anyone wish to tackle this.
Here's a timetable that would be nice to adhere to:
Sept. 1: Final versions of the Occurrences and Identifications tables published to Fusion Tables. Write access will only be granted to developers. These will be the test tables. The day before the bioblitz, they will become the development tables, and test records will be expunged.
Sept. 3: Table documentation and sample code for writing to the tables.
Sept. 8: Taxanomic resolution, validation, assignment of taxon GUIDS/LSIDs.
Sept. 15: At least one Image identification framework in place.
Sept. 15 - Sept. 28: Testing and work on visualization and other data-oriented services (e.g. publication as DwC or Linked Data, etc.)
Best to all - Joel.
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
Feel free to suggest something different, if ..... you believe that to not follow your suggestion would be a grave mistake.
<...>
- The table will contain Kingdom, Phylum, Class, Order,
and Family columns. Javier will write software to resolve identifications to the Catalog of Life
Adding taxonomic hierarchy is imperative. However, using Catalog of Life is likely to disappoint because it appears to be based on a relatively recent APG system which requires gene sequences which are NOT available for most plants, hence a huge number of plants landing in the category called "not assigned", causing a loss of information held in the centuries old Chronquist hierarchy. see: http://delta-intkey.com/angio/www/apg.htm http://www.botany.hawaii.edu/faculty/carr/pfamilies.htm
There are advantages to APG if the classification usage is restricted to genetic work but it is not ready for prime time for other purposes. Therefore, I recommend that the Chronquist scheme be used for plants.
Soaring Bear Ph.D. Pharmacology soaringbear at yahoo.com http://soaringbear.com/nature/WeedsforNeeds.html http://www.nlm.nih.gov/mesh/presentations/bear_2005_aug/index.htm author of http://HerbMed.org & http://HerbInsight.com
--- On Thu, 8/26/10, joel sachs jsachs@csee.umbc.edu wrote:
From: joel sachs jsachs@csee.umbc.edu Subject: [tdwg-tag] Unresloved technical issues - proposed decisions. To: tdwg-bioblitz@googlegroups.com, tdwg-tag@lists.tdwg.org Date: Thursday, August 26, 2010, 11:45 AM Hi Everyone -
Javier and I chatted yesterday about unresolved technical issues. Our primary goal was simplicity. Here are the outcomes. Feel free to suggest something different, if either i) you're planning on developing around your suggestion; or ii) you believe that to not follow your suggestion would be a grave mistake.
NOTE: by "app" we mean anything that is writing to the table - could be a smartphone app, could be a web form, could be a screenscraper or tweet-parser.
- All apps will write to the same table. Each app will
need a Google account, which will be given write-access to the table.
- The occurrence_id will be the row_id assigned by Fusion
Tables. This is not seen in the web interface, but is available from the API.
- To support crowdsourcing of image identification,
alternative identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these columns will also exist in the Occurrences table. When the observation is first reported, any identification will be entered in the Occurrences table, with subsequent identifications in the Identifications table. (The main motivation for keeping the initial identification in the Occurrences table is to ease the process for app developers.)
- Multiple multimedia URLs will be listed in a single
column, comma separated.
- The lat and long columns will be WGS84. If app
developers wish, they are welcome to build support for additional datums into their apps, since the apps will have authority to create additional columns. If this doesn't happen, anyone planning to use another datum will need to announce themselves ahead of time, and we will create appropriate columns for them.
- The table will contain Kingdom, Phylum, Class, Order,
and Family columns. Javier will write software to resolve identifications to the Catalog of Life, and to populate the taxonomy columns. If someones want to resolve observations to a different classification, they are free to do so, and are encouraged to publish the results in Fusion Tables.
- In addition to the identification table, there is scope
for creating an Annotations table and front-end, should anyone wish to tackle this.
Here's a timetable that would be nice to adhere to:
Sept. 1: Final versions of the Occurrences and Identifications tables published to Fusion Tables. Write access will only be granted to developers. These will be the test tables. The day before the bioblitz, they will become the development tables, and test records will be expunged.
Sept. 3: Table documentation and sample code for writing to the tables.
Sept. 8: Taxanomic resolution, validation, assignment of taxon GUIDS/LSIDs.
Sept. 15: At least one Image identification framework in place.
Sept. 15 - Sept. 28: Testing and work on visualization and other data-oriented services (e.g. publication as DwC or Linked Data, etc.)
Best to all - Joel.
tdwg-tag mailing list tdwg-tag@lists.tdwg.org http://lists.tdwg.org/mailman/listinfo/tdwg-tag
- To support crowdsourcing of image identification, alternative
identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these
Add support for identification giving an id/id-source or guid instead of a string?
Also, "Kingdom" may not be known and changing, but dwc:nomenclaturalCode will always be. The same scientificName may refer to different organisms under different codes.
- Multiple multimedia URLs will be listed in a single column, comma
separated.
comma is valid character in URLs.
...Only alphanumerics [0-9a-zA-Z], the special characters "$-_.+!*'()," [not including the quotes - ed], and reserved characters used for their reserved purposes may be used unencoded within a URL. - from RFC 1738
Standard character to separate URL list elements would be the blank
Gregor
comma is valid character in URLs.
...Only alphanumerics [0-9a-zA-Z], the special characters "$-_.+!*'()," [not including the quotes - ed], and reserved characters used for their reserved purposes may be used unencoded within a URL.
- from RFC 1738
Standard character to separate URL list elements would be the blank
Nice. Well spotted, thanks. +1
Hi Gregor -
On Fri, 27 Aug 2010, Gregor Hagedorn wrote:
- To support crowdsourcing of image identification, alternative
identifications, arguments, etc., there will be a 2nd table with columns {occurrence_id, scientificName, vernacularName, Kingdom, identifiedBy, identificationResources, identificationRemarks}. One of each of these
Add support for identification giving an id/id-source or guid instead of a string?
Yes - we should have a "taxonConceptID" column, as we do in the Occurrences table. In most cases, this will be filled out automatically after the observation has been entered, by a tool that maps names to guids. But, this being TDWG, you're right that some may identify the taxon with a guid. (I'd love to see this.)
Also, "Kingdom" may not be known and changing, but dwc:nomenclaturalCode will always be. The same scientificName may refer to different organisms under different codes.
This columns exists to help disambiguate scientificName, since, as you point out, the same scientificName may refer to different organisms under different codes. (For example there are both Pieris shrubs and Pieris butterflies in Cape Cod.) Since this column gets filled in the field, I thought it should be intuitive, and that everyone would understand "Kingdom" with a small controlled vocabulary of choices. I agree it's not perfect, but neither are the codes.
- Multiple multimedia URLs will be listed in a single column, comma
separated.
comma is valid character in URLs.
an oversight on my part.
Thanks - Joel.
...Only alphanumerics [0-9a-zA-Z], the special characters "$-_.+!*'()," [not including the quotes - ed], and reserved characters used for their reserved purposes may be used unencoded within a URL.
- from RFC 1738
Standard character to separate URL list elements would be the blank
Gregor
Yes - we should have a "taxonConceptID" column, as we do in the Occurrences table. In most cases, this will be filled out automatically after the observation has been entered, by a tool that maps names to guids. But, this being TDWG, you're right that some may identify the taxon with a guid. (I'd love to see this.)
:-)
But I was actually thinking of something way more pragmatic, like local ID of a ID source and name of ID-Source. In practice of the observations databases I know, they all have ID-numbers for taxa, but often only nationally standardized, or even by project. Still, I believe this would mean great benefit.
Gregor
Out of curiosity (you play above my level :-) ) I took a look at the proposed columns. Two random notes:
trophicLevel seems a bit odd for occurrence data - are some organisms sometimes on one level and sometimes on another?
Is there a vocabularly for that? I just wonder where to put my fungi...
-------
dateTime
can I the choose semantics? :-) it might be playing safe to make sure already in the label whether this shall be dateTime of observation, accession, data entry, etc. pp.
Gregor
Gregor -
I stole the "trophic level" field from the 300 Committee bioblitz data sheet. They used a controlled vocabulary consisting of {primary producer (pp); herbivore (h); primary carnivore (pc); higher-level carnivore (hc); decomposer (d)}. It was pretty cool (for me) to see the shape of the food web emerge (coloured pins on a poster board) as the observations came in. Yes, those designations are attributes of species, don't really change across ecosystems, and could, in theory, be assigned in post-processing. But I liked having it in the form, as it encouraged thinking on the part of observers.
An early draft of documentation for the data fields is here: http://groups.google.com/group/tdwg-bioblitz/web/tdwg-bioblitz-profile-v1-1
I will post a revised version to the wiki. These will be guidelines; it is up to the app developers to decide what fields they want to include, and how to walk the line between completeness and simplicity. Essentially, the data profile should be interpreted as saying "If you use fields with these names, the following syntax and semantics will be assumed ..."
Best - Joel.
On Fri, 27 Aug 2010, Gregor Hagedorn wrote:
Out of curiosity (you play above my level :-) ) I took a look at the proposed columns. Two random notes:
trophicLevel seems a bit odd for occurrence data - are some organisms sometimes on one level and sometimes on another?
Is there a vocabularly for that? I just wonder where to put my fungi...
dateTime
can I the choose semantics? :-) it might be playing safe to make sure already in the label whether this shall be dateTime of observation, accession, data entry, etc. pp.
Gregor
participants (5)
-
Gregor Hagedorn -
Javier de la Torre -
joel sachs -
Michael Giddens -
Soaring Bear