Content Discovery ================= .. module:: SearchMetadata .. currentmodule:: SearchMetadata .. contents:: Searching for content is a functionality that is primarily supported by Coordinating Nodes, programmatically through the CN_read.search method, and through a web browser interface that connects through a user interface implemented by Mercury_ which utilizes a SOLR_ index to support search operations. This document describes the properties of the SOLR index, how it is populated, and how it can be used to support programmatic search and introspection into the DataONE holdings. The SOLR index is populated from content that is copied to Coordinating Nodes, that is :term:`System Metadata`, :term:`Science Metadata` and :term:`Resource Maps`. Each index entry represents information about the information referenced by a single identifier (PID). For PIDs that refer to science data objects the index entry will be constructed from system metadata and zero or more resource maps that reference the object. If a PID identifies a science metadata object, the index entry will have fields populated with content extracted from the system metadata for the object, the science metadata document and zero or more resource maps that reference the science metadata. Similarly, if the PID identifies a resource map, then the index entry fields for that PID will be populated by the System Metadata and resource map values for that PID. The general sequence for indexing content is illustrated in Figure 1 below. The general process of indexing content involves retrieving a System Metadata document, parsing system metadata properties such as permission rules to generate a SOLR index document, then optionally adding to the document if the PID refers to a science metadata or resource map object. The general sequence of operations is outlined in Figure 1. If the PID refers to a resource map, then it is necessary to update the :attr:`~SearchMetadata.documents` and :attr:`~SearchMetadata.isDocumentedBy` properties of the Science Metadata and data objects that appear in the data package defined by the Resource Map. .. figure:: images/index_sequence.png .. @startuml images/index_sequence.png (*) -right-> "System metadata updated" --> "Index Task" --> "Extract system metadata" if "Science\nMetadata?" then -right-> [true] "Parse science metadata" --> "Update SOLR" else if "Resource\nMap?" then -right-> [true] "Parse resource map" --> "Update data and metadata\n resource map index fields" --> "Update SOLR" else --> "Update SOLR" --> (*) endif endif @enduml A full list of index fields is provided in Table 4 below. Querying the SOLR Index ----------------------- The `SOLR query syntax`_ is based on the `Lucene query syntax`_ and adds a couple of extensions. The SOLR search index is exposed directly as a SOLR search service with the base url of:: http(s)://server.name/solr/ System Metadata Index Properties -------------------------------- The main goals for indexing system metadata include: 1. Support access control rules for search results 2. Support efficient discovery of objects by their properties Table 1 provides a list of SOLR attributes that are populated by values taken from System Metadata. **Table 1.** Elements derived from System Metadata and available for all objects. "Multi" fields may be populated with more than a single value per document (PID) indexed. .. sqltable:: System Metadata Properties :header: Attribute,Type,Multi?,Content Origin :widths: 3 2 1 10 :driver: xlsx :source: source/design/data/SearchFields.xlsx :sql: SELECT ':attr:`~SearchMetadata.'||SOLR||'`', FieldType, Multi, Reference FROM SystemMetadata; Populating Permission Fields ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This process is fairly straight forward, and requires simply de-duping entries so that each of the permission fields contains a distinct list of subjects granted that permission. If the DataONE :term:`public user` is present in the read permissions, then the *isPublic* field is set to True. Example Queries ~~~~~~~~~~~~~~~ The following provide some example queries that may be issued directly against the SOLR index operating on the Coordinating Nodes. Note that search terms must be properly escaped for the SOLR query syntax, and additionally if submitted directly as part of a URL, must also be properly URL escaped. 1. Index entries that match PID (should be at most one match):: id:"PID" where PID = identifier of object being located 2. Index entries with PID that starts with "some_prefix":: id:"some_prefix*" 3. Index entries that match a particular format id:: formatId:"format_a" where format_a = the formatId being located 4. Entries with formatId matching "fmtid_1" or "fmtid_2":: formatId:"fmtid_1" || formatId:"fmtid_2" 5. Entries with byte size less than or equal to 10000:: size:[* TO 10000] 6. Entries with byte size less than 10000:: size:{* TO 10000} 7. Objects modified before 09:56:04 on 2012-01-03 UTC:: datemodified:{* TO 2012-01-03T09:56:04.000Z} 8. Objects modified less than 10 minutes ago:: datemodified:[NOW-10MINUTE TO *] 9. Objects of a formatId equal to "format_a" and modified within one day:: formatId:"format_a" AND datemodified:[NOW-1DAY TO *] 10. Metadata associated with data objects of relevance to Photosynthesis:: photosynthesis AND documents:[* TO *] Properties of the Index Derived from Resource Maps -------------------------------------------------- Table 2 provides a list of SOLR attributes that are populated by values obtained by parsing resource map objects. **Table 2.** Elements derived from Resource Map documents. "Multi" fields may be populated with more than a single value per document (PID) indexed. .. sqltable:: Resource Map Properties :header: Attribute,Type,Multi?,Content Origin :widths: 3 2 1 10 :driver: xlsx :source: source/design/data/SearchFields.xlsx :sql: SELECT ':attr:`SearchMetadata.'||SOLR||'`', FieldType, Multi, Reference FROM ResourceMap; Populating Object Relation Fields ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The fields *resourceMap*, *documents*, and *isDocumentedBy* contain identifiers of other objects in the DataONE system related to the subject of the current record. These relationships are recorded in OAI-ORE resource map documents as described in the section :doc:`DataPackage`. Since the values for these fields are contained in resource maps, it is necessary to build these fields in progression as additional resource maps are synchronized with the coordinating nodes. For each object, a query is issued against the index to retrieve the list of resource maps in which the object identifier is present, enabling population of the *resourceMaps* field for the object. Each of those resource maps is examined to obtain the list of identifiers for the *documents* field if the object is science metadata, and the *isDocumentedBy* field if the object is data. **Example Sequence for Index Population** The following sequence steps through the generation of index entries for three data packages, providing an example of how the index entries are updated depending on relationships between the objects being added. Adding Package 1 ................ .. graphviz:: digraph RM1 { graph [rankdir="LR", label="Package 1"]; A [shape=folder]; B [shape=note]; C [shape=oval]; A -> B; A -> C; } A simple data package with a Resource Map *A* indicating that the metadata document *B* and the data object *C* together form a data package, and that *B* documents *C* and that *C* isDocumentedBy *B*. Index entries after Package 1 is added. .. list-table:: :header-rows: 1 :widths: 3 3 3 3 * - PID - resourceMaps - documents - isDocumentedBy * - A - - - * - B - A - C - * - C - A - - B Adding Package 2 ................ .. graphviz:: digraph RM1 { graph [rankdir="LR", label="Package 2"]; D [shape=folder]; B [shape=note]; E [shape=oval]; D -> B; D -> E; } A simple data package with a Resource Map *D* indicating that the metadata document *B* and the data object *E* together form a data package, and that *B* documents *E* and that *E* isDocumentedBy *B*. Note that the metadata document *B* is also used in Package 1. Index entries after package 2 is added. New entries and changes to existing are indicated in **bold**. .. list-table:: :header-rows: 1 :widths: 3 3 3 3 * - PID - resourceMaps - documents - isDocumentedBy * - A - - - * - B - A, **D** - C, **E** - * - C - A - - B * - D - - - * - E - **D** - - **B** Adding Package 3 ................ .. graphviz:: digraph RM1 { graph [rankdir="LR", label="Package 3"]; F [shape=folder]; G [shape=note]; D [shape=folder]; F -> G; F -> D; } A data package that references another, already indexed data package (represented by the Resource Map *D*) as its data object. Index entries after package 3 is added. New entries and changes to existing are indicated in **bold**. .. list-table:: :header-rows: 1 :widths: 3 3 3 3 * - PID - resourceMaps - documents - isDocumentedBy * - A - - - * - B - A, D - C, E - * - C - A - - B * - D - **F** - - **G** * - E - D - - B * - F - - - * - G - **F** - **D** - Limitations of Multi-Valued Fields in SOLR ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Multi-valued fields in SOLR have an internal representation that can be likened to a delimited concatenation of all the values for the field. The fields have a configurable limit to the maximum field size, and this limit may be encountered in the various entries such as the permission fields or object relation fields where a potentially large number of subjects or identifiers may be stored. Further research has indicated this problem is unlikely since simply increasing the SOLR *maxFieldLength* configuration value will enable larger field content, with examples in the wild of several thousand entries equivalent to DataONE identifiers in a single field. .. _Mercury: http://mercury.ornl.gov/ .. _OAI-ORE: http://www.openarchives.org/ore/ Values Extracted from Science Metadata -------------------------------------- Indexing the system metadata entries provides a mechanism for selection of content using low level attributes of the objects such as the type, size and relationships, however is not useful for locating content relevant to a particular topic or purpose. The :term:`science metadata` contains such information and the index records associated with these documents contain additional information described below. The initial web user interface to DataONE utilizes the Mercury_ system. This in turn relies upon a SOLR index to support the search, faceting, and sub-setting operations that are exposed through Mercury. The underlying SOLR index is also exposed through as an API, so other clients (such as the file system driver) can leverage the search capabilities thus provided. The remainder of this document describes the search properties that are extracted from the various formats and syntaxes of of science metadata that are supported by the DataONE indexing system. **Table 3.** Index fields populated from science metadata documents. "Multi" fields may be populated with more than a single value per document (PID) indexed. .. sqltable:: Resource Map Properties :header: Field,Type,Multi?,Description :widths: 2 1 1 10 :driver: xlsx :source: source/design/data/SearchFields.xlsx :sql: SELECT ':attr:`SearchMetadata.'||SOLR||'`', FieldType, Multi, Description FROM Fields WHERE Source='S'; Date Representations in Science Metadata Documents -------------------------------------------------- The following are examples of date values extracted from FGDC, ISO-19115, and EML science metadata documents currently in use. The literal value appearing in the document and the interpreted date value are shown. Examples of date values appearing in FGDC **: ================================== ==================== Value Interpreted Value ================================== ==================== Unknown Null unknown Null Unpublished material Null unpublished material Null 1993 1993-01-01 00:00:00Z 199607 1996-07-01 00:00:00Z 20000101 2000-01-01 00:00:00Z 19981231 1998-12-31 00:00:00Z 196820405 1968-01-01 00:00:00Z 1992 onwards 1992-01-01 00:00:00Z 1989 and 1990 1989-01-01 00:00:00Z varies Null Present Null 1995/1996 1995-01-01 00:00:00Z 1991-1992 1991-01-01 00:00:00Z variouis Null April 1999 1999-04-01 00:00:00Z 1980 on 1980-01-01 00:00:00Z 2005-06-24 2005-06-24 00:00:00Z NA Null 1990- [unpublished annual reports] 1990-01-01 00:00:00Z November, 1994 1994-11-01 00:00:00Z ================================== ==================== Examples of date values appearing in ISO-19115 **: ================================== ==================== Value Interpreted Value ================================== ==================== 1999 1999-01-01 00:00:00Z 2010-03-03 2010-03-03 00:00:00Z ================================== ==================== Examples of date values appearing in EML **: ================================== ==================== Value Interpreted Value ================================== ==================== 2002-06-20 2002-06-20 00:00:00Z 1998 1998-01-01 00:00:00Z 2004-02-13 2004-02-13 00:00:00Z ================================== ==================== Science metadata examples ------------------------- Examples for integration testing: https://repository.dataone.org/software/cicore/trunk/d1_integration/src/test/resources/d1_testdocs/ FGDC: http://mercury.ornl.gov/metadata/nbii/xml/nbii/ ISO 19115: http://mercury.ornl.gov/metadata/ornldaac/xml/daac/iso/ EML: 2.0.0: https://repository.dataone.org/software/cicore/trunk/d1_integration/src/test/resources/d1_testdocs/eml200/ 2.0.1: https://repository.dataone.org/software/cicore/trunk/d1_integration/src/test/resources/d1_testdocs/eml201/ 2.1.0: https://repository.dataone.org/software/cicore/trunk/d1_integration/src/test/resources/d1_testdocs/eml210/ Standard Specific Metadata Extraction Notes ------------------------------------------- .. toctree:: :maxdepth: 1 SearchMetadata_eml SearchMetadata_fgdc SearchMetadata_dryad Attribute Descriptions and Notes -------------------------------- A list of all fields in search index, their origin and brief description. Source column indicates which DataONE content the values are retrieved from: Y = System Metadata, S = Science Metadata, R = Resource Map, and C = copied internally by the search index. .. sqltable:: . . . :header: Index Field,Multi?,Source,Description :widths: 1 1 1 10 :driver: xlsx :source: source/design/data/SearchFields.xlsx :sql: SELECT '.. attribute:: '||SOLR, Multi, Source, Description FROM Fields ORDER BY LOWER(SOLR) ASC; ---- Creating Citations from Index Fields ------------------------------------ **Table 3.** Cross reference of index fields and elements useful in citation best practices as exemplified in the `interagency data stewardship`_ wiki page of ESIP. * = Mandatory (if applicable). COinS tags provide a mechanism of embedding `Z39.88-2004`_ elements in HTML for consumption by citation manager tools. .. list-table:: :header-rows: 1 :widths: 5 5 5 5 5 * - Citation Element - FGDC CSDGM field - DataCite Metadata Scheme ID and Property - DataONE Index Field - COinS_ Tag * - Author or Creator* - idinfo > citation > citeinfo > "origin" - 2 Creator* - :attr:`author` - rft.creator * - Release Date* - idinfo > citation > citeinfo > "pubdate" and sometimes "othercit" - 5 PublicationYear* - :attr:`datePublished` - rft.date * - Title* - idinfo > citation > citeinfo > "title" and possibly "edition" - 3 Title* - :attr:`title` - rft.title * - Version* - - 15 Version - See notes below. - * - Archive and/or Distributor* - idinfo > citation > citeinfo > "publish" - 4 Publisher* - See notes below. - rft.publisher * - Locator, Identifier, or Distribution Medium* - idinfo > citation > citeinfo > "othercit" or "onlink" - 1 Identifier* - :attr:`pid` - rft.identifier * - Access Date and Time* - not applicable - 8 Date - perhaps :attr:`dateUploaded` - * - Subset Used - not applicable - 12 RelatedIdentifier DataCite recommends obtaining an identifier for any subset that needs to be cited as well as an identifier for the larger whole. - perhaps a reference to the resource map? - * - Editor or Other Important Role - idinfo > citation > citeinfo > "origin" - 7 Contributor - See notes below. - rft.source * - Publication Place - idinfo > citation > citeinfo > "pubplace" - 17 Description - See notes below. - * - Distributor, Associate Archive, or other Institutional Role - idinfo > citation > citeinfo > "othercit" - 7 Contributor or possibly 4 Publisher - "http://dataone.org" - * - Data Within a Larger Work - idinfo > citation > citeinfo > "othercit" or "lworkcit" - 12 RelatedIdentifier - perhaps a reference to the resource map? - * - - - - :attr:`keyWord` - rft.subject * - - - - :attr:`abstract` - rft.description * - - - - object format name, or "data" | "metadata" | "resource map" - rft.type * - - - - object format type - rft.format * - - - - constructed from bounding box - rft.coverage **Notes** :Version: There is no clear recommendation on how to express version. For an object with a linear revision history (A is obsoleted by B is obsoleted by C), object C might be considered to have a version of "2". However, it is possible for an object to have a tree lineage (A1 and A2 are obsoleted by B1, and B1, A3 and Z are obsoleted by C). A suggestion for a version indicator in this case may be the total number of parent objects, 5 in this case. Another option may be to indicate the number of contributing objects at each revision, 1.1 in the first case, and 2.3 in the second. :Publisher: The publisher in DataONE may in some cases be the origin member node, however this will not always be the case. For example, many data sets in the KNB are published by LTER, though KNB would be the member node. Where possible the publisher information should be pulled from the science metadata, though this may be complicated where multiple data and metadata entries occur within a data package and there are several publishers listed. In this case, a simple concatenation of the publishers (semi-colon + space delimited) may be sufficient. :Date: In the DataCite 2.2 document, a mapping between Date and the dcterms:date is suggested. The description of dcterms:date is "A point or period of time associated with an event in the lifecycle of the resource." In DataONE there are two main dates related to an object, when the content is added to the system, and when the properties of the object (access control, and so forth) are modified. Since the content of the object does not change once submitted to DataONE, the :attr:`SystemMetadata.dateUploaded` value seems appropriate here, expressed as :attr:`dateUploaded` in the search index. :Subset Used: DataONE does not provide a mechanism for identifying sets of information other than through resource maps which describe data packages. It is possible that a resource map may contain references to additional resource maps, conceptually equivalent to subsets of content. The simplest option here would be to use the identifier of the resource map object describing the package in question. :Contributor: Described in DataCite 2.2 as "The institution or person responsible for collecting, creating, or otherwise contributing to the development of the dataset", this optional element can be populated by the citeinfo/origin element from FGDC. :Publication Place: In CSDGM, "the name of the city (and state or province, and country, if needed to identify the city) where the data set was published or released". :Data Within a Larger Work: This is similar to the "Subset Used" entry except in the other direction of containment. The identifier of the containing resource map relevant to the current package should be used. .. _interagency data stewardship: http://wiki.esipfed.org/index.php/Interagency_Data_Stewardship/Citations/provider_guidelines .. _DataCite terms: http://wiki.esipfed.org/index.php/Interagency_Data_Stewardship/Citations/provider_guidelines .. _Z39.88-2004: http://www.niso.org/kst/reports/standards/kfile_download?id%3Austring%3Aiso-8859-1=Z39-88-2004.pdf&pt=RkGKiXzW643YeUaYUqZ1BFwDhIG4-24RJbcZBWg8uE4vWdpZsJDs4RjLz0t90_d5_ymGsj_IKVaGZww13HuDlZQ8NBt1sTxP_v4iiGqH7rSaAeVDnMfeKJrrJ_JSEGPB .. _COinS: http://ocoins.info/ .. _Well Known Text: http://en.wikipedia.org/wiki/Well-known_text .. _SOLR: http://lucene.apache.org/solr/ .. _SOLR query syntax: http://wiki.apache.org/solr/SolrQuerySyntax .. _Lucene: http://lucene.apache.org/java/docs/index.html .. _Lucene query syntax: http://lucene.apache.org/java/2_9_1/queryparsersyntax.html .. NOTES etc (not rendered below) Searching content in DataONE involves the process of matching a user provided value with some value extracted from the science metadata. Values can be simple (e.g. string, integer, floating point, date) or compound (e.g. spatial coordinate constructed from two floats - the latitude and longitude and a string - the coordinate system identifier). Expression in science metadata Expression in extracted values that are indexed Expression in a query * Integer * Float * String * DateTime * SpatialFeature (use `Well Known Text`_ ) * Publication (A structured type, perhaps search by decomposing to elements) * ScientificName (A structured type, perhaps search by decomposing to elements)