Natural History of System Metadata

Revisions:
Date Comment
20100303 (D.V.) Initial draft
20100304

(D.V.) Revised draft and added scenario where system metadata is stored only on the Coordinating Nodes

Corrected DataSysMetadataModified changes in scenario B.

This document describes the generation of system metadata associated with a data object and its associated science metadata document as they are added to a DataONE Member Node and subsequently propagated through the DataONE infrastructure.

Process Overview

There are two scenarios examined here: A) the system metadata is stored on all nodes (MN and CN) and B) system metadata is stored only on the CNs. The general sequence of operations is the same for both scenarios with the exclusion of a couple of steps from scenario B.

  1. Scientist using an ITK tool generates some science data and some science metadata describing the science data.

  2. The ITK populates the required fields of the system metadata documents.

  3. Not necessarily in this order:

    1. The ITK uploads the science data
    2. The ITK uploads the science metadata

    On receipt, the MN updates the system metadata fields for which it is responsible.

  4. The CN detects new content is available on the MN using the listObjects() method.

  5. Copies of the science metadata and the two (incomplete) system metadata documents are retrieved by the CN.

    A:The CN updates the system metadata replication information, and indicates to the MN that it should update its copy of the system metadata objects.
    B:The CN updates the system metadata replication information.
  6. The CN issues a replication order to an MN.

    A:The recipient MN retrieves copies of the system metadata and associated objects (science data and metadata)
    B:The recipient MN retrieves copies of the various objects indicated by the CN
  7. The recipient MN requests content from the origin MN, which in turn verifies the request with the CN.

    A:The origin MN and the CN update copies of system metadata
    B:The CN updates system metadata
  8. After verifying the successful transfer of content, the recipient MN indicates completion to the CN.

    A:The CN updates the system metadata for the objects and ensures that the MN copies of the system metadata are accurate.
    B:The CN updates the system metadata for the objects.

The change in system metadata state is shown in the following process detail sections. In each case, the identifiers used for the various elements are indicated in table 1.

Table 1. Identifiers used in the example sequence.
Identifier Description
sciD.1 Science Data
sciM.1 Science metadata describing sciD.1
sysD.1 System metadata for object sciD.1
sysM.1 System metadata for object sciM.1
cn1.dataone.org A Coordinating Node
mn1.dataone.org Member Node that is initial recipient of data
mn2.dataone.org Member Node that receives a copy of the objects

The following fictitious script provides an example of how uploading content to a Member Node using a DataONE client library might occur.

import sys
import logging
from pyd1.client import *
from pyd1.exceptions import *

d1client = pyd1.client(identity="/home/vieglais/.d1/credentials",
                       defaults="/home/vieglais/.d1/defaults.ini")
d1client.login()

# The science data and metadata as files on disk
data_file = "/home/vieglais/data/mydata.csv"
meta_file = "/home/vieglais/data/mymeta.xml"

# Reserve a couple of identifiers. Note that this reservation will
# be system-wide.
try:
  data_id = d1client.reserve(u"sysD.1")
  meta_id = d1client.reserve(u"sysM.1")
except IdentifierInUseException, e:
  logging.fatal(repr(e))
  sys.exit()

# Generate stub system metadata documents. Default values
# for access rules etc are loaded from defaults.ini
sysm_data = d1client.generateSysMeta(id=data_id,
                                     format="text/csv",
                                     describedBy=meta_id)
sysm_meta = d1client.generateSysMeta(id=meta_id,
                                     format="eml://ecoinformatics.org/eml-2.1.0",
                                     describedBy=data_id)
# Open file handles for data and metadata
fdata = file(data_file, "r")
fmetadata = file(meta_file, "r")

# Send the data to the Member Node
target_node = 'mn1.dataone.org'
data_sysid = d1client.create(data_id, fdata, sys_data,
                             target=target_node)
meta_sysid = d1client.create(meta_id, fmeta, sys_meta,
                             target=target_node)
fdata.close()
fmeta.close()

Process Detail - Scenario A

Step A1.

The scientist generates a data set and associated science metadata.

Step A2.

Using tools in the ITK, the scientist generates two initial system metadata documents, one describing the science metadata (sysm_M.1), one describing the science data (sysm_D.1).

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0
size 1450238 8132
submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
obsoletes
obsoletedBy
derivedFrom
describes sciD.1
describedBy sciM.1
checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342
checksumAlgorithm SHA-1 SHA-1
accessRule[1]
.rule Allow Allow
.service Read Read
.principal * *
accessRule[2]
.rule Allow Allow
.service Write Write
.principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org
replicationPolicy
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded
dateSysMetadataModified
originMemberNode
authoritativeMemberNode
replica[1]
.replicaMemberNode
.replicationStatus
.replicaVerified

Step A3.

The client tool transmits the data, science metadata and two stub system metadata documents to a Member Node. The Member Node updates values of the system metadata for which it is responsible.

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0
size 1450238 8132
submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
obsoletes
obsoletedBy
derivedFrom
describes sciD.1
describedBy sciM.1
checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342
checksumAlgorithm SHA-1 SHA-1
accessRule[1]
.rule Allow Allow
.service Read Read
.principal * *
accessRule[2]
.rule Allow Allow
.service Write Write
.principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org
replicationPolicy
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Queued Queued
.replicaVerified

Since there are many copies of system metadata from this point onwards, only the replication properties of the sysD.1 document are considered, summarized in a table as follows. MN1, CN, MN2 = copy of system metadata on mn1.dataone.org, cn1.dataone.org, mn2.dataone.org respectively.

Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org    
R[1].replicationStatus Queued    
R[1].replicaVerified      

Step A4.

A CN detects new content available on mn1.dataone.org through the listObjects() method.

There is no change of state in system metadata resulting from this operation.

Step A5.

Copies of sciM.1, sysM.1, and sysD.1 are retrieved to the CN, and the system metadata objects are updated. There are three steps to this process:

  1. The CN requests a copy of the objects.
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org    
R[1].replicationStatus Requested    
R[1].replicaVerified      
  1. The CN receives a copy of the metadata.
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org  
R[1].replicationStatus Requested Requested  
R[1].replicaVerified      
  1. The CN verifies the checksums and indicates to the MN that the copy is complete, providing the time stamp that it used for the verification (ensuring identical copies of the system metadata).
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org  
R[1].replicationStatus Completed Completed  
R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z  

Step A6.

The CN initiates replication of content from mn1.dataone.org to mn2.dataone.org.

  1. The CN sends a replication request to MN2, indicating that it should retrieve the object from MN1.
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org  
R[1].replicationStatus Completed Completed  
R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z  
R[2].replicaMemberNode   mn2.dataone.org  
R[2].replicationStatus   Queued  
R[2].replicaVerified      
  1. MN2 requests the content from MN1.
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org
R[1].replicationStatus Completed Completed Completed
R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z
R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org
R[2].replicationStatus Requested Queued Requested
R[2].replicaVerified      
  1. MN1 checks with CN that the object was requested
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org
R[1].replicationStatus Completed Completed Completed
R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z
R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org
R[2].replicationStatus Requested Requested Requested
R[2].replicaVerified      
  1. MN2 verifies that the checksum is correct, then indicates to MN1 that the copy was correctly transferred, forwarding the timestamp for when the object was verified to MN1.
Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org
R[1].replicationStatus Completed Completed Completed
R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z
R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org
R[2].replicationStatus Completed Requested Completed
R[2].replicaVerified 2010-03-04T20:00:00.0Z   2010-03-04T20:00:00.0Z

Step A8.

MN2 informs the Coordinating Node that it has retrieved a valid copy of the object from MN1, forwarding the time stamp for when the verification was made.

Property MN1 CN MN2
R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org
R[1].replicationStatus Completed Completed Completed
R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z
R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org
R[2].replicationStatus Completed Completed Completed
R[2].replicaVerified 2010-03-04T20:00:00.0Z 2010-03-04T20:00:00.0Z 2010-03-04T20:00:00.0Z

Process Detail - Scenario B.

This scenario uses the notion that system metadata is stored on the CNs only.

Step B1.

The scientist generates a data set and associated science metadata.

Step B2.

Using tools in the ITK, the scientist generates two initial system metadata documents, one describing the science metadata (sysm_M.1), one describing the science data (sysm_D.1).

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0
size 1450238 8132
submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
obsoletes    
obsoletedBy    
derivedFrom    
describes   sciD.1
describedBy sciM.1  
checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342
checksumAlgorithm SHA-1 SHA-1
accessRule[1]    
.rule Allow Allow
.service Read Read
.principal * *
accessRule[2]    
.rule Allow Allow
.service Write Write
.principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded    
dateSysMetadataModified    
originMemberNode    
authoritativeMemberNode    
replica[1]    
.replicaMemberNode    
.replicationStatus    
.replicaVerified    

Step B3.

The client tool transmits the data, science metadata and two stub system metadata documents to a Member Node (mn1.dataone.org). On receipt, the Member Node sets the values of several fields as indicted in the table below.

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0
size 1450238 8132
submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
obsoletes    
obsoletedBy    
derivedFrom    
describes   sciD.1
describedBy sciM.1  
checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342
checksumAlgorithm SHA-1 SHA-1
accessRule[1]    
.ruleType Allow Allow
.service Read Read
.principal * *
accessRule[2]    
.ruleType Allow Allow
.service Write Write
.principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]    
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Queued Queued
.replicaVerified    

Step B4.

A CN detects new content available on mn1.dataone.org through MN_replication.listObjects().

There is no change of state in system metadata resulting from this operation.

Step B5.

Copies of sciM.1, sysM.1, and sysD.1 are retrieved to the Coordinating Node, and the system metadata objects are updated on the Coordinating Node. For sake of brevity, only the elements from ReplicationPolicy onwards are shown in the remaining state tables since the remainder of the content does not change.

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
...    
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]    
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z

Step B6.

The Coordinating Node initiates replication of the objects sciM.1 and sciD.1 from mn1.dataone.org to mn2.dataone.org. It does this by calling the “replicate()” method on mn2.dataone.org.

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
...    
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T20:58:00.0Z 2010-03-04T20:58:00.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]    
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z
replica[2]    
.replicaMemberNode mn2.dataone.org mn2.dataone.org
.replicationStatus Queued Queued
.replicaVerified    

Step B7.

The node mn2.dataone.org requests the objects sciM.1 and sciD.1 from mn1.dataone.org. After verifying the request with a Coordinating Node (which has a side effect of setting the system metadata ReplicationStatus to Requested), mn1.dataone.org returns the objects. (Note that in actuality, each object is requested individually.)

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
...    
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T20:59:00.0Z 2010-03-04T20:59:00.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]    
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z
replica[2]    
.replicaMemberNode mn2.dataone.org mn2.dataone.org
.replicationStatus Requested Requested
.replicaVerified    

Step B8.

The node mn2.dataone.org verifies the retrieved objects against their checksums and indicates to the Coordinating Node that transfer was successful.

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
...    
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T21:00:00.0Z 2010-03-04T21:00:00.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]    
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z
replica[2]    
.replicaMemberNode mn2.dataone.org mn2.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T21:00:00.0Z 2010-03-04T21:00:00.0Z

Final State of System Metadata

Assuming the replication policy was for one replica to be created for sciM.1 and sciD.1, then the eventual end state of the associated system metadata documents (sysM.1 and sysD.1 respectively) after synchronization of the Coordinating Nodes, would be something like the following.

Note that synchronization between Coordinating Nodes does not trigger an update to the DataSysMetadataModified field. This is necessary to avoid a situation where the state of the metadata is never consistent between Coordinating Nodes.

Note also that even though the replication policy of sciD.1 indicates that NumberReplicas is one, there are actually three additional replicas stored on the Coordinating Nodes.

Field sysD.1 sysM.1
identifier sciD.1 sciM.1
objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0
size 1450238 8132
submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org
obsoletes    
obsoletedBy    
derivedFrom    
describes   sciD.1
describedBy sciM.1  
checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342
checksumAlgorithm SHA-1 SHA-1
accessRule[1]    
.ruleType Allow Allow
.service Read Read
.principal * *
accessRule[2]    
.ruleType Allow Allow
.service Write Write
.principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org
replicationPolicy    
.preferredMemberNode mn2.dataone.org mn2.dataone.org
.blockedMemberNode    
.replicationAllowed true true
.numberReplicas 1 1
dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
dateSysMetadataModified 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z
originMemberNode mn1.dataone.org mn1.dataone.org
authoritativeMemberNode mn1.dataone.org mn1.dataone.org
replica[1]    
.replicaMemberNode mn1.dataone.org mn1.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z
replica[2]    
.replicaMemberNode mn2.dataone.org mn2.dataone.org
.replicationStatus Completed Completed
.replicaVerified 2010-03-04T21:00:00.0Z 2010-03-04T21:00:00.0Z

Conclusions

Maintaining complete copies of system metadata on all nodes is cumbersome and seems to be overly complicated. However, it is still necessary for Member Nodes to provide accurate information for some elements of the system metadata (to assist with transfer verification, to validate a data object is intact).

As such, it seems that scenario B above is the most tractable and provides the desired functionality for tracking content in the DataONE infrastructure. The role of system metadata on Member Nodes and Coordinating Nodes is further clarified below.

The recommendations are:

  • Complete, authoritative copies of system metadata are maintained on the Coordinating Nodes.
  • A system metadata record becomes authoritative once the replicationStatus is set to complete after initial transfer of content to the Coordinating Node (step B5 above)
  • Member Nodes still provide access to system metadata through the MN_crud.getSystemMetadata() calls, though the copy of system metadata returned is authoritative for the properties controlled by the Member Nodes only.
  • Member Nodes should not return any of the SystemMetdata.replica attributes.