.. _UC06: Use Case 06 - MN Synchronize ---------------------------- .. index:: Use Case 06, UC06, MN Synchronize, synchronize Revisions View document revision history_. Goal Science metadata records and system metadata for all objects available on a Member Node are synchronized to a Coordinating Node. Summary As content is added to Member Nodes, the associated system metadata, science metadata, and resource map documents are retrieved by a Coordinating Node. Once the objects are registered with the Coordinating Node, they are considered to be part of the DataONE system, and so will be managed accordingly - the metadata and resource map object will be replicated between Coordinating Nodes, metadata will be indexed, and the objects will be replicated to other Member Nodes as required. Actors Member Node, Coordinating Node Preconditions - CN and MN operational - New content available on a Member Node Triggers - Periodic polling of MNs - Signal to CN issued by a MN that is requesting synchronization Post Conditions - All system and science metadata records from a MN are also present on the CN - The search index is updated with the new content .. image:: images/06_uc.png *Figure 1.* Use case diagram indicating the components and other use cases involved in this use case. .. image:: images/06_seq.png *Figure 2.* Sequence diagram for use case 06. Synchronize content between Member and Coordinating Node. Detail for the *processObject()* step provided in the activity diagram below. .. image:: images/06_act.png *Figure 3.* Activity diagram indicating the execution flow after attempting to retrieve the system metadata for the object from the CN. .. @startuml images/06_uc.png package "DataONE" actor "Coordinating Node" as CN actor "Member Node" as MN usecase "13. Authorization" as author usecase "06. Synchronize Metadata" as SYNC usecase "XX. Notify Watchers" as NOTIFY CN -- SYNC MN -- SYNC SYNC ..> author: <> SYNC ..> NOTIFY: <> @enduml .. @startuml images/06_seq.png skinparam notebordercolor #AAAAAA skinparam notefontcolor #222222 participant "Repl Task Queue" as cn_repl << Cluster >> participant "Replication" as cn_replication << CN >> participant "Index Task Queue" as cn_index << Cluster >> participant "Indexer" as cn_indexer << CN >> participant "Object Store" as cn_objs << CN >> participant "System Metadata Map" as cn_sysmeta << Cluster >> participant "Sync Task Queue" as cn_queue << CN >> participant "Node Map" as cn_nodes << Cluster >> participant "Synchronization" as cn_sync << CN >> participant "Read API" as mn_read << MN >> 'm_crud -> c_notify: notify(session, PID, OBJECT_CREATED) 'c_notify -> cn_queue: addTask(SyncTask, node, PID) 'note right ' notification triggered by successful ' create operation on MN. 'end note 'm_rep -> c_notify: setReplicationStatus(token, PID, COMPLETE) 'c_notify -> cn_queue: addTask(SyncTask, node, PID) 'note right ' notification triggered by completed ' replication operation on MN. 'end note group populateSynchronizationQueue cn_sync -> cn_nodes: lock(node_id) activate cn_sync #D74F57 note right Start of synchronization process triggered by quartz end note activate cn_nodes #D74F57 cn_nodes --> cn_sync: OK cn_sync -> cn_nodes: getLastUpdateTime(node_id) activate cn_nodes #D74F57 note right The Node Map is a hash of (node_id, Node), use Hazelcast query Map.values() passing in SqlPredicate end note cn_nodes --> cn_sync: startTime deactivate cn_nodes cn_sync -> mn_read: listObjects(session, startTime, ...) activate mn_read #D74F57 cn_sync <-- mn_read: ObjectList deactivate mn_read loop for each PID cn_sync -> cn_sync: createTask(PID) note right Each SyncTask implements Callable and will be submitted to the ExecutorService to be executed on a CN end note cn_sync -> cn_queue: offer(taskid, SyncTask) activate cn_queue #D74F57 cn_queue --> cn_sync: OK deactivate cn_queue end note right adding SyncTasks should fail if PID is already in the list and the PID is NOT locked. This enables very recent updates to a PID to occur and be correctly managed by the overall synchronization process. end note cn_sync -> cn_nodes: setLastUpdateTime(node_id, startTime) activate cn_nodes #D74F57 cn_nodes --> cn_sync: OK deactivate cn_nodes cn_sync -> cn_nodes: unlock(node_id) deactivate cn_sync deactivate cn_nodes end group processSynchronizationQueue cn_sync -> cn_sync: entryAdded(EntryEvent) note left Synchronization implements EntryListener, monitors the Sync Queue for changes. end note activate cn_sync #D74F57 cn_sync -> cn_queue: poll(timeout) activate cn_queue #D74F57 note right Only one CN will win the poll and process the SyncTask end note cn_queue --> cn_sync: SyncTask deactivate cn_queue cn_sync -> cn_sync: ExecutorService.submit(SyncTask) cn_sync -> cn_sysmeta: lock(PID) activate cn_sysmeta #D74F57 group ProcessPID( PID ) cn_sync -> mn_read: getSystemMetadata(PID) mn_read -> cn_sync: SystemMetadata cn_sync -> mn_read: get( PID ) mn_read -> cn_sync: object cn_sync -> cn_sync: work note right Check for new object, updates to properties end note end cn_sync -> cn_objs: createOrUpdate(session, PID, object, SystemMetadata) activate cn_objs #D74F57 cn_indexer -> cn_indexer: entryAdded() activate cn_indexer #D74F57 note left Indexer implements EntryListener, monitors the System Metadata Map for inserts, updates, deletes. end note cn_indexer -> cn_indexer: createTask(PID) note left Each IndexTask implements Callable and will be submitted to the ExecutorService to be executed on the local CN end note cn_indexer -> cn_index: offer(taskid, IndexTask) activate cn_index #D74F57 cn_objs --> cn_sync: OK deactivate cn_objs cn_sync -> cn_sysmeta: unlock(PID) deactivate cn_queue deactivate cn_sysmeta cn_replication -> cn_replication: entryAdded() activate cn_replication #D74F57 note left Indexer implements EntryListener, monitors the System Metadata Map for inserts, updates, deletes. end note cn_replication -> cn_replication: createTask(PID) note left Each ReplTask implements Callable and will be submitted to the ExecutorService to be executed on the a CN end note cn_replication -> cn_repl: offer(taskid, ReplTask) activate cn_repl #D74F57 deactivate cn_queue end deactivate cn_sync @enduml .. @startuml images/06_act.png (*) --> [processObject()] if "CNRead.getSystemMetadata(PID)" then if "Duplicate Object?" then -->[Yes] "Duplicate Content" --> "Update Replica Info" --> "Store System Metadata" --> "Notify watchers\nIndex, Replication" --> (*) else -->[No] "Error: Duplicate Identifier" --> "Notify MN" --> (*) endif else ->[FAIL] "New content" note right The content is unknown to DataONE so needs to be examined, its location recorded, and optionally retrieved and stored on the CN. end note if "Science Metadata \n or Resource Map?" then ->[yes] "Get object from MN" --> "Store object on CN" --> "Update Replica Info" else -->[no] "Update Replica Info" endif endif @enduml Implementation ~~~~~~~~~~~~~~ The Member Node synchronization process will operate in an asynchronous manner, with a task queue on the CN service containing a list of objects that need to be added to the coordinating node populated through two mechanisms: 1. scheduled calls to :func:`MNRead.listObjects` 2. through a new notification API that is called by a Member Node after a successful create operation. The task queue is processed by a pool of workers which retrieve the content using :func:`MNRead.get` and :func:`MNRead.getSystemMetadata`, and store the content in the Coordinating Node using the internal :func:`CNStorage.create` method. The initial implementation will focus on using only scheduled calls to :func:`MNRead.listObjects` to populate the task queue. The scheduling information shall be set during the MN registration process. Multiple threads should be able to process the task queue, and the number of threads should be adjustable by an administrator. The total number of threads hitting on a particular MN should be limited, the limit should be specified in the MN capabilities recorded during the registration process. The tasks queue will need to be thread safe. In the future it may be feasible to use a distributed queue (based on Apache MQ or Zookeeper for example; shared across the CNs) so that processes on any CN can issue the :func:`MN_get` and :func:`MN_getSystemMetadata` requests. There will likely be a significant load on the system for indexing as new content is being added - this is the case for both Mercury and Metacat. Options for disabling indexing for bulk load should be considered. The synchronization process will certainly be more sophisticated down the road, but for now (i.e. prototype implementation) simplicity should be a priority: - MNs are only scanned by the CN where they are registered (restriction to be dropped down the road). - listObjects is polled (with max objects set to some limit), with results stored in a queue on the CN - as the queue is processed and shrinks to some size, then listObjects process is polled again - the queue is processed by n worker threads, with each task being to pop an entry from the queue, pull the content from the target, and store it in the object store. - when there is some number of new entries in the object store, then the mercury indexing process needs to kick in. Ideally this should be updating rather than re-creating the index at each pass. This may be initiated by a cron job for the prototype implementation. This fairly simplistic approach should be enough to get things started. Not ideal, but should suffice to get some data moving around. To implement, there is need for a few new components - a queue, a place to store state information, the code that does the polling, the code that does the object retrieval, the worker thread code, and an overall controller service. In more detail .............. Retrieve objects from Member Nodes. getNextMemberNode() gets the next Member Node to work on - this could be serial selection from the list of registered MNs, could be based on the last update, perhaps from some hint returned in :func:`MN_health.ping`. New structures:: MemberNodeState String url; //URL of the member node DateTime lastCheckTime; // time stamp for when the MN was last polled for listOjects RetrievalTask String url; ObjectInfo objectInfo; Populating the harvest task queue pseudo code:: void populateTaskQueue () { ThreadSafeQueue taskQueue = getSharedTaskQueue() while ( keepRunning() ) { if (taskQueue.length < MIN_TASKS) { MemberNodeState = getNextMemberNode(); // This could be executed in a different thread, which would // enable multiple MNs to be polled at a tme if necessary int numNewEntries = loadMNTasks(taskQueue, mnState); } } } Processing the harvest task queue pseudo code:: int loadMNTasks(ThreadSafeQueue taskQueue, MemberNodeState mnState) { AuthToken token = getSystemToken(); D1cient client = D1Client( mnState.url ); // get a list of all the new entries since lastCheckTime DateTime tstamp = now(); // Will actually need to loop here to page through results ObjectList objectList = client.listObjects(token, mnState.lastCheckTime); for (ObjectInfo objinfo : objectList) { RetrievalTask task = RetrievalTask(mnState.url, objinfo) taskQueue.push(task) } mnState.lastCheckTime = tstamp; setMemberNodeState(mnState); return objectList.length; } Worker threads are responsible for retrieving each object listed in the taskQueue. The number of workers should be adjustable. An obvious target for improving efficiency is to enable requesting multiple objects in a single call to a MN (later enhancement):: // object is the next entry from the taskQueue bool retrieveObject(String mnurl, ObjectInfo object) { Authtoken token = getSystemToken(); D1Client client = D1Client(mnurl); SystemMetadata sysmeta = client.getSystemMetadata( object.identifier ); if ( isScienceMetadata(sysmeta.objectClass) ) { scimeta = client.get( object.identifier ); CN_SERVICE.create( object.identifier, sysmeta, scimeta ); } else { CN_SERVICE.create( object.identifier, sysmeta ); } } .. _history: https://redmine.dataone.org/projects/d1/repository/changes/documents/Projects/cicore/architecture/api-documentation/source/design/UseCases/06_uc.txt