Data Element Definition

For the data element definition the user will define metadata elements that will be used to describe the products registered with the CAS. A data element definition will represent a column in the product metadata table for an associated dataset. Dataset associations are discussed in the Dataset Definition section. Each data element must have an entry in the element_policy table. The supported columns in this table are described below:

The following is an example of SQL for Oracle representing the Test.String data element definition:

VARIABLE next_id NUMBER

EXEC get_next_element_id (:next_id);

INSERT INTO element_policy 
   (element_id, element_name, product_metadata_column,
    dc_element, data_type, length,
    description) VALUES
   (:next_id, 'Test.String', 'Test_String',
    NULL, 'STRING' 30,
    'Test element for a string value.');

COMMIT;
            

Task Definition

For the task definition the user will define tasks to be executed once a product has been registered with the CAS. For information on task development see the Task/Rule Development section of this document. Each task must have an entry in the task_policy table. The columns in this table are described below:

The following is an example of SQL for Oracle representing the NoOpTask task definition:

VARIABLE next_id NUMBER

EXEC get_next_task_id (:next_id);

INSERT INTO task_policy 
   (task_id, task_name,
    description) VALUES
   (:next_id, 'jpl.eda.archive.server.NoOpTask',
    'This task does absolutely nothing.');

COMMIT;
            

Rule Definition

For the rule definition the user will define rules to be executed prior to the execution of a task. For information on rule development see the Task/Rule Development section of this document. Each rule must have an entry in the rule_policy table. The columns in this table are described below:

The following is an example of SQL for Oracle representing the NoOpRule rule definition:

VARIABLE next_id NUMBER

EXEC get_next_rule_id (:next_id);

INSERT INTO rule_policy 
   (rule_id, rule_name,
    description) VALUES
   (:next_id, 'jpl.eda.archive.server.NoOpRule',
    'This rule does absolutely nothing.');

COMMIT;
            

Dataset Definition

For the dataset definition the user will define the product to be registered with the CAS. Each dataset must have an entry in the dataset_policy table and may have entries in the dataset_element_relation , dataset_task_relation and dataset_task_rule_relation tables. The policy for the dataset defines how the CAS will handle products registered with this dataset. The columns in the dataset_policy table are described below:

The following is an example of SQL for Oracle representing the Test.Metadata.Tasks dataset definition for the dataset_policy table:

VARIABLE next_id NUMBER

EXEC get_next_dataset_id (:next_id);

INSERT INTO dataset_policy
   (dataset_id, dataset_name,
    product_metadata_table, product_reference_table, repository,
    description) VALUES
   (:next_id, 'Test.Metadata.Tasks',
    NULL, NULL, NULL,
    'Test dataset with associated metadata and tasks.');

UPDATE dataset_policy SET 
   product_metadata_table =
      'product_metadata_' || to_char(dataset_id),
   product_reference_table =
      'product_reference_' || to_char(dataset_id),
   repository =
      '/home/user/repository/product_' || to_char(dataset_id)
WHERE dataset_name = 'Test.Metadata.Tasks';

COMMIT;
            

The data element relationship policy for the dataset defines which data elements the CAS will accept as metadata for a product registered with this dataset. The supported columns in the dataset_element_relation table are described below:

The following is an example of SQL for Oracle representing the Test.Metadata.Tasks dataset/data element relationship definition for the dataset_element_relation table:

INSERT INTO dataset_element_relation
   (dataset_id, element_id, index_flag)
   (SELECT dataset_id, element_id, , 'N'
    FROM dataset_policy, element_policy 
    WHERE dataset_name = 'Test.Metadata.Tasks'
    AND element_name = 'Test.String');

COMMIT;
            

The task relationship policy for the dataset defines which tasks to execute for a product registered with this dataset. The columns in the dataset_task_relation table are described below:

The following is an example of SQL for Oracle representing the Test.Metadata.Tasks dataset/task relationship definition for the dataset_task_relation table:

INSERT INTO dataset_task_relation
   (dataset_id, task_id,
    event_exec_flag, event_priority, event_fail_flag,
    queue_exec_flag, queue_priority, wait_length)
   (SELECT dataset_id, task_id, 'Y', 1, 'N', 'Y', 1, 0
    FROM dataset_policy, task_policy
    WHERE dataset_name = 'Test.Metadata.Tasks'
    AND task_name = 'jpl.eda.archive.server.NoOpTask');

COMMIT;
            

The rule relationship policy for the task and dataset defines which rules to execute prior to a task being executed as the result of a product registered with this dataset. The columns in the dataset_task_relation table are described below:

The following is an example of SQL for Oracle representing the Test.Metadata.Tasks dataset/task/rule relationship definition for the dataset_task_rule_relation table:

INSERT INTO dataset_task_rule_relation
   (dataset_id, task_id, rule_id, rule_exec_flag, rule_priority,
    rule_type, wait_length)
   (SELECT dataset_id, task_id, rule_id, 'Y', 1, 'MANDATORY', 0
    FROM dataset_policy, task_policy, rule_policy
    WHERE dataset_name = 'Test.Metadata.Tasks'
    AND task_name = 'jpl.eda.archive.server.NoOpTask'
    AND rule_name = 'jpl.eda.archive.server.NoOpRule');

COMMIT;
            

Interfaces

A Java class representing a task or rule to be executed by the CAS, must conform to the Task or Rule interface, respectively. The two interfaces are very similar in that they consist of a single run() method with the only differences being that the Task interface passes an additional execId parameter and the Rule interface returns a boolean value (true or false) indicating whether the rule was satisfied or not. The common parameters are as follows:

The source distribution for the CAS includes two template classes, NoOpTask and NoOpRule , which provide a blank slate for developing tasks and rules, respectively. Each of these classes includes an empty run() method as well as a main() method for executing the class from the command-line. Although the two interfaces are very similar, tasks and rules differ in the manner and situation that they are executed by the CAS which is dictated by the policy. The conceptual difference between the two is that a rule should not modify data in any way, whereas a task is intended to do just that. The subsequent sections provide Java code examples for common functions which can be performed in either tasks or rules.

Message Logging

The CAS has a built in logging mechanism which allows tasks and rules to generate messages for inclusion in the log. See the Catalog and Archive Service section of the Operation Guide for more information regarding the logging properties. The logging mechanism supports five categories of messages: Trace , Debug , Info , Warn and Error . The Log class from the edm-commons component is utilized for generating messages.

Log.get("Error").println(
   "MyTask.run(): An exception occurred doing something.");
            

The above code will result in the following message being posted to the log, which includes a date/time stamp, the CAS URN and the specified category:

2005-04-01T12:30:45.000-08:00 urn:eda:rmi:JPL.Test.Archive Error: \
MyTask.run(): An exception occurred doing something.
            

Retrieve Product Metadata

Each registered product may have associated metadata elements as defined in the CAS policy. A map of the element/value pairs or a list of the elements and their policy can be retrieved from the Product class. If the task or rule is executed during the product registration transaction, the element values will only be available if the metadata was passed in via the metadata buffer and not available if passed in via a metadata file. All metadata information is available if the task or rule is executed from the queue. In the following example, the value associated with the task specific element Project.Name , is extracted from the map:

HashMap map = product.getProductMetadata();
String projectName = (String) map.get("Project.Name");
            

The example above returns the value as a string even if the element has a data type of integer or some other type. The example below utilizes the ElementList and ElementAttrInfo classes to provide a means for returning the appropriately typed value based on the CAS policy.

ElementList elementList = product.getElementList();
ElementAttrInfo elementAttrInfo = 
   elementList.getElementAttrInfo("Project.Name");
String projectName = elementAttrInfo.getStringValue();
            

Update Product Metadata

Several scenarios for task development involve determining additional metadata for a product and updating that metadata in the database. This can be accomplished utilizing the MetadataElementList class. In the following example, two metadata elements are added to the list with their associated values and then the database is updated for the specified product:

try {
   MetadataElementList metadataList = 
      new MetadataElementList(db, dataset, product.getProductId());
   metadataList.addElement("Project.Name", "MyProject");
   metadataList.addElement("Project.Id", "1000");
   metadataList.addToDatabase();
}
catch (MetadataElementListException e) {}
            

In the example above, notice that the Project.Id element value is passed in as a string when the poli cy dictates a data type of integer. The MetadataElementList class determines the data type of the element based on the CAS policy and inserts it into the database appropriately.

The alternative is to update the metadata element values directly in the product metadata table using SQL. Although the previous example is preferred, this example demonstrates the flexibility available to the developer. In the following example, SQL is utilized to update the product metadata table where the table name is extracted from the dataset policy and the column names are extracted from the element policy (SQL reserved words in the SQL command are capitalized for clarity):

try {
   ElementList elementList = dataset.getElementList();
   ElementAttrInfo elemName = 
      elementList.getElementAttrInfo("Project.Name");
   ElementAttrInfo elemId = 
      elementList.getElementAttrInfo("Project.Id");
   String projectName = "MyProject";
   int projectId = 1000;

   String sqlCmd = 
      "UPDATE " + dataset.getMetadataTable() + " SET " + 
      elemName.getColumn() + " = '" + projectName + "', " + 
      elemId.getColumn() + " = " + projectId + 
      " WHERE product_id = " + product.getProductId();
   Log.get("Debug").println("MyTask.run(): SQL: " + sqlCmd);

   db.executeSQLCommand(sqlCmd);
}
catch (SQLException e) {}
            

Retrieve Product References

Each registered product may have a list of product items or files associated with the product. This list is represented in the ProductList class which can be retrieved from the Product class. In the following example, the file name is extracted from the product reference for each remote file registered with the product:

try {
   ProductList productList = product.getProductList();
   for (int i = 0;  i < productList.getCount(); i++) {
      String productRefType = productList.getProductRefType(i);
      if (productRefType.equals("FILE_DATA_REMOTE")) {
         String fileSpec = 
            new URL(productList.getProductRef(i)).getPath();
         String fileName = 
            fileSpec.substring(fileSpec.lastIndexOf('/') + 1);
      }
   }
}
catch (MalformedURLException e) {}
            

If the product item was transferred to the CAS, (reference types FILE_DATA_LOCAL and FILE_METADATA ), then the archive reference can be obtained from the ProductList class with the getArchiveRef() method. This reference specifies the location of the file in the CAS repository.

Query for Products

Several scenarios for task and rule development involve determining whether another product is currently registered with the CAS. This other product could be registered with the current product's dataset or with another dataset. In the following example, information about another dataset is determined and the other product is queried for from that dataset's product metadata table:

try {
   Dataset otherDataset = new Dataset(db, "Dataset.Name");
   ElementList elementList = otherDataset.getElementList();
   ElementAttrInfo elemName = 
      elementList.getElementAttrInfo("Project.Name");
   String projectName = "MyProject";

   String sqlCmd = 
      "SELECT product_id FROM " + otherDataset.getMetadataTable() + 
      " WHERE " + elemName.getColumn() + " = '" + projectName + "'";

   ResultSet rs = db.executeQuery(sqlCmd);
   int otherProductId;
   int count = 0;
   while (rs.next()) {
      productId = rs.getInt(1);
      count++;
   }

   if (count < 1) {
      // Problem: no product found
   }
   else if (count > 1) {
      // Problem: too many products found
   }

   Product otherProduct = new Product(db, otherDataset);
   otherProduct.load(otherProductId);
}
catch (DatasetException e) {}
catch (SQLException e) {}
catch (ProductException e) {}
            

If this were a rule, the product was found or not and the rule was satisfied or not. If this were a task, the developer may want to do something with the discovered product or investigate that product's metadata further.

Execute Command

A very common scenario involves executing an external command, whether it be a Perl script or an application of some sort, from a task. In the following example, the command is constructed using a metadata element value as an argument and the command is executed:

try {
   HashMap map = product.getProductMetadata();
   String projectName = (String) map.get("Project.Name");

   String execCmd = "/usr/local/bin/task_command " + projectName;
   Log.get("Debug").println(
      "MyTask.run(): Execution Command: " + execCmd);

   Process process = Runtime.getRuntime().exec(execCmd);
}
catch (Exception e) {}
            

Exception Handling

In each of the examples above, an attempt has been made to detail the appropriate exception to be caught for the snippet of code provided. This section details a suggested method for handling most exceptions. There are three typical steps to perform:

• Generate and log a message for the caught exception.
• Generate and log a message describing what was occurring when the exception was thrown.
• Throw a TaskException or RuleException with the message and exception object as parameters.

catch (Exception e) {
   Log.get("Error").println(
      "MyTask.run(): Exception '" + e.getClass().getName() + 
      "' occurred with message '" + e.getMessage() + "'");
   String message = 
      "MyTask.run(): An exception occurred doing something.";
   Log.get("Error").println(message);
   throw new TaskException(message, e);
}
            

Activity Storage

Maybe this section will find its way to the Common Components documentation. Until then, it will reside here. When utilzing the DatagramLogger application for storing activities, there are a number of storage mechanisms available. This section will focus on two of them.

XMLStandardOutputStorage

The XMLStandardOutputStorage class captures the activity from the DatagramLogger application and sends an XML formatted representation to the terminal's standard output stream. The following example represents an activity with a single ActivityStopped incident.

<?xml version="1.0" encoding="UTF-8"?>
<activity id="d2da241b7831ddd14e3549b52808f6e">
  <jpl.eda.activity.ActivityStopped time="1104772913440">
    jpl.eda.activity.ActivityStopped
  </jpl.eda.activity.ActivityStopped>
</activity>
            

SQLDatabaseStorage

The SQLDatabaseStorage storage class captures the activity from the DatagramLogger application and sends it to the specified database. This class expects the following table to exist in the specified database:

The incidents table stores activity incidents. Data types used in the column descriptions below are specific to the Oracle Database Management System (DBMS). This table consists of the following columns:

• activityID
  The generated identifier for the activity. This column is part of the primary key for the table. Data type: VARCHAR2(32).
  
• className
  The name of the incident class. This column is part of the primary key for the table. Data type: VARCHAR2(255).
  
• occurTime
  The time the incident occurred, represented as the number of milliseconds since January 1, 1970, 00:00:00 GMT. Data type: LONG.
  
• detail
  The details associated with the incident. The CAS specific incident classes represent the incident details in an XML structure. Data type: CLOB.
  

In order to create the incidents table in an Oracle Database, perform the following SQL commands using sqlplus or another appropriate tool:

CREATE TABLE incidents (
   activityID     VARCHAR2(32)  NOT NULL,
   className      VARCHAR2(255) NOT NULL,
   occurTime      LONG          DEFAULT 0 NOT NULL,
   detail         CLOB          NULL
);

ALTER TABLE incidents
   ADD CONSTRAINT incidents_pk
   PRIMARY KEY (activityID, className, occurTime);
            

The above syntax should be suppported by any of the database vendors supported by this class. In most cases the data types will need to be modified. For example, the data types for each of the columns in a PostgreSQL database would be VARCHAR(32) , VARCHAR(255) , BIGINT and TEXT respectively.

Activity Retrieval

Maybe this section will also find its way to the Common Components documentation. Until then, it will reside here. There is currently only one retrieval mechanism available. The SQLDatabaseRetrieval class retrieves the activities from the specified database. This class accesses the same table that is described in the Activity Storage section above. The activities are returned as a list of StoredActivity classes. This class in turn provides access to the activit y identifier and the list of associated incidents.

CAS Specific Incidents

Although the Activity Tracking capability is generic, the classes representing the incidents that occur within the CAS for each activity, are specific to the CAS.

ClientInfo

The ClientInfo incident class captures information about the client gathered by the ArchiveClient class and sent to the CAS server at the start of each event. The detailed pieces of information captured by this incident are identified by element names which are defined in the following table:

The details are stored in an XML structure. The following is an example of this structure:

<detail>
  <userName>shardman</userName>
  <hostName>avatar.jpl.nasa.gov</hostName>
  <hostAddress>137.79.16.217</hostAddress>
  <osName>Mac OS X</osName>
</detail>
            

EventBegin

The EventBegin inc ident class captures information about the reqeusted event. The detailed pieces of information captured by this incident are identified by element names which are defined in the following table:

The details are stored in an XML structure. The following is an example of this structure:

<detail>
  <object>urn:eda:rmi:JPL.Test.Archive</object>
  <event>ADD</event>
</detail>
            

EventRequest

The EventRequest incident class captures identifying information for the requested event. This incident is not captured for the QUEUE and PING events. The detailed pieces of information captured by this incident are identified by element names which are defined in the following table:

The details are stored in an XML structure. The following is an example of this structure:

<detail>
  <query>
    CAS.ProductType = Test.Metadata.NoTasks AND 
    CAS.ProductName = Test
  </query>
</detail>
            

EventResult

The EventResult incident class captures result information for the event. This incident is only captured for the ADD , GET and GETPROFILES events. The detailed pieces of information captured by this incident are identified by element names which are defined in the following table:

The details are stored in an XML structure. The following is an example of this structure:

<detail>
  <count>3</count>
  <length>3575</length>
</detail>
            

EventException

The EventException incident class captures exception information for the event. Existence of this incident may preclude the existence of other expected incidents for a given activity. The detailed pieces of information captured by this incident are identified by element names which are defined in the following table:

The details are stored in an XML structure. The following is an example of this structure:

<detail>
  <exception>jpl.eda.archive.server.ProductException</exception>
  <message>
    Product.add(): The product name ':Test' is not a valid name.
  </message>
</detail>