Extension Points

There are several extension points for the File Manager. An extension point is an interface within the file manager that can have many implementations. This is particularly useful when it comes to software component configuration because it allows different implementations of an existing interface to be selected at deployment time. So, the File Manager component may communicate with a Database-based Catalog, and an XML-based Element Store (called a Validation Layer), or it may use a Lucene-based Catalog and a Database-based Validation Layer. The selection of the actual component implementations is handled entirely by the extension point mechanism. Using extension points, it is fairly simple to support many different types of what are typically referred to as “plug-in architectures” Each of the core extension points for the File Manager is described below:

The relat ionships between the extension points for the File Manager are shown in the below Figure.

Key Capabilities

The File Manager is responsible for providing the necessary key capabilities for managing files and metadata. Each high level capability provided by the File Manager is detailed below:

1. Easy Management of different types of Products – The Repository Manager extension point is responsible for managing Product Types, and their associated information. Management of Product Types includes adding new ones, deleting and updating existing ones, and retrieving Product Types, by their ID or by their name.
2. Support for different kinds of back end catalogs – The Catalog extension point allows Product instance metadata and file location information to be stored in different types of back end data stores quite easily. Existing implementations of the Catalog interface include a JDBC based backend database, along with a flat-file based, Lucene index.
3. Management of Product instance information – The management includes adding, deleting and updating product instance information, including file locations (References), along with Product Metadata. It also includes getting Metadata, and getting References associated with existing Products. It also includes obtaining the Products themselves.
4. Separating out the Element management layer for Metadata – The File Manager Validation Layer extension points allows for the management of Element policy information in different types of back end stores. For instance, Element policy could be stored in XML files, a Database, or even a Metadata Registry.
5. Supporting different Data Transfer Mechanisms – By having an extension point for Data Transfer, the File Manager can support different Data Transfer protocols, both local and remote.
6. Allowing for different Back End File Repository Layouts – The Versioner extension points allows for different File Repository Layouts based on Product Types.
7. Allowing for Hierarchical collections of files and directories making up a Product – The File Manager Client allows for Products to be Flat, or Hierarchical-based. Flat products are collections of singular files that are aggregated together to make a Product. Hierarchical Products are Products that contain collections of directories, and sub-directories, and files.
8. Scalability – The File Manager uses the popular client-server paradigm, allowing new File Manager servers to be instantiated, as needed, without affecting the File Manager clients, and vice-versa.
9. Communication over lightweight, standard protocols – The File Manager uses XML-RPC, as its main external interface, between File Manager client and server. XML-RPC, the little brother of SOAP, is fast, extensible, and uses the underlying HTTP protocol for data transfer.
10. RSS based Product Syndication – The File Manager web interface allows for the RSS-based syndication of Product feeds based on Product Type.
11. Data Transfer Status Tracking – The File Manager tracks all current Product and File transfers and even publishes an RSS-feed of existing transfers.

This capability set is not exhaustive, and is meant to give the user a “feel” for what general features are provided by the File Manager. Most likely the user will find that the File Manager provides many other capabilities besides those described here.

Current Extension Point Implementations

There are at least two implementations of all of the aforementioned extension points for the File Manager. Each extension point implementation is detailed below:

• Catalog
  
  
  1. Data Source based Catalog – an implementation of the Catalog extension point interface that uses a JDBC accessible database backend.
  2. Lucene based Catalog – an implementation of the Catalog extension point interface that uses the Lucene free text index system to store Product instance information.
• Data Transfer
  
  
  1. Local Data Transfer – an implementation of the Data Transfer interface that uses Apache’s commons-io to perform local, disk based filesystem data transfer. This implementation also supports locally accessible Network File System (NFS) disks.
  2. Remote Data Transfer – an implementation of the Data Transfer interface that uses the XML-RPC File Manager client to transfer files to a remote XML-RPC File Manager server.
  3. InPlace Data Transfer - an implementation of the Data Transfer interface that avoids transfering any products -- this can be used in the situation where metadata about a particular product should be recorded, but no physical transfer needs to occur.
• Repository Manager
  
  
  1. Data Source based Repository Manager – an implementation of the Repository Manager extension point that stores Product Type policy information in a JDB C accessible database.
  2. XML based Repository Manager – an implementation of the Repository Manager extension point that stores Product Type policy information in an XML file called product-types.xml
• Validation Layer
  
  
  1. Data Source based Validation Layer – an implementation of the Validation Layer extension point that stores Element policy information in a JDBC accessible database.
  2. XML based Validation Layer – an implementation of the Validation Layer extension point that stores Element policy information in 2 XML files called elements.xml and product-type-element-map.xml
• System (File Manager client and File Manager server)
  
  
  1. XML-RPC based File Manager server – an implementation of the external server interface for the File Manager that uses XML-RPC as the transportation medium.
  2. XML-RPC based File Manager client – an implementation of the client interface for the XML-RPC File Manager server that uses XML-RPC as the transportation medium.

Project Organization

The cas-filemgr project follows the traditional Subversion-style trunk , tag and branches format. Trunk corresponds to the latest and greatest development on the cas-filemgr. Tags are official release versions of the project. Branches correspond to deviations from the trunk large enough to warrant a separate development tree.

For the purposes of this the User Guide, we'll assume you already have downloaded a built release of the file manager, from its web site. If you were building cas-filemgr from the trunk, a tagged release (or branch) the process would be quite similar. To build cas-filemgr, you would need the Apache Maven software. Maven is an XML-based, project management system similar to Apache Ant, but with many extra bells and whistles. Maven makes cross-platform project development a snap. You can download Maven from: http://maven.apache.org The cas-filemgr is constructed to be compatible with the 1.x.x series of Maven, despite Maven 2.x having already been released. Once you have Maven installed, follow the procedures in the below Sections to build a fresh copy of the File Manager:

Building the File Manager

1. cd to cas-filemgr, and then type:

   # maven dist

   This will pe rform several tasks, including compiling the source code, downloading required jar files, running unit tests, and so on. When the command completes, cd to the target/distributions directory within cas-filemgr. This will contain the build of the File Manager component, of the following form:

               cas-filemgr-${version}.tar.gz
              

   This is a distribution tar ball, that you would copy to a deployment directory, such as /usr/local/ , and then unpack using # tar xvzf . The resultant directory layout from the unpacked tarball is as follows:

               bin/ etc/ logs/ docs/ lib/ policy/ LICENSE.txt CHANGES.txt
              

   • bin - contains the "filemgr" server script, and the "filemgr-client" client script.
   • etc - contains the logging.properties file for the File Manager, and the filemgr.properties file used to configure the server options.
   • logs - the default directory for log files to be written to.
   • docs - contains Javadoc documentation, and user guides for using the particular CAS component.
   • lib - the required Java jar files to run the File Manager.
   • policy – the default XML-based element and product type policy in case the user is using the XML Repository Manager and/or the XML Validation Layer.
   • CHANGES.txt - contains the CHANGES present in this released version of the CAS component.
   • LICENSE.txt - the LICENSE for the File Manager project.

Deploying the File Manager

To deploy the file manager, you'll need to create an installation direct ory. Typically this would be somewhere in /usr/local (on *nix style systems), or C:\Program Files\ (on windows style systems). We'll assume that you're installing on a *nix style system though the Windows instructions are quite similar.

Follow the process below to deploy the File Manager:

1. Copy the binary distribution to the deployment directory

   # cp -R cas-filemgr/trunk/target/distributions/cas-filemgr-${version}.tar.gz /usr/local/

2. Untar the distribution

   # cd /usr/local ; tar xvzf cas-filemgr-${version}.tar.gz

3. Set up a symlink

   # ln -s /usr/local/cas-filemgr-${version} /usr/local/filemgr

4. edit /usr/local/filemgr/bin/filemgr
   • Set the SERVER_PORT variable to the desired port you'd like to run the File Manager server on.
   • Set the JAVA_HOME variable to point to the location of your installed JRE runtime.
   • Set the RUN_HOME variable to point to the location you'd like the File Manager PID file written to. Typically this should default to /var/run , but not all system administrators allow users to write to /var/run .
5. edit /usr/local/filemgr/bin/filemgr-client
   • Set the JAVA_HOME variable to point to the location of your installed JRE runtime.
6. (optional) edit /usr/local/filemgr/etc/logging.properties
   • Set the logging levels for each subsystem to the desired level. The system defaults are fairly considerate and prevent much of the logging at levels b elow INFO to the console.
7. edit /usr/local/filemgr/etc/filemgr.properties
   • This java properties file contains all of the default information properties to configure the File Manager. By default, the File Manager is built to use the XML-based repository manager and validation layer extension points, the DataSource based catalog extension point, and the local data transfer interface. These defaults can be changed quite easily by changing the factory classes that are pointed to for each extension point. For example, to use the Lucene-based cataog extension point, you would change the following property, filemgr.catalog.factory to gov.nasa.jpl.oodt.cas.filemgr.catalog.LuceneCatalogFactory
   • You need to configure the properties for each of the extension points that you are using. By default, you would at least need to configure:
     • The JDBC connection information for the data source catalog.
     • The paths to the directories where the XML policy files are stored for the validation layer and for the repository manager. A good default location is to place these files within /usr/local/filemgr/policy.

Other configuration options are possible: check the API documentation , as well as the comments within the filemgr.properties file to find out the rest of the configurable properties for the extension points you choose. A full listing of all the extension point factory class names are provided in the Appendix. After step 7, you are officially done configuring the File Manager for deployment.

Running the File Manager

To run the filemgr, cd to /usr/local/filemgr/bin and type:

# ./filemgr start

This will startup the file manager XML-RPC server interface. Your File Manager is now ready to run! You can test out the file manager by running a simple ingest command using the filemgr-client command below. First create a simple text file called "blah.txt" and place it inside /usr/local/filemgr/bin. Then, create a blank metadata file for the product, using the schema or DTD provided in the cas-metadata project. An example XML file might be:

        <cas:metadata xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
        </cas:metadata>
      

Call this metadata file blah.txt.met , and place it also in /usr/local/filemgr/bin . Then, run the below command, assuming that you started the File Manager on the default port of 9000 :

# ./filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName Blah.txt \
      --productStructure Flat --productTypeName GenericFile --metadataFile file:/usr/local/filemgr/bin/blah.txt.met \
      --clientTransfer --dataTransfer gov.nasa.jpl.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory \
      --refs file:/usr/local/filemgr/bin/blah.txt
      

You should see a response message at the end similar to:

      Jul 15, 2006 10:37:53 PM gov.nasa.jpl.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
      INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
      Jul 15, 2006 10:37:54 PM gov.nasa.jpl.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct
      FINEST: File Manager Client: clientTransfer enabled: transfering product [Blah.txt]
      Jul 15, 2006 10:37:54 PM gov.nasa.jpl.oodt.cas.filemgr.versioning.VersioningUtils 
      createBasicDataStoreRefsFlat
      FINE: VersioningUtils: Generated data store ref: file:/tmp/files/Blah.txt/blah.txt from 
      origRef: file:/usr/local/filemgr/bin/blah.txt
      Jul 15, 2006 10:37:54 PM gov.nasa.jpl.oodt.cas.filemgr.datatransfer.LocalDataTransferer 
      moveFilesToProductRepo
      INFO: LocalDataTransfer: Moving File: file:/usr/local/filemgr/bin/blah.txt to 
      file:/tmp/files/Blah.txt/blah.txt
      ingestProduct: Result: 3a812d86-148d-11db-a25a-f388f524a371
      

which means that everything installed okay!