Difference between revisions of "Installing and Configuring LDDTool"

From The SBN Wiki
Jump to navigation Jump to search
(→‎Aliveness Test: Update for 11.3.1 release)
 
(39 intermediate revisions by the same user not shown)
Line 1: Line 1:
''LDDTool'' is the low-level tool that takes an input XML document and converts it into the various types of structured files used to define, document, and use a local data dictionary in PDS4 labels.
+
''LDDTool'' is the Java-based tool that takes an input XML document (using the ''<Ingest_LDD>'' structure found in the PDS4 master schema) and converts it into the various types of structured files used to define, document, and use a local data dictionary in PDS4 labels. This tool is now being hosted on the NASA GitHub: https://nasa-pds.github.io/pds4-information-model/model-lddtool/index.html.  This site includes older versions as well.
 +
 
 +
{| class="wikitable" style="background-color: yellow"
 +
|
 +
'''''Update in Progress'':''' An update of this page has begun for the latest release of the LDDTool (compatible with the 1.14.0.0 version of the IM).
 +
|}
 +
 
 +
''Previous update for tool version 8.0.0 (Information Model version 1.9.0.0):'' 2018-03-11, A. Raugh
 +
 
  
 
= Introduction =
 
= Introduction =
Line 7: Line 15:
 
== '''''Caveat Usor''''' ==
 
== '''''Caveat Usor''''' ==
  
Be advised: There is a fair amount of hands-on setup work required to get the LDDTool working in your local environment the way you want it to.  And because this is beta-release software, you may well have to repeat this process in new, interesting, and undocumented ways with each new release.  We'll try to keep this page updated to reflect the latest version of the tool.  Feel free to add additional information about LDDTool versions or OS versions not specifically mentioned here.  Contact Anne Raugh at the Small Bodies Node for permission to edit this Wiki if you don't have it already.  Thanks!
+
Be advised: There is a moderate amount of hands-on setup work required to get the LDDTool working in your local environment the way you want it to.  And because this tool is updated for each new model release and to accommodate the sometimes complex needs of the discipline dictionaries still in development, you will likely have to repeat this process with each new release for the foreseeable future.  We'll try to keep this page updated to reflect the latest version of the tool.  Feel free to add additional information about LDDTool versions or OS versions not specifically mentioned here.  Contact Anne Raugh at the Small Bodies Node for permission to edit this Wiki if you don't have it already.  Thanks!
 
 
  
 
== Goal ==
 
== Goal ==
  
Our goal in this set of pages is to start with the LDDTool installation package and end up with the tool installed for general use on the target system.  "General use" in this case means you can invoke the tool any directory where you happen to be working with a command line that looks something like this:
+
Our goal in this set of pages is to start with the LDDTool installation package and end up with the tool installed for general use on the target system.  "General use" in this case means you can invoke the tool in any directory where you happen to be working with a command line that looks something like this:
  
 
<pre>
 
<pre>
 
     % lddtool -lp &lt;input_file&gt;
 
     % lddtool -lp &lt;input_file&gt;
 
</pre>
 
</pre>
 
  
 
== Part List ==
 
== Part List ==
Line 23: Line 29:
 
To run the LDDTool locally, you'll need the following:
 
To run the LDDTool locally, you'll need the following:
  
* '''The LDDTool ZIP package'''.  Because this software is not officially released, you'll need to get this from your PDS contactIf you don't have a PDS contact, try contacting Anne Raugh at the Small Bodies Node.
+
* '''The LDDTool package'''.  The package is available as either a ZIP file or a compressed TAR file from the left margin menu on the  hosting page, https://nasa-pds.github.io/pds4-information-model/model-lddtool/index.htmlEither format will do.
  
* '''Java 1.6 or later'''.  Type "<code>java -version</code>" at your command line to see what version of Java, if any, you have available.  If you don't have Java installed, or want to work with a later version, you'll usually need administrator privileges on your computer to download and install a newer version from the Oracle web site https://java.com/download.  Java 1.7 and later includes a handy feature that will help with configuration later on, so if you're still running a (relatively) ancient version, you now have one more reason to upgrade.
+
* '''Java 1.7 or later'''.  Type "<code>java -version</code>" at your command line to see what version of Java, if any, you have available.  If you don't have Java installed, or want to work with a later version than what is currently installed, you'll usually need administrator privileges on your computer to download and install a newer version from the Oracle web site https://java.com/download (if you want to install it for all users for all purposes, that is).  Java 1.7 and later include a handy feature that will help with configuration later on, so if you're still running a (relatively) ancient version, you now have one more reason to upgrade.
  
* '''A text editor''' that can handle simple text files for batch processing without filling them up with stupid control characters.  On linux-based systems, things like ''<code>vi</code>'', ''<code>pico</code>'', or ''<code>gedit</code>'' will work; from the Windows DOS command line, you can use the ''<code>edit</code>'' command on older systems (pre-Windows7), or ''<code>Notepad</code>'' (which can be invoked from the command line) on newer ones.
+
* '''A text editor''' that can handle simple text files without filling them up with stupid control characters.  On linux-based systems, things like ''<code>vi</code>'', ''<code>pico</code>'', or ''<code>gedit</code>'' will work; from the Windows DOS command line, you can use the ''<code>edit</code>'' command on older systems (pre-Windows7), or ''Notepad'' (which can be invoked from the Windows command line as ''<code>notepad</code>) on newer ones.
 
 
* '''An XML editor''', while optional, will make editing the output schema files easier, and you'll probably want one for creating the input file anyway.  A schema-aware editor like ''Eclipse'' (open source) or ''oXygen'' (commercial) can be very handy for one-off file creation and editing.  For the minor fix-up editing needed in the LDDTool output schemas, though, you can use the same simple text editor you used to edit the batch file or command wrapper.
 
  
 +
* '''An XML editor''', while optional, will make editing the input file much easier.  A schema-aware editor like ''Eclipse'' (open source) or ''oXygen'' (commercial) can be very handy for one-off file creation and editing. The files created by LDDTool should not require further editing, but can be viewed and edited in the same text editor used for the ''lddtool'' batch file/script editing, should the need or desire arise.
  
 
== General Procedure ==
 
== General Procedure ==
Line 38: Line 43:
 
# Unzip the LDDTool package.
 
# Unzip the LDDTool package.
 
# Move the directories you actually need to run the tool to a permanent location.
 
# Move the directories you actually need to run the tool to a permanent location.
# Edit the wrapper script for the local environment.
+
# Edit the batch file or script for the local environment.
# Install the wrapper script.
+
# Install the batch file or script.
 
# Test the installation with the supplied sample files.
 
# Test the installation with the supplied sample files.
 
# Rejoice in the knowledge of a job well done.
 
# Rejoice in the knowledge of a job well done.
Line 47: Line 52:
 
== Unzip the LDDTool Package ==
 
== Unzip the LDDTool Package ==
  
Use any standard ZIP tool (<code>unzip</code> on linux-based systems; the ''Extract All'' option in ''Windows Explorer'') to extract the files from the ZIP package.  You will likely end up with a directory with the name ''<code>LDDTool</code>''.  On a Windows system, by default this directory will be underneath a directory with the same name as the ZIP package, less the ".zip" extension. You can unpack it anywhere - we'll move the stuff we need to a new home once we've picked one out.  If you haven't inspected previous ''LDDTool'' delivery packages, you should probably take a few minutes to familiarize yourself with the contents.
+
Use any standard ZIP tool (<code>unzip</code> on linux-based systems; the ''Extract All'' option in ''Windows Explorer'') to extract the files from the ZIP package.  For the tar file, use the ''z'' option to uncompress while you extract. You should end up with a directory with a name that starts with ''lddtool-'' and ends with the version number of the toolAs of this writing, the latest version of the tool is 11.3.1, so the delivery package unpacks into a directory called ''lddtool-11.3.1''. You can unpack it anywhere - we'll move the stuff we need to a new home once we've picked one out.  If you haven't inspected previous ''LDDTool'' delivery packages, you should probably take a few minutes to familiarize yourself with the contents.
  
Here's what you'll find in the package.
+
Here's what you'll find in the unpacked directory:
  
 
=== Executables ===
 
=== Executables ===
Line 57: Line 62:
 
: You'll actually only need one of the files from this directory, but you'll have to carry the directory along nonetheless.  We'll be modifying one of these scripts to work on your local system, and then installing just that modified file into an ''LDDTool''-specific <code>bin/</code> directory.
 
: You'll actually only need one of the files from this directory, but you'll have to carry the directory along nonetheless.  We'll be modifying one of these scripts to work on your local system, and then installing just that modified file into an ''LDDTool''-specific <code>bin/</code> directory.
 
;Data/
 
;Data/
: At least one of these files is referenced directly by the LDDTool java code, so it needs to be present.
+
: These files are referenced directly by the LDDTool java code, so it needs to be present.
 
;lib/
 
;lib/
 
: This directory contains only the <code>DMDocument.jar</code> jar file, which contains the actual Java code.
 
: This directory contains only the <code>DMDocument.jar</code> jar file, which contains the actual Java code.
Line 63: Line 68:
 
=== Documentation ===
 
=== Documentation ===
  
The <code>doc</code> subdirectory contains documentation files, mainly in the form of PDF files.   
+
The <code>doc</code> subdirectory contains an HTML directory treePoint your browser to the ''index.html'' file to see it in its intended format.  There are brief summary instructions for installation and usage in the nominal case.
* The file with "Installation" in the name provides some general configuration information for LDDTool.
 
* The file with "Operations" in the title provides an overview of how to run the tool with some example command lines.
 
* The file with just "IngestLDD" followed by a string of numbers and underscores that looks like it might be a version number is a rough guide to filling out the Ingest_LDD document that is the input to the LDDTool.  There are some very useful tables in it you might want to keep handy when it comes time to create your input file.
 
  
 
=== Examples ===
 
=== Examples ===
  
This directory contains sets of input/output files produced by ''LDDTool''.  They are undocumented, but at least if you run the usual ''<code>lddtool -lp</code>'' command on the ''IngestLDDTool_*.xml'' you should get output very similar to the file set here.  We'll use one of these for testing the installation. Apart from that, they may be useful examples for how to code some of the more specific, more complex behaviour found in the more intricate discipline dictionaries.
+
This directory seems to contain an old set of files for the Display Dictionary, as it was built for the 1.8.0.0 version of the IM. Not sure why it's here - I think there have been changes in the Ingest_LDD structure that would make this file incompatible with the current version, but feel free to try it out. The result should be similar, but not quite identical.
  
 
=== Peanuts ===
 
=== Peanuts ===
Line 76: Line 78:
 
Like packing peanuts, these files are included in the ZIP but are not, as far as I have found, particularly useful once the package is opened:
 
Like packing peanuts, these files are included in the ZIP but are not, as far as I have found, particularly useful once the package is opened:
  
* ''AAREADME.txt'': This looks like output from a linux <code>man</code> command, and could be a handy one-page reference for command line format and options if typing "<code>lddtool</code>" becomes onerous.
+
* ''LICENSE.txt'': Standard boilerplate license (JPL employees produced this code, and JPL is part of the California Institute of Technology)
* ''runapp.bat'': A Windows-style batch file that demonstrates how to invoke an executable in the bin directory for the sample file ''IngestLDDTool.xml''. It will not run out of the box - you must configure the lddtool.bat file it invokes first, and then make this file executable. It is not generally configurable as is because input and output file names are hard-coded.
+
* ''README.txt'': This file just directs you to the ''doc/index.html'' file.
* ''Schemas/'': This directory contains a couple of superseded PDS4 schema files for the core PDS4 namespace.  I'm not sure why it's here; the tool seems to run fine if this directory is expunged.
+
* ''export/'' : This directory appears to contain dumps of the PDS4 core namespace in various different formats.
 +
* ''Extract/'' : Seems to contain a stub text file for ... something.
 +
* ''Model_DataDictionary/'' : Not as interesting as its title would suggest; this file appears to contain dumps of the PDS4 IM in the format used by the Protege database.
 +
* ''SchemasXML4/'' : This directory contains a copy of the current IM core schema files. Handy if you don't already have a local PDS4 schema tree but want to use a schema-aware editor to create your ''LDDTool'' input file.
  
 
== Install the Executable and Support Directories ==
 
== Install the Executable and Support Directories ==
  
Unless you're seriously hardcore, you will be running LDDTool by invoking a wrapper script (or batch file).  This script sets up some environment variables and then calls Java with the appropriate options and arguments to run on the ''DMDocument.jar'' file with the options and arguments passed to the wrapper script.
+
Unless you're seriously hardcore, you will be running LDDTool by invoking a batch file or script (depending on your OS).  This batch file/script sets up some environment variables and then calls Java with the appropriate Java options and arguments needed to run the ''DMDocument.jar'' file with the options and arguments passed on from the wrapper script. There is a lot of optioning and argumenting involved in that whole process - thus the batch file/script.
  
'''Note:''' The classes in the ''DMDocument.jar'' file read all the environment variables set by the wrapper script/batch file, and also contain hard-coded references to the <code>Data</code> subdirectory in the installation tree.  So wherever you install LDDTool, you're going to need to preserve the delivery tree structure for the <code>bin/</code>, <code>lib/</code>, and <code>Data/</code> subdirectories - and the wrapper script ''must'' be physically located in that <code>bin/</code> directory.
+
'''Note:''' The classes in the ''DMDocument.jar'' file read all the environment variables set by the batch file/script, and also contain hard-coded references to the <code>Data</code> subdirectory in the installation tree.  So wherever you install LDDTool, you're going to need to preserve the delivery tree structure for the <code>bin/</code>, <code>lib/</code>, and <code>Data/</code> subdirectories - and the executable batch file/script ''must'' be physically located in that <code>bin/</code> directory.
  
 
=== Choosing an Installation Location ===
 
=== Choosing an Installation Location ===
 
  
 
On linux-based multi-user systems, you can install LDDTool for general use by all users either by installing into one of the standard locations (<code>/usr/share</code>, for example), or in shared disk space.  If the latter, users wanting to execute LDDTool will likely have to add the appropriate location to their <code>$PATH</code> setting. Alternately, you can install it into your own <code>~/bin/</code> directory for personal use. Note that if you haven't created or used a personal <code>~/bin/</code> directory before, you may have to add it to your <code>$PATH</code> to use it.
 
On linux-based multi-user systems, you can install LDDTool for general use by all users either by installing into one of the standard locations (<code>/usr/share</code>, for example), or in shared disk space.  If the latter, users wanting to execute LDDTool will likely have to add the appropriate location to their <code>$PATH</code> setting. Alternately, you can install it into your own <code>~/bin/</code> directory for personal use. Note that if you haven't created or used a personal <code>~/bin/</code> directory before, you may have to add it to your <code>$PATH</code> to use it.
  
 
In any event, on a linux-based system you will ultimately have to choose one of these options:
 
In any event, on a linux-based system you will ultimately have to choose one of these options:
:# Add the <code>LDDTool/bin</code> directory to your <code>$PATH</code>; which requires editing your shell resource file; or
+
:# Add the <code>lddtool-[version]/bin</code> directory to your <code>$PATH</code>; which requires editing your shell resource file; or
:# Create a link to <code>LDDTool/bin/lddtool</code> in a directory already in your <code>$PATH</code>, which requires an additional edit to the <code>lddtool</code> wrapper script; or
+
:# Create a link to <code>lddtool-[version]/bin/lddtool</code> in a directory already in your <code>$PATH</code>, which requires an additional edit to the <code>lddtool</code> wrapper script; or
 
:# Type the full, absolute path to the <code>lddtool</code> script every time you want to run it.
 
:# Type the full, absolute path to the <code>lddtool</code> script every time you want to run it.
  
Line 101: Line 105:
 
=== What to Copy/Move ===
 
=== What to Copy/Move ===
  
 +
Create a directory in your chosen installation location to hold the LDDTool tree.  You can name this ''lddtool'', or include a version number, or rename it anything convenient.  The name of this directory is not significant to the code.
  
Create a directory in your chosen installation location to hold the LDDTool treeYou can name this ''LDDTool'', or include a version number, or rename it anything convenient. The name of this directory is not significant to the code.
+
Under this directory, copy over the entire contents of the <code>lib/</code> and <code>Data/</code> directories from the installation package.  You will also need to create a <code>bin/</code> directory, into which you should copy either the <code>lddtool</code> linux script or the <code>lddtool.bat</code> Windows batch file, as appropriate for your environment.  For linux users, you will likely also have to make the <code>lddtool</code> script executable.
 +
 
 +
At this point you may also want to copy over the contents of the <code>doc/</code> directory, for easy referenceI also copy the ''README.TXT'' and ''LICENSE.TXT'' files from the root of the install package into this directory, just in case I want to find them again later.
 +
 
 +
== Edit the Wrapper Script/Batch File ==
  
 +
The <code>lddtool</code> script (linux) or <code>lddtool.bat</code> file (Windows) is used to run the tool.  This file will need to be edited to conform to the installation environment.  Any simple text editor can do the job.
  
Under this directory, copy over the entire contents of the <code>lib/</code> and <code>Data/</code> directories from the installation packageYou will also need to create a <code>bin/</code> directory, into which you should copy either the <code>lddtool</code> linux script or the <code>lddtool.bat</code> Windows batch file, as appropriate for your environmentFor linux users, you will likely also have to make the <code>lddtool</code> script executable.  
+
=== Windows Batch File ''lddtool.bat'' ===
 +
 
 +
You'll likely want or need to make a couple changes to this file.  Lines beginning with '::' are comments - feel free to add more.
 +
 
 +
The first executable line in the file is:
 +
:: <code>@echo off</code>
 +
which stops the shell from printing every executable line to your command window when you run the batch file.  Comment this line out if you're trying to trouble-shoot something.
 +
 
 +
Immediately after the "@echo off" line, you should probably add this line:
 +
:: <code>SETLOCAL</code>
 +
This makes sure that any variables that are set by this script do not permanently overwrite any environment variables with the same name that might have already existed for other reasons.
 +
 
 +
Following the next set of comments you'll see the (uncommented) lines that check whether the ''%JAVA_HOME%'' environment variable is already set, and if it isn't, sets itSee the [[Finding and Setting JAVA_HOME]] page on this wiki for detailed steps to check the variable and find the right value to insert here if the variable is not already set.
 +
 
 +
The last executable line in the script before the <code>:END</code> statement looks like this:
 +
:: <code>"%JAVA_HOME%"\bin\java -jar "%DMDOC_JAR%" %*</code>
 +
Remove the quotes from around <code>%JAVA_HOME%</code>.  If they were needed to set the value, then they are already part of the string and the additional quotes will cause a syntax error.
  
 +
Paths with embedded blanks and extra sets of quotes can cause failures, frequently with messages about unexpected information or invalid paths.  If you see that sort of message when you test the batch file, comment out the <code>@echo off</code> line so you can see where the script is failing, and you may have to add or remove quotes on that line (or an earlier line) to adjust for the actual paths in your environment.
  
At this point you may also want to copy over the contents of the <code>doc/</code> directory, for easy reference.  I also copy the ''AAREADME.TXT'' file from the root of the install package into this directory, just in case I want to find it again later.
+
=== Linux ''lddtool'' script ===
  
== Edit the Wrapper Script/Batch File ==
+
If your <code>$JAVA_HOME</code> environment variable is not already set, you will need to edit the <code>export JAVA_HOME</code> line to set it.  See the [[Finding and Setting JAVA_HOME]] page on this wiki for gory details.
  
The <code>lddtool</code> script (linux) or <code>lddtool.bat</code> file (Windows) is used to run the tool.  This file will need to be edited to conform to the installation environment.  Any simple text editor can do the job.
+
'''Note:''' The ''lddtool'' wrapper script is written to be run in the Bourne shell, so use Bourne shell syntax to set <code>JAVA_HOME</code> in the script, regardless of what your login shell is.
  
The first line you'll need to change is the one that contains the definition of the <code>JAVA_HOME</code> environment variableFinding the right value for and then setting this variable can be difficult, especially for folks who are not Java programmers.  Since you'll need to do this a lot for other PDS-provided PDS4 tools, we've put the info into a separate page for consistent referencing:
+
If you're planning to add <code>lddtool</code> to an existing <code>bin/</code> directory (as opposed to adding a new element to your ''$PATH'' to access the tool) you'll need to edit one additional line in the <code>lddtool</code> wrapper - the line beginning <code>export PARENT_DIR</code> (line 34 in the current distribution)Replace the back ticks (`) and everything inside them with the absolute path to the <code>LDDTool</code> installation directory (''without'' ticks or quotes).
  
;;* [[Finding and Setting JAVA_HOME]]
+
Now make the <code>lddtool</code> script file executable, and you're ready to test it.
  
'''Linux users note:''' The ''lddtool'' wrapper script is written to be run in the Bourne shell, so use Bourne shell syntax to set <code>JAVA_HOME</code> in the script, regardless of what your login shell is.
+
=== Testing ===
  
Windows users, you're done at this point.
+
To make sure the batch file or script can properly invoke the tool, you can run it from its bin/ directory home. At the command prompt (Windows or Linux) do:
 +
<pre>
 +
  lddtool -v
 +
</pre>
  
If you are working on a linux-based system and are planning to add the ''LDDTool'' directory to your <code>PATH</code>, make the <code>lddtool</code> file executable and you're done.
+
The response should look something like this:
  
That just leaves linux-based folks who want to add <code>lddtool</code> to an existing <code>bin/</code> directory.  A few more steps are involved, here:
+
<pre>
:# You'll need to edit one additional line in the <code>lddtool</code> wrapper - the line beginning "<code>export SCRIPT_DIR</code>" (line 30 in the current distribution). Replace the back ticks (`) and everything inside them with the absolute path to the ''LDDTool'' <code>bin</code> directory (''without'' ticks or quotes).
+
  LDDTool Version: 11.3.1
:# Make the <code>lddtool</code> wrapper executable.
+
  Built with IM Version: 1.14.0.0
:# Create a link in the existing <code>bin/</code> directory to the <code>lddtool</code> wrapper executable.
+
  Build Date: 2020-05-21 21:02:39
 +
</pre>
  
And now you're done as well.
+
The ''LDDTool Version'' number here should correspond to the version number in the name of the ZIP or tar file you installed from.
  
Once you have had some experience with running LDDTool and tried out some of the other options available (see the "Operations" document supplied in the ZIP package), you may want to further modify the script or batch file to automatically include certain options, provide a standard output file redirect, and otherwise customize tool behaviour.  Knock yourself out.
+
Once you have had some experience with running LDDTool and tried out some of the other options available, you may want to further modify the script or batch file to automatically include certain options, provide a standard output file redirect, and otherwise customize tool behavior.  Knock yourself out.
  
 
== Install the Wrapper Script/Batch File ==
 
== Install the Wrapper Script/Batch File ==
  
The first step for installing the wrapper is a simple matter of copying to the installation <code>bin/</code> directory (if it isn't there already). On linux-based systems, you will also likely have to change the script permissions to make it executable by those allowed to execute it.
+
Assuming, of course, that you don't want to do all your dictionary work in the LDDTool ''bin/'' directory, the last step is making sure you can invoke ''lddtool'' from wherever you will be working. For Windows users this will almost certainly mean adding a new directory to your ''%PATH%'' environment variable.  Linux users have the option of adding a link in a directory already in their command path to the ''lddtool'' script wherever it lives.
  
You can, of course, always execute the script/batch file by using its full, absolute path on the command line.  For ease of use, though, most people prefer to have their executables available in their path. Unfortunately, because of the hard-coded references in the LDDTool code, you can't just copy the script or batch file into a directory already in your path.  But you can add the LDDTool <code>bin/</code> directory to your path, or in linux-based systems you have the option of defining an alias to the physical location in an existing path directory.
+
You can, of course, always execute the script/batch file by using its full, absolute path on the command line.  For ease of use, though, most people prefer to have their executables available in their path.  
  
 
=== Setting Windows %PATH% ===
 
=== Setting Windows %PATH% ===
  
If you only want to add the <code>lddtool.bat</code> location to your path temporarily, say for testing, you can enter something like this at the command prompt:
+
If you only want to add the ''lddtool.bat'' location to your path temporarily, say for testing, you can enter something like this at the command prompt:
  
 
<pre>
 
<pre>
Line 147: Line 178:
 
</pre>
 
</pre>
  
where <code>C:\Users\LDDTool\bin</code> should be replaced with whatever the full path is to your LDDTool <code>bin/</code> directory. This appends the path you provide to the current value of the <code>%PATH%</code> variable.
+
where <code>C:\Users\LDDTool\bin</code> should be replaced with whatever the full path is to your LDDTool ''bin/'' directory. This appends the path you provide to the current value of the ''%PATH%'' variable. You will need to use double quotes around the path you are adding if it contains embedded blanks.
 
 
  
If you'd like to add the LDDTool path to your default <code>PATH</code> once and for all, you can follow the instructions on this page for your particular flavor of Windows:
+
If you'd like to add the LDDTool path to your default ''%PATH%'' once and for all, you can follow the instructions on this page for your particular flavor of Windows:
 
*[http://www.computerhope.com/issues/ch000549.htm How to set the path and environment variables in Windows], by ComputerHope.com.
 
*[http://www.computerhope.com/issues/ch000549.htm How to set the path and environment variables in Windows], by ComputerHope.com.
  
Line 156: Line 186:
 
=== Setting Linux-based $PATH ===
 
=== Setting Linux-based $PATH ===
  
The method used for adding a directory to your current <code>PATH</code> varies based on the shell you use.  The Bourne shell requires an assignment followed by an <code>export</code> command to make the new path visible to programs you run:
+
The method used for adding a directory to your current ''PATH'' varies based on the shell you use.  The Bourne shell requires an assignment followed by an <code>export</code> command to make the new path visible to programs you run:
  
 
<pre>
 
<pre>
Line 169: Line 199:
 
</pre>
 
</pre>
  
where <code>/usr/share/LDDTool/bin</code> is replaced with the full path to the LDDTool installation tree <code>bin/</code> directory.
+
where <code>/usr/share/LDDTool/bin</code> is replaced with the full path to the LDDTool installation tree ''bin/'' directory.
  
 
For C-shell and related shells, use a <code>setenv</code> command:
 
For C-shell and related shells, use a <code>setenv</code> command:
Line 187: Line 217:
 
If you don't know what any of this means, it is time to seek out your friendly, neighborhood Linux programmer and ask, or try Googling "Setting environment variables" for your particular operating system.
 
If you don't know what any of this means, it is time to seek out your friendly, neighborhood Linux programmer and ask, or try Googling "Setting environment variables" for your particular operating system.
  
=== Linux Alternative to Extending $PATH: Aliases ===
+
=== Linux Alternative to Extending $PATH: Links ===
  
So far, at least, as long as the <code>lddtool</code> script is physically located in the LDDTool installation tree as described previously, you can create a link to the script from some more convenient place so that you don't have to modify your <code>$PATH</code> just to run <code>lddtool</code>.  You'll need to have write permission to some directory already in your path.  You can do this in your own <code>~/bin/</code> directory, for example (assuming it's already in your path).
+
So far, at least, as long as the ''lddtool'' script is physically located in the LDDTool installation tree as described previously, you can create a link to the script from some more convenient place so that you don't have to modify your ''$PATH'' just to run ''lddtool''.  You'll need to have write permission to some directory already in your path.  You can do this in your own ''~/bin/'' directory, for example (assuming it's already in your path).
  
To do this, simply create a link to the <code>lddtool</code> script from the directory already in your path.  Say, for example, that the LDDTool tree is in your home directory and is called <code>LDDTool</code>:
+
To do this, simply create a link to the ''lddtool'' script from the directory already in your path.  Say, for example, that the LDDTool tree is in your home directory and is called <code>LDDTool</code>:
 
<pre>
 
<pre>
 
   % ls ~/LDDTool
 
   % ls ~/LDDTool
 
   bin            Data          doc            lib
 
   bin            Data          doc            lib
 
</pre>
 
</pre>
Create a link to the <code>~/LDDTool/bin/lddtool</code> file in the <code>~/bin/</code> directory thus:
+
Create a link to the ''~/LDDTool/bin/lddtool'' file in the ''~/bin/'' directory thus:
 
<pre>
 
<pre>
 
   % cd ~/bin
 
   % cd ~/bin
Line 202: Line 232:
 
</pre>
 
</pre>
  
 +
If you want to start using ''lddtool'' immediately in the same shell window, you will have to <code>source</code> your shell resource file to force it to re-read your path contents.  Apart from that rare circumstance, ''lddtool'' should be in your path every time you start a new shell from now on.
  
If you want to start using <code>lddtool</code> immediately in the same shell window, you will have to source your shell resource file to force it to re-read your path contents.  Apart from that rare circumstance, <code>lddtool</code> should be in your path every time you start a new shell from now on.
+
A similar method can be employed (by users with sufficient privileges) to create a link in an existing system ''bin/'' directory for general use.
 
 
 
 
A similar method can be employed (by users with sufficient privileges) to create a link in an existing system <code>bin/</code> directory for general use.
 
  
  
 
==== Mac Users Note ====
 
==== Mac Users Note ====
  
 +
Mac users should be aware of a minor but possibly annoying detail when defining aliases.  The Mac flavor of Linux, while allowing mixed-case file names, does not consider case significant when comparing file names.  So if, for example, you decided to install LDDTool into ''~/bin/LDDTool'', and then tried to create a link called "lddtool" to ''~/bin/LDDtool/bin/lddtool'' in the same directory, you'd get an error message telling you a file by that name already exists.
  
Mac users should be aware of a minor but possibly annoying detail when defining aliases.  The Mac flavor of Linux, while allowing mixed-case file names, does not consider case significant when comparing file names.  So if, for example, you decided to install LDDTool into <code>~/bin/LDDTool</code>, and then tried to create a link called "lddtool" to <code>~/bin/LDDtool/bin/lddtool</code> in the same directory, you'd get an error message telling you a file by that name already exists.
+
To get around this you can, of course, move the LDDTool tree; or you can give the link a different name using the second argument to the <code>ln</code> command:
 
 
 
 
To get around this you can, of course, move the LDDTool tree; or you can give the link a different name using the second argument to the "<code>ln</code>" command:
 
 
<pre>
 
<pre>
 
   % ln ~/bin/LDDTool/bin/lddtool makeldd
 
   % ln ~/bin/LDDTool/bin/lddtool makeldd
 
</pre>
 
</pre>
Now to invoke the <code>lddtool</code> script, you would use the ''makeldd'' alias, e.g.:
+
Now to invoke the ''lddtool'' script, you would use the ''makeldd'' alias, e.g.:
 
<pre>
 
<pre>
   % makeldd -lp IngestLDDtool.xml
+
   % makeldd -lpM IngestLDDtool.xml
 
</pre>
 
</pre>
  
I haven't actually tested whether or not you can rename the <code>LDDTool</code> directory itself.  If you do try that and have anything to report, let me know...
+
I haven't actually tested whether or not you can rename the ''LDDTool'' directory itself.  If you do try that and have anything to report, let me know...
  
 +
== Test the Installation ==
  
== Test the Installation ==
+
Once you think you've got the LDDTool executables tucked into their homes, you should test the installation and configuration.  You can use the [[File:LDDTool 701 examples.zip|LDDTool Example File Set]] for testing if you don't have an input file of your own yet.
 +
 
 +
=== Aliveness Test ===
 +
 
 +
To test whether you can successfully invoke the executable, try getting the help listing.  This command:
 +
:<code>lddtool -h</code>
 +
Should produce something like this:
 +
<pre>
 +
Usage: lddtool -pl [OPTION]... FILE1 FILE2 ...
 +
Parse a local data dictionary definition file and generate PDS4 data standard files.
  
Once you think you've got the LDDTool executables tucked into their homes, you should test the installation and configuration. Fortunately, the sample files provided in the ZIP package work well for performing a basic aliveness test.
+
Example: lddtool -pl inputFileName
  
=== Running LDDTool on the Sample File ===
+
Process control:
 +
  -p, --PDS4      Set the context to PDS4
 +
  -l, --LDD      Process a local data dictionary input file
 +
  -D, --DataDict  Write the Data Dictionary DocBook file.
 +
  -J, --JSON      Write the master data dictionary to a JSON formatted file.
 +
  -m, --merge    Generate file to merge the local dictionary into the master dictionary
 +
  -M, --Mission  This flag has been deprecated as of PDS4 IM Version 1.14.0.0. See the LDDTool User's Manual for more information on how to provide this information.
 +
  -n, --nuance    Write nuance property maps to LDD schema annotation in JSON
 +
  -N, --Namespace Print the list of configured namespaces to the log
 +
  -1, --IM Spec  Write the Information Model Specification with LDD.
 +
  -v, --version  Returns the LDDTool version number
 +
  -h, --help      Print this message
  
In the top level directory of the package you'll see a file called ''IngestLDDTool.xml'', along with a bunch of other files with similar, but longer, names and a variety of extensions.  These were all output as a result of running <code>lddtool</code> on the ''IngestLDDTool.xml'' file.  You should probably save these files for comparison to the ones you're about to create, especially if you're not familiar with the usual output from <code>lddtool</code>, which is fairly verbose and contains debug information (beta-test code, remember).
+
Input control:
 +
  FILE provides the file name of an input file. The file name extension .xml is assumed.
 +
    If there are more than one file, the first files are considered references
 +
    for the last file. The last file is considered the primary local data dictionary.
  
If you've got the executables, paths, and any aliases properly set up, then you should be able to copy the ''IngestLDDTool.xml'' file anywhere, with any name (that ends in ".xml") and still run this test.
+
Output control:
 +
  FILE is used to provide the file name for the output files. The file name extensions are distinct.
 +
  .xsd -- XML Schema file
 +
  .sch -- schematron file
 +
  .xml -- label file
 +
  .csv -- data dictionary information in csv formatted file.
 +
  .JSON -- dump of model in JSON format.
 +
  .txt -- process report in text format
 +
  .pont -- ontology file for merge
 +
</pre>
  
The basic command line, in either linux or Windows, to reproduce the files included in the ZIP package is this:
+
Alternately, you can view the version number for the executable:
 +
:<code>lddtool -v</code>
 +
which should produce something like this:
 
<pre>
 
<pre>
  lddtool -lp IngestLDDTool.xml > list.txt
+
LDDTool Version: 11.3.1
 +
Built with IM Version: 1.14.0.0
 +
Build Date: 2020-05-21 21:02:39
 
</pre>
 
</pre>
  
(If you've renamed things then of course use the names you've created, not the names above.)  This command redirects the rather long listing produced even by successful runs into the "<code>list.txt</code>" file. You can omit this, if you like, in which case the messages will be dumped to the screen; or you can give a different file name.  The output files will be written into the current working directory, so linux users make sure you have write permission in that directory before running the test.
+
Anything else indicates a configuration error of some sort.  Re-check your paths and script/batch file editing and try again.  If you can't resolve the problem yourself, contact your local PDS consultant for additional assistance. If you haven't got a PDS consultant to call your own yet, you can contact the PDS Engineering node, or Anne Raugh - the author of this wiki - as "raugh" at the Small Bodies Node at the University of Maryland ("astro.umd.edu"). Please provide the failing file(s) and as much detail as possible.
 +
 
 +
=== Running LDDTool on the Example Files ===
 +
 
 +
If you haven't already, download the [[File:LDDTool 1900 examples.zip]] package from this wiki, and unzip it into a working directory.  This package contains some example ''Ingest_LDD'' input files that demonstrate dictionary creation techniques.  The package contains output files from running ''lddtool'' to generate the included schemas - you probably want to stow those somewhere to compare them to what you are about to create.
 +
 
 +
The input dictionary files are called "IngestLDD_Example_Attributes.xml" and "IngestLDD_Example_Classes.xml".  To invoke ''lddtool'' to duplicate the output files included in the package, do:
 +
: <code>lddtool -lpM IngestLDD_Example_Attributes.xml &gt; IngestLDD_Attributes.out</code>
 +
and/or:
 +
: <code>lddtool -lpM IngestLDD_Example_Classes.xml &gt; IngestLDD_Classes.out</code>
 +
The commands above redirect the information that would normally scroll by on your screen to the ''.out'' file so you can examine it at your leisure and compare it to the version provided in the package.  It is also where errors detected by ''lddtool'' are reported.
  
 
=== Expected Results ===
 
=== Expected Results ===
  
The example <code>lddtool</code> command, above, will generate a total of seven output files in addition to the <code>list.txt</code> output.  The file names for six of them are created by taking the input file name and appending namespace and version identifiers found inside the ''IngestLDDTool.xml'' file. The file names will all be the same, but will have different extensions. Here are those extensions, in approximate order of usefulness:
+
The example ''lddtool'' command above will generate a total of five output files in addition to the ''.out'' listing file. The files will all have the same ''lddtool''-generted names but different extensions. Here are those extensions, in approximate order of usefulness:
  
 
* '''.xsd''': This is the XML Schema file that you will reference in your labels when you want to use classes from this dictionary.
 
* '''.xsd''': This is the XML Schema file that you will reference in your labels when you want to use classes from this dictionary.
Line 252: Line 325:
 
* '''.csv''': This is a CSV-formatted summary of the dictionary contents.  You might find this a useful way to review the results if you're averse to reading schema and don't have labels already written to exercise the newly-produced schemas.  You might also find this to be a useful file for passing to reviewers who want to see class and attribute definitions - though maybe with a little editing first.
 
* '''.csv''': This is a CSV-formatted summary of the dictionary contents.  You might find this a useful way to review the results if you're averse to reading schema and don't have labels already written to exercise the newly-produced schemas.  You might also find this to be a useful file for passing to reviewers who want to see class and attribute definitions - though maybe with a little editing first.
 
* '''.xml''': This is a label for the XML Schema and Schematron files; probably only useful as a template.  SBN strongly recommends that rather than creating a label from scratch each time, you modify an existing label at reasonable intervals in order to maintain a <code>&lt;Modification_History&gt;</code> within the label that accurately reflects the development history of the dictionary (as any other product label should for an archival product).  At the very least, the schema label should be modified to identify the unique origin and application of the dictionary files it describes. Trying to get the unmodified label produced here through an SBN review is unlikely to be successful.
 
* '''.xml''': This is a label for the XML Schema and Schematron files; probably only useful as a template.  SBN strongly recommends that rather than creating a label from scratch each time, you modify an existing label at reasonable intervals in order to maintain a <code>&lt;Modification_History&gt;</code> within the label that accurately reflects the development history of the dictionary (as any other product label should for an archival product).  At the very least, the schema label should be modified to identify the unique origin and application of the dictionary files it describes. Trying to get the unmodified label produced here through an SBN review is unlikely to be successful.
* '''.txt''': A detailed listing of program settings, messages, and dictionary content.  Might be useful for debugging subtle definition issues.  In general, SBN recommends you avoid subtlety in dictionary definitions.
 
* '''.pont''': This file is formatted specifically for input to the PDS ontology database at JPL that holds dictionary information.  Outside of that context, it is likely not useful.
 
* '''PDS4SchematronRules.pins''': Some sort of encoding of PDS4 Schematron rules into a structured text file.  No idea why this is generated or what it might be used for.
 
  
 
=== Checking for Success ===
 
=== Checking for Success ===
  
Unfortunately, there is no big, friendly, "You Succeeded!" (or "Thanks for trying!") message when the program completes.
+
If you compare the output files from the above commands to what came in the example zip file, you should see differences in date stamps and local paths, but otherwise nothing else.  Mage sure your option string is '''-lpM''' (lowercase letter "ell", uppercase letter "em"), or you will get a slightly different set of output files or a different namespace definition in the output schemas.
  
The first thing to check is the program output listing. The last line of the "<code>list.txt</code>" file, or the last line on your screen if you didn't redirect output, should be this:
+
If you run the tool on your own input file, the first thing to check is the program output listing, which will scroll past on your screen if you don't redirect it to a file. The last line of that listing should look like this:
 
<pre>
 
<pre>
 
   >>info    - LDDTOOL Exit
 
   >>info    - LDDTOOL Exit
 
</pre>
 
</pre>
  
This indicates at least some measure of success.  The file will contain a ''LOT'' (several hundred lines) of warning messages about various override conditions.  This is normal.  It should ''not'' contain any error messages (lines beginning with "<code>&gt;&gt;error</code>").  These indicate some sort of failure.
+
This indicates at least some measure of success.  Depending on how complex your input file is, there will be a few dozen to a few hundred "INFO" lines containing messages about various override conditions.  This is normal.  It should ''not'' contain any "ERROR" lines or lines beginning with "<code>&gt;&gt;error</code>".  These indicate some sort of failure.  There will likely also be two "WARNING" lines that look like this:
 +
<pre>
 +
  WARNING  Header:  - New steward has been specified:sbn
 +
  WARNING  Header:  - New namespace id has been specified:ex
 +
</pre>
 +
unless you are updating a dictionary that is already known to LDDTool.  Other "WARNING" statements, however, are problematic and should be investigated.
 +
 
 +
Once you've verified expected output, you should be good to go.
 +
 
 +
=== Common Failures ===
 +
 
 +
The common failures encountered at this point come from system references not resolving.  Here are the most likely suspects:
 +
 
 +
'''''Cannot find DMDocument jar file in ['''some directory''']'''''
 +
 
 +
:This error is reported back to the command line by the <code>lddtool</code> wrapper, which checks for the existence of the ''DMDocument.jar'' file before invoking java on it.  If you haven't previously set <code>PARENT_DIR</code> in the wrapper to point to the <code>LDDTool</code> installation directory, do so (in some environments this may be required even if you're using the default configuration).  If you have already modified <code>PARENT_DIR</code>, search it for typos.  The <code>PARENT_DIR</code> directory ''must'' contain a <code>lib/</code> subdirectory, which in turn must contain the <code>DMDocument.jar</code> file.
 +
 
 +
:You'll also get this message if there is a typo in the <code>lib/</code> subdirectory name or <code>DMDocument.jar</code> name (case counts).
 +
 
 +
'''''>>error  - Required data file was not found: ['''some path to an XML file''']'''''
 +
 
 +
:This will show up as the last line in your listing if you forgot to include the <code>Data/</code> subdirectory of the ''LDDTool'' distribution in your installation <code>LDDTool</code> directory tree.  Spelling and case count, so if you're on a linux-based system and accidentally changed "<code>Data</code>" to "<code>data</code>", for example, you'll get this message.
 +
 
 +
'''''/bin/java: No such file or directory''''' (This is the linux version of this error)
 +
 
 +
:Messages like this are reported to the command line and indicate that the <code>JAVA_HOME</code> setting either failed or points to the wrong place.  If there's a typo in the <code>JAVA_HOME</code> setting, you might also see characters before "/bin/java" indicating what the script thinks <code>JAVA_HOME</code> was set to.  The <code>JAVA_HOME</code> directory ''must'' contain a <code>bin/</code> subdirectory, which in turn must contain the ''java'' executable.
  
At this point linux-based users can make good use of <code>diff</code> to check for any substantive variation between the files just produced and those that came in the ZIP package.  If they are identical +/- date- and system-specific notations, then you can consider the test a success. Windows users without software with a "diff" feature can check file sizes and eyeball the contents in any plain text editor (NotePad or WordPad will work in a pinch).
+
'''''['''usage info dump''']'''''
  
Once you've verified expected output, you should be ready to run <code>lddtool</code> on you own input files.
+
:If you get a dump of ''lddtool'' usage information when you run the test command on the example file, look at the very top - there's probably an error waiting for you.  Make sure you spelled the input file name correctly and included the required "<code>-lp</code>" option set.
  
 
== Rejoice ==
 
== Rejoice ==

Latest revision as of 15:18, 20 July 2020

LDDTool is the Java-based tool that takes an input XML document (using the <Ingest_LDD> structure found in the PDS4 master schema) and converts it into the various types of structured files used to define, document, and use a local data dictionary in PDS4 labels. This tool is now being hosted on the NASA GitHub: https://nasa-pds.github.io/pds4-information-model/model-lddtool/index.html. This site includes older versions as well.

Update in Progress: An update of this page has begun for the latest release of the LDDTool (compatible with the 1.14.0.0 version of the IM).

Previous update for tool version 8.0.0 (Information Model version 1.9.0.0): 2018-03-11, A. Raugh


Introduction

A local data dictionary is a set of schema files that define a namespace that is under the control of someone other than the PDS4 managers. It includes the PDS discipline dictionaries for things like display orientation and geometry, as well as node- and mission-specific dictionaries. There are web-based and GUI-based tools in development at various places to help users who prefer to do dictionary development in an web/GUI environment - ask your friendly, neighborhood PDS node consultant what's currently available if that's what you're looking for. These pages are for the roll-your-own crowd that either prefers or has no choice but to work at the command line and see how the sausage is made.

Caveat Usor

Be advised: There is a moderate amount of hands-on setup work required to get the LDDTool working in your local environment the way you want it to. And because this tool is updated for each new model release and to accommodate the sometimes complex needs of the discipline dictionaries still in development, you will likely have to repeat this process with each new release for the foreseeable future. We'll try to keep this page updated to reflect the latest version of the tool. Feel free to add additional information about LDDTool versions or OS versions not specifically mentioned here. Contact Anne Raugh at the Small Bodies Node for permission to edit this Wiki if you don't have it already. Thanks!

Goal

Our goal in this set of pages is to start with the LDDTool installation package and end up with the tool installed for general use on the target system. "General use" in this case means you can invoke the tool in any directory where you happen to be working with a command line that looks something like this:

    % lddtool -lp <input_file>

Part List

To run the LDDTool locally, you'll need the following:

  • Java 1.7 or later. Type "java -version" at your command line to see what version of Java, if any, you have available. If you don't have Java installed, or want to work with a later version than what is currently installed, you'll usually need administrator privileges on your computer to download and install a newer version from the Oracle web site https://java.com/download (if you want to install it for all users for all purposes, that is). Java 1.7 and later include a handy feature that will help with configuration later on, so if you're still running a (relatively) ancient version, you now have one more reason to upgrade.
  • A text editor that can handle simple text files without filling them up with stupid control characters. On linux-based systems, things like vi, pico, or gedit will work; from the Windows DOS command line, you can use the edit command on older systems (pre-Windows7), or Notepad (which can be invoked from the Windows command line as notepad) on newer ones.
  • An XML editor, while optional, will make editing the input file much easier. A schema-aware editor like Eclipse (open source) or oXygen (commercial) can be very handy for one-off file creation and editing. The files created by LDDTool should not require further editing, but can be viewed and edited in the same text editor used for the lddtool batch file/script editing, should the need or desire arise.

General Procedure

Here's the general procedure for setting up the tool:

  1. Unzip the LDDTool package.
  2. Move the directories you actually need to run the tool to a permanent location.
  3. Edit the batch file or script for the local environment.
  4. Install the batch file or script.
  5. Test the installation with the supplied sample files.
  6. Rejoice in the knowledge of a job well done.

Procedure

Unzip the LDDTool Package

Use any standard ZIP tool (unzip on linux-based systems; the Extract All option in Windows Explorer) to extract the files from the ZIP package. For the tar file, use the z option to uncompress while you extract. You should end up with a directory with a name that starts with lddtool- and ends with the version number of the tool. As of this writing, the latest version of the tool is 11.3.1, so the delivery package unpacks into a directory called lddtool-11.3.1. You can unpack it anywhere - we'll move the stuff we need to a new home once we've picked one out. If you haven't inspected previous LDDTool delivery packages, you should probably take a few minutes to familiarize yourself with the contents.

Here's what you'll find in the unpacked directory:

Executables

The executable elements of the package include:

bin/
You'll actually only need one of the files from this directory, but you'll have to carry the directory along nonetheless. We'll be modifying one of these scripts to work on your local system, and then installing just that modified file into an LDDTool-specific bin/ directory.
Data/
These files are referenced directly by the LDDTool java code, so it needs to be present.
lib/
This directory contains only the DMDocument.jar jar file, which contains the actual Java code.

Documentation

The doc subdirectory contains an HTML directory tree. Point your browser to the index.html file to see it in its intended format. There are brief summary instructions for installation and usage in the nominal case.

Examples

This directory seems to contain an old set of files for the Display Dictionary, as it was built for the 1.8.0.0 version of the IM. Not sure why it's here - I think there have been changes in the Ingest_LDD structure that would make this file incompatible with the current version, but feel free to try it out. The result should be similar, but not quite identical.

Peanuts

Like packing peanuts, these files are included in the ZIP but are not, as far as I have found, particularly useful once the package is opened:

  • LICENSE.txt: Standard boilerplate license (JPL employees produced this code, and JPL is part of the California Institute of Technology)
  • README.txt: This file just directs you to the doc/index.html file.
  • export/ : This directory appears to contain dumps of the PDS4 core namespace in various different formats.
  • Extract/ : Seems to contain a stub text file for ... something.
  • Model_DataDictionary/ : Not as interesting as its title would suggest; this file appears to contain dumps of the PDS4 IM in the format used by the Protege database.
  • SchemasXML4/ : This directory contains a copy of the current IM core schema files. Handy if you don't already have a local PDS4 schema tree but want to use a schema-aware editor to create your LDDTool input file.

Install the Executable and Support Directories

Unless you're seriously hardcore, you will be running LDDTool by invoking a batch file or script (depending on your OS). This batch file/script sets up some environment variables and then calls Java with the appropriate Java options and arguments needed to run the DMDocument.jar file with the options and arguments passed on from the wrapper script. There is a lot of optioning and argumenting involved in that whole process - thus the batch file/script.

Note: The classes in the DMDocument.jar file read all the environment variables set by the batch file/script, and also contain hard-coded references to the Data subdirectory in the installation tree. So wherever you install LDDTool, you're going to need to preserve the delivery tree structure for the bin/, lib/, and Data/ subdirectories - and the executable batch file/script must be physically located in that bin/ directory.

Choosing an Installation Location

On linux-based multi-user systems, you can install LDDTool for general use by all users either by installing into one of the standard locations (/usr/share, for example), or in shared disk space. If the latter, users wanting to execute LDDTool will likely have to add the appropriate location to their $PATH setting. Alternately, you can install it into your own ~/bin/ directory for personal use. Note that if you haven't created or used a personal ~/bin/ directory before, you may have to add it to your $PATH to use it.

In any event, on a linux-based system you will ultimately have to choose one of these options:

  1. Add the lddtool-[version]/bin directory to your $PATH; which requires editing your shell resource file; or
  2. Create a link to lddtool-[version]/bin/lddtool in a directory already in your $PATH, which requires an additional edit to the lddtool wrapper script; or
  3. Type the full, absolute path to the lddtool script every time you want to run it.


On Windows systems, you can install LDDTool into the "Program Files\" directory for general use (this may require admin privileges), or in your own directory space for personal use. You will likely have to modify %PATH% setting information to make the lddtool.bat executable visible to users without requiring a complete path specification to run the batch file. More on that later.

What to Copy/Move

Create a directory in your chosen installation location to hold the LDDTool tree. You can name this lddtool, or include a version number, or rename it anything convenient. The name of this directory is not significant to the code.

Under this directory, copy over the entire contents of the lib/ and Data/ directories from the installation package. You will also need to create a bin/ directory, into which you should copy either the lddtool linux script or the lddtool.bat Windows batch file, as appropriate for your environment. For linux users, you will likely also have to make the lddtool script executable.

At this point you may also want to copy over the contents of the doc/ directory, for easy reference. I also copy the README.TXT and LICENSE.TXT files from the root of the install package into this directory, just in case I want to find them again later.

Edit the Wrapper Script/Batch File

The lddtool script (linux) or lddtool.bat file (Windows) is used to run the tool. This file will need to be edited to conform to the installation environment. Any simple text editor can do the job.

Windows Batch File lddtool.bat

You'll likely want or need to make a couple changes to this file. Lines beginning with '::' are comments - feel free to add more.

The first executable line in the file is:

@echo off

which stops the shell from printing every executable line to your command window when you run the batch file. Comment this line out if you're trying to trouble-shoot something.

Immediately after the "@echo off" line, you should probably add this line:

SETLOCAL

This makes sure that any variables that are set by this script do not permanently overwrite any environment variables with the same name that might have already existed for other reasons.

Following the next set of comments you'll see the (uncommented) lines that check whether the %JAVA_HOME% environment variable is already set, and if it isn't, sets it. See the Finding and Setting JAVA_HOME page on this wiki for detailed steps to check the variable and find the right value to insert here if the variable is not already set.

The last executable line in the script before the :END statement looks like this:

"%JAVA_HOME%"\bin\java -jar "%DMDOC_JAR%" %*

Remove the quotes from around %JAVA_HOME%. If they were needed to set the value, then they are already part of the string and the additional quotes will cause a syntax error.

Paths with embedded blanks and extra sets of quotes can cause failures, frequently with messages about unexpected information or invalid paths. If you see that sort of message when you test the batch file, comment out the @echo off line so you can see where the script is failing, and you may have to add or remove quotes on that line (or an earlier line) to adjust for the actual paths in your environment.

Linux lddtool script

If your $JAVA_HOME environment variable is not already set, you will need to edit the export JAVA_HOME line to set it. See the Finding and Setting JAVA_HOME page on this wiki for gory details.

Note: The lddtool wrapper script is written to be run in the Bourne shell, so use Bourne shell syntax to set JAVA_HOME in the script, regardless of what your login shell is.

If you're planning to add lddtool to an existing bin/ directory (as opposed to adding a new element to your $PATH to access the tool) you'll need to edit one additional line in the lddtool wrapper - the line beginning export PARENT_DIR (line 34 in the current distribution). Replace the back ticks (`) and everything inside them with the absolute path to the LDDTool installation directory (without ticks or quotes).

Now make the lddtool script file executable, and you're ready to test it.

Testing

To make sure the batch file or script can properly invoke the tool, you can run it from its bin/ directory home. At the command prompt (Windows or Linux) do:

  lddtool -v

The response should look something like this:

  LDDTool Version: 11.3.1
  Built with IM Version: 1.14.0.0
  Build Date: 2020-05-21 21:02:39

The LDDTool Version number here should correspond to the version number in the name of the ZIP or tar file you installed from.

Once you have had some experience with running LDDTool and tried out some of the other options available, you may want to further modify the script or batch file to automatically include certain options, provide a standard output file redirect, and otherwise customize tool behavior. Knock yourself out.

Install the Wrapper Script/Batch File

Assuming, of course, that you don't want to do all your dictionary work in the LDDTool bin/ directory, the last step is making sure you can invoke lddtool from wherever you will be working. For Windows users this will almost certainly mean adding a new directory to your %PATH% environment variable. Linux users have the option of adding a link in a directory already in their command path to the lddtool script wherever it lives.

You can, of course, always execute the script/batch file by using its full, absolute path on the command line. For ease of use, though, most people prefer to have their executables available in their path.

Setting Windows %PATH%

If you only want to add the lddtool.bat location to your path temporarily, say for testing, you can enter something like this at the command prompt:

    C:>set PATH=%PATH%;C:\Users\LDDTool\bin

where C:\Users\LDDTool\bin should be replaced with whatever the full path is to your LDDTool bin/ directory. This appends the path you provide to the current value of the %PATH% variable. You will need to use double quotes around the path you are adding if it contains embedded blanks.

If you'd like to add the LDDTool path to your default %PATH% once and for all, you can follow the instructions on this page for your particular flavor of Windows:


Setting Linux-based $PATH

The method used for adding a directory to your current PATH varies based on the shell you use. The Bourne shell requires an assignment followed by an export command to make the new path visible to programs you run:

   % PATH=$PATH:/usr/share/LDDTool/bin
   % export PATH

or this shortcut should also work:

   % export PATH=$PATH:/usr/share/LDDTool/bin

where /usr/share/LDDTool/bin is replaced with the full path to the LDDTool installation tree bin/ directory.

For C-shell and related shells, use a setenv command:

   % setenv PATH $PATH":/usr/share/LDDTool/bin"

or the set command:

  % set PATH=($PATH /usr/share/LDDTool/bin)

For either type of shell, you can do this at the command line before beginning your work with LDDTool, or you can add the lines to your shell resource file so it's already there every time you log on.

If you don't know what any of this means, it is time to seek out your friendly, neighborhood Linux programmer and ask, or try Googling "Setting environment variables" for your particular operating system.

Linux Alternative to Extending $PATH: Links

So far, at least, as long as the lddtool script is physically located in the LDDTool installation tree as described previously, you can create a link to the script from some more convenient place so that you don't have to modify your $PATH just to run lddtool. You'll need to have write permission to some directory already in your path. You can do this in your own ~/bin/ directory, for example (assuming it's already in your path).

To do this, simply create a link to the lddtool script from the directory already in your path. Say, for example, that the LDDTool tree is in your home directory and is called LDDTool:

   % ls ~/LDDTool
   bin            Data           doc            lib

Create a link to the ~/LDDTool/bin/lddtool file in the ~/bin/ directory thus:

   % cd ~/bin
   % ln ~/LDDtool/bin/lddtool

If you want to start using lddtool immediately in the same shell window, you will have to source your shell resource file to force it to re-read your path contents. Apart from that rare circumstance, lddtool should be in your path every time you start a new shell from now on.

A similar method can be employed (by users with sufficient privileges) to create a link in an existing system bin/ directory for general use.


Mac Users Note

Mac users should be aware of a minor but possibly annoying detail when defining aliases. The Mac flavor of Linux, while allowing mixed-case file names, does not consider case significant when comparing file names. So if, for example, you decided to install LDDTool into ~/bin/LDDTool, and then tried to create a link called "lddtool" to ~/bin/LDDtool/bin/lddtool in the same directory, you'd get an error message telling you a file by that name already exists.

To get around this you can, of course, move the LDDTool tree; or you can give the link a different name using the second argument to the ln command:

   % ln ~/bin/LDDTool/bin/lddtool makeldd

Now to invoke the lddtool script, you would use the makeldd alias, e.g.:

   % makeldd -lpM IngestLDDtool.xml

I haven't actually tested whether or not you can rename the LDDTool directory itself. If you do try that and have anything to report, let me know...

Test the Installation

Once you think you've got the LDDTool executables tucked into their homes, you should test the installation and configuration. You can use the File:LDDTool 701 examples.zip for testing if you don't have an input file of your own yet.

Aliveness Test

To test whether you can successfully invoke the executable, try getting the help listing. This command:

lddtool -h

Should produce something like this:

Usage: lddtool -pl [OPTION]... FILE1 FILE2 ...
Parse a local data dictionary definition file and generate PDS4 data standard files.

Example: lddtool -pl  inputFileName

Process control:
  -p, --PDS4      Set the context to PDS4
  -l, --LDD       Process a local data dictionary input file
  -D, --DataDict  Write the Data Dictionary DocBook file.
  -J, --JSON      Write the master data dictionary to a JSON formatted file.
  -m, --merge     Generate file to merge the local dictionary into the master dictionary
  -M, --Mission   This flag has been deprecated as of PDS4 IM Version 1.14.0.0. See the LDDTool User's Manual for more information on how to provide this information.
  -n, --nuance    Write nuance property maps to LDD schema annotation in JSON
  -N, --Namespace Print the list of configured namespaces to the log
  -1, --IM Spec   Write the Information Model Specification with LDD.
  -v, --version   Returns the LDDTool version number
  -h, --help      Print this message

Input control:
  FILE provides the file name of an input file. The file name extension .xml is assumed.
    If there are more than one file, the first files are considered references
    for the last file. The last file is considered the primary local data dictionary.

Output control:
  FILE is used to provide the file name for the output files. The file name extensions are distinct.
  .xsd -- XML Schema file
  .sch -- schematron file
  .xml -- label file
  .csv -- data dictionary information in csv formatted file.
  .JSON -- dump of model in JSON format.
  .txt -- process report in text format
  .pont -- ontology file for merge

Alternately, you can view the version number for the executable:

lddtool -v

which should produce something like this:

LDDTool Version: 11.3.1
Built with IM Version: 1.14.0.0
Build Date: 2020-05-21 21:02:39

Anything else indicates a configuration error of some sort. Re-check your paths and script/batch file editing and try again. If you can't resolve the problem yourself, contact your local PDS consultant for additional assistance. If you haven't got a PDS consultant to call your own yet, you can contact the PDS Engineering node, or Anne Raugh - the author of this wiki - as "raugh" at the Small Bodies Node at the University of Maryland ("astro.umd.edu"). Please provide the failing file(s) and as much detail as possible.

Running LDDTool on the Example Files

If you haven't already, download the File:LDDTool 1900 examples.zip package from this wiki, and unzip it into a working directory. This package contains some example Ingest_LDD input files that demonstrate dictionary creation techniques. The package contains output files from running lddtool to generate the included schemas - you probably want to stow those somewhere to compare them to what you are about to create.

The input dictionary files are called "IngestLDD_Example_Attributes.xml" and "IngestLDD_Example_Classes.xml". To invoke lddtool to duplicate the output files included in the package, do:

lddtool -lpM IngestLDD_Example_Attributes.xml > IngestLDD_Attributes.out

and/or:

lddtool -lpM IngestLDD_Example_Classes.xml > IngestLDD_Classes.out

The commands above redirect the information that would normally scroll by on your screen to the .out file so you can examine it at your leisure and compare it to the version provided in the package. It is also where errors detected by lddtool are reported.

Expected Results

The example lddtool command above will generate a total of five output files in addition to the .out listing file. The files will all have the same lddtool-generted names but different extensions. Here are those extensions, in approximate order of usefulness:

  • .xsd: This is the XML Schema file that you will reference in your labels when you want to use classes from this dictionary.
  • .sch: This is the Schematron file that you will also reference in your labels when you want to use classes from this dictionary.
  • .csv: This is a CSV-formatted summary of the dictionary contents. You might find this a useful way to review the results if you're averse to reading schema and don't have labels already written to exercise the newly-produced schemas. You might also find this to be a useful file for passing to reviewers who want to see class and attribute definitions - though maybe with a little editing first.
  • .xml: This is a label for the XML Schema and Schematron files; probably only useful as a template. SBN strongly recommends that rather than creating a label from scratch each time, you modify an existing label at reasonable intervals in order to maintain a <Modification_History> within the label that accurately reflects the development history of the dictionary (as any other product label should for an archival product). At the very least, the schema label should be modified to identify the unique origin and application of the dictionary files it describes. Trying to get the unmodified label produced here through an SBN review is unlikely to be successful.

Checking for Success

If you compare the output files from the above commands to what came in the example zip file, you should see differences in date stamps and local paths, but otherwise nothing else. Mage sure your option string is -lpM (lowercase letter "ell", uppercase letter "em"), or you will get a slightly different set of output files or a different namespace definition in the output schemas.

If you run the tool on your own input file, the first thing to check is the program output listing, which will scroll past on your screen if you don't redirect it to a file. The last line of that listing should look like this:

   >>info    - LDDTOOL Exit

This indicates at least some measure of success. Depending on how complex your input file is, there will be a few dozen to a few hundred "INFO" lines containing messages about various override conditions. This is normal. It should not contain any "ERROR" lines or lines beginning with ">>error". These indicate some sort of failure. There will likely also be two "WARNING" lines that look like this:

   WARNING  Header:  - New steward has been specified:sbn
   WARNING  Header:  - New namespace id has been specified:ex

unless you are updating a dictionary that is already known to LDDTool. Other "WARNING" statements, however, are problematic and should be investigated.

Once you've verified expected output, you should be good to go.

Common Failures

The common failures encountered at this point come from system references not resolving. Here are the most likely suspects:

Cannot find DMDocument jar file in [some directory]

This error is reported back to the command line by the lddtool wrapper, which checks for the existence of the DMDocument.jar file before invoking java on it. If you haven't previously set PARENT_DIR in the wrapper to point to the LDDTool installation directory, do so (in some environments this may be required even if you're using the default configuration). If you have already modified PARENT_DIR, search it for typos. The PARENT_DIR directory must contain a lib/ subdirectory, which in turn must contain the DMDocument.jar file.
You'll also get this message if there is a typo in the lib/ subdirectory name or DMDocument.jar name (case counts).

>>error - Required data file was not found: [some path to an XML file]

This will show up as the last line in your listing if you forgot to include the Data/ subdirectory of the LDDTool distribution in your installation LDDTool directory tree. Spelling and case count, so if you're on a linux-based system and accidentally changed "Data" to "data", for example, you'll get this message.

/bin/java: No such file or directory (This is the linux version of this error)

Messages like this are reported to the command line and indicate that the JAVA_HOME setting either failed or points to the wrong place. If there's a typo in the JAVA_HOME setting, you might also see characters before "/bin/java" indicating what the script thinks JAVA_HOME was set to. The JAVA_HOME directory must contain a bin/ subdirectory, which in turn must contain the java executable.

[usage info dump]

If you get a dump of lddtool usage information when you run the test command on the example file, look at the very top - there's probably an error waiting for you. Make sure you spelled the input file name correctly and included the required "-lp" option set.

Rejoice