Difference between revisions of "Python PDS4 Tools"

From The SBN Wiki
Jump to navigation Jump to search
(Fixed leftover 'data object' to 'data structure' terminology)
m (Update user manual URL)
 
(33 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
<div style="clear: {{{clear|right}}}; margin-bottom: .5em; float: right; padding: .5em 0 .8em 1.4em; background: none; width: {{{width|{{{1|auto}}}}}};" {{#if:{{{limit|}}}|class="toclimit-{{{limit}}}"}}>__TOC__</div><noinclude></noinclude>
  
== Introduction ==
+
'''Python and PDS4'''
  
 
This document describes the current status and usage of Python tools developed at PDS-SBN to read and visualize PDS4 data in Python. Please note that a [http://pds-smallbodies.astro.umd.edu/tools/tools_readPDS.shtml PDS4 reader and visualizer for IDL] is also available.
 
This document describes the current status and usage of Python tools developed at PDS-SBN to read and visualize PDS4 data in Python. Please note that a [http://pds-smallbodies.astro.umd.edu/tools/tools_readPDS.shtml PDS4 reader and visualizer for IDL] is also available.
Line 8: Line 9:
 
=== Introduction ===  
 
=== Introduction ===  
  
This section describes a Python package that can read and display PDS4 data and meta data. In the future this tool is expected to support all PDS4 data structures, currently support is limited to structures given in the [[#Supported Data Structures|Supported Data Structures]] section. The package expects labels that pass PDS4 Schema and Schematron validation.
+
This section describes a Python package that can read and display PDS4 data and meta data. In the future this tool is expected to support all PDS4 data structures, currently support is limited to structures given in the [[#Supported Data Structures|Supported Data Structures]] section. The package expects valid PDS4 labels formatted according to the PDS4 Standard.
  
 
Contact [[User:lnagdi1|Lev Nagdimunov]] with questions or comments regarding this code or its description.
 
Contact [[User:lnagdi1|Lev Nagdimunov]] with questions or comments regarding this code or its description.
Line 16: Line 17:
 
Python 2.6+ or 3.3+
 
Python 2.6+ or 3.3+
  
pds4_read: None <br>
+
pds4_read: [http://www.numpy.org/ NumPy] <br>
 
pds4_viewer: [http://www.numpy.org/ NumPy], [http://www.matplotlib.org/ matplotlib]
 
pds4_viewer: [http://www.numpy.org/ NumPy], [http://www.matplotlib.org/ matplotlib]
 
You may use <tt>pds4_read</tt> to read-in data without any extra packages; <tt>pds4_viewer</tt> requires recent versions of the additional packages.
 
 
==== Optional Features ====
 
 
pds4_read: [http://www.numpy.org/ NumPy]<br>
 
Recommended for Arrays and Tables containing GROUP fields to allow for multi-dimensional indexing. Can result in significant improvements in memory usage and read-in speed for some data structures.<br>
 
 
pds4_viewer: None
 
  
 
=== Supported Data Structures ===
 
=== Supported Data Structures ===
  
PDS4 Data Standards < v1.3 are not officially supported but may work.<br>
+
PDS4 Data Standards >= v1.0 are supported.<br>
PDS4 Data Standards >= v1.3 are supported.<br>
 
 
PDS3 Data Standards are not supported.
 
PDS3 Data Standards are not supported.
  
 
The table below lists the main [[Filling_Out_the_File_Area_Observational_Classes#Data_Structures|PDS4 data structures]] and the current status. <br><br>
 
The table below lists the main [[Filling_Out_the_File_Area_Observational_Classes#Data_Structures|PDS4 data structures]] and the current status. <br><br>
Read-in column indicates support by <tt>pds4_read()</tt> <br>  
+
Read-in column indicates support by <tt>pds4_tools.read()</tt> <br>  
Display columns indicate support by <tt>pds4_viewer()</tt>.  
+
Display columns indicate support by <tt>pds4_tools.view()</tt>.  
  
 
{| class="wikitable" style="text-align: center; width: 800px; "  
 
{| class="wikitable" style="text-align: center; width: 800px; "  
Line 45: Line 36:
 
! Display as Image
 
! Display as Image
 
! Display Columns as Plot
 
! Display Columns as Plot
 +
|-
 +
| style="text-align: left;" | Header
 +
| Yes
 +
| No
 +
| No
 +
| No
 
|-
 
|-
 
| style="text-align: left;" | Array  
 
| style="text-align: left;" | Array  
Line 50: Line 47:
 
| Yes
 
| Yes
 
| Yes, N-dims
 
| Yes, N-dims
| Under Development
+
| Yes, 1-D only
 
|-
 
|-
 
| style="text-align: left;" | Array_2D  
 
| style="text-align: left;" | Array_2D  
Line 56: Line 53:
 
| Yes
 
| Yes
 
| Yes
 
| Yes
| Under Development
+
| No
 
|-
 
|-
 
| style="text-align: left;" | Array_2D_*  
 
| style="text-align: left;" | Array_2D_*  
Line 62: Line 59:
 
| Yes
 
| Yes
 
| Yes
 
| Yes
| Under Development
+
| No
 
|-
 
|-
 
| style="text-align: left;" | Array_3D
 
| style="text-align: left;" | Array_3D
Line 68: Line 65:
 
| Yes
 
| Yes
 
| Yes
 
| Yes
| Under Development
+
| No
 
|-
 
|-
 
| style="text-align: left;" | Array_3D_*
 
| style="text-align: left;" | Array_3D_*
Line 74: Line 71:
 
| Yes
 
| Yes
 
| Yes
 
| Yes
| Under Development
+
| No
 
|-
 
|-
 
| style="text-align: left;" | Table_Character
 
| style="text-align: left;" | Table_Character
Line 80: Line 77:
 
| Yes
 
| Yes
 
| No
 
| No
| Under Development
+
| Yes
 
|-
 
|-
 
| style="text-align: left;" | Table_Binary
 
| style="text-align: left;" | Table_Binary
Line 86: Line 83:
 
| Yes
 
| Yes
 
| No  
 
| No  
| Under Development
+
| Yes
 
|-
 
|-
 
| style="text-align: left;" | Table_Delimited
 
| style="text-align: left;" | Table_Delimited
| Future development
+
| Yes
| Future development
+
| Yes
| Future Development
+
| No
| Future Development
+
| Yes
 +
|-
 +
| style="text-align: left;" | Composite_Structure
 +
| No
 +
| No
 +
| No
 +
| No
 
|}
 
|}
 +
 +
=== User Manual ===
 +
 +
Online [https://pdssbn.astro.umd.edu/tools/pds4_tools_docs/current/ usage documentation], both for scientists and for developers, is available.
  
 
=== Download ===
 
=== Download ===
  
Download the ZIP file [[File:PDS4_tools-0.4.zip]]. Released on December 3, 2015.
+
Download the ZIP file <span class="plainlinks">[https://pdssbn.astro.umd.edu/toolsrc/readpds_python/1.3/PDS4_tools-1.3.zip File:PDS4 tools-1.3.zip]</span>. Released on October 10, 2021.
 
 
Note: This is software that is still actively being developed.  
 
  
 
Note: A distributable version of the viewer only, which does not require Python, is [[PDS4 Viewer|available]].
 
Note: A distributable version of the viewer only, which does not require Python, is [[PDS4 Viewer|available]].
Line 107: Line 112:
 
==== Option 1 ====
 
==== Option 1 ====
  
Use "<tt>pip install PDS4_tools-0.4.zip</tt>" or "<tt>easy_install PDS4_tools-0.4.zip</tt>". You can also extract the ZIP file and use "<tt>python /path/to/extracted/setup.py install</tt>". Note that there is no uninstall script provided (although "<tt>pip uninstall pds4_tools</tt>" should work), and that this tool will be updated in the future.
+
Use "<tt>pip install PDS4_tools-1.3.zip</tt>"  
  
 
==== Option 2 ====
 
==== Option 2 ====
Line 115: Line 120:
 
<pre>
 
<pre>
 
import sys
 
import sys
sys.path.extend(['/path/to/your/extraction/directory'])
+
sys.path.extend([r'/path/to/extraction_directory/'])
  
# On a windows machine use backslashes (/) instead of windows' normal forward slashes to specify paths
+
import pds4_tools
 
</pre>
 
</pre>
  
 
=== Example Usage ===
 
=== Example Usage ===
 +
 +
'''See also the [https://pdssbn.astro.umd.edu/tools/pds4_tools_docs/current/ User Manual].'''
  
 
==== pds4_read ====
 
==== pds4_read ====
  
You may call <tt>pds4_read</tt> from command line or from your own script.  The following is the docstring for <tt>pds4_read</tt>:
+
Import via "<tt>import pds4_tools</tt>". You may then call <tt>pds4_tools.read()</tt> from your own code.  The following is the docstring for <tt>pds4_tools.read()</tt>:
  
 
<pre>
 
<pre>
    Reads PDS4 compliant data into a `StructureList`
+
        Reads PDS4 compliant data into a `StructureList`.
   
+
 
    Given a PDS4 label, reads the PDS4 data described in the label and
+
        Given a PDS4 label, reads the PDS4 data described in the label and
    associated label meta data into an `StructureList`, with each PDS4 data
+
        associated label meta data into a `StructureList`, with each PDS4 data
    structure (e.g. Array_2D, Table_Binary, etc) as its own `Structure`. By
+
        structure (e.g. Array_2D, Table_Binary, etc) as its own `Structure`. By
    default all data structures described in the label are read-in.
+
        default all data structures described in the label are immediately
   
+
        read into memory.
    Notes
+
 
    -----
+
        Notes
    Currently supports Array structures, Table_Character and Table_Binary.
+
        -----
    Packed bit fields in Table_Binary are not yet supported, all other
+
        Python 2 v. Python 3: Non-data strings (label, meta data, etc)  in
    features of previously mentioned structures are fully supported.
+
        Python 2 will be decoded to ``unicode`` and in Python 3 they will
   
+
        be decoded to ``str``. The return type of all data strings is
    Parameters
+
        controlled by *decode_strings*.
    ----------
+
 
    filename : str
+
        Remote URLs are downloaded into an on-disk cache which is cleared on
        The filename, including full or relative path if necessary, of
+
        Python interpreter exit.
        the PDS4 label describing the data.
+
 
    quiet : bool, optional
+
        Parameters
        Suppresses all info/warnings from being output.
+
        ----------
    use_numpy : bool, optional
+
        filename : str or unicode
        Returned data will be an ndarray and use NumPy data types.
+
            The filename, including full or relative path, or a remote
        Defaults to True if NumPy is installed.
+
            URL to the PDS4 label describing the data.
    structure_num : integer, optional
+
        quiet : bool, int or str, optional
        Instead of reading all data structures, only read the n^th
+
            If True, suppresses all info/warnings from being output to stdout.
        structure, where n = structure_num and is zero-based.
+
            Supports log-level style options for more fine grained control.
    structure_name : str, optional
+
            Defaults to False.
        Instead of reading all data structures, only read the structure
+
        lazy_load : bool, optional
        with a name equal to structure_name.
+
            If True, then the data of each PDS4 data structure will not be
    structure_lid : str, optional
+
            read-in to memory until the first attempt to access it. Additionally,
        Instead of reading all data structures, only read the structure
+
            for remote URLs, the data is not downloaded until first access.
        with a local identifier equal to structure_lid.
+
            Defaults to False.
   
+
        no_scale : bool, optional
    Returns
+
            If True, returned data will be exactly as written in the data file,
    -------
+
            ignoring offset or scaling values. Defaults to False.
    StructureList
+
        decode_strings : bool, optional
        Contains PDS4 data `Structure`s, each of which contains the data,
+
            If True, strings data types contained in the returned data will be
        the meta data and the label portion describing that data structure.
+
            decoded to the a unicode in Python 2, and to the str type in
        `StructureList` can be treated/accessed/used like a ``dict`` or
+
            Python 3. If False, leaves string types as byte strings.
        ``list``.
+
            Defaults to True.
   
+
 
    Examples
+
        Returns
    --------
+
        -------
   
+
        StructureList
    Below we document how to read data described by an example label
+
            Contains PDS4 data `Structure`'s, each of which contains the data,
    which has two data structures, an Array_2D_Image and a Table_Binary.
+
            the meta data and the label portion describing that data structure.
    An outline of the label, including the array and a table with 3
+
            `StructureList` can be treated/accessed/used like a ``dict`` or
    fields, is given.
+
            ``list``.
   
+
 
    >>> struct_list = pds4_read('/path/to/Example_Label.xml')
+
        Examples
   
+
        --------
    Example Label Outline:
+
 
   
+
        Below we document how to read data described by an example label
        Array_2D_Image: unnamed
+
        which has two data structures, an Array_2D_Image and a Table_Binary.
        Table_Binary: Observations
+
        An outline of the label, including the array and a table with 3
            Field: order
+
        fields, is given.
            Field: wavelength
+
 
            Group: unnamed
+
        >>> # Local file
                Field: pos_vector
+
        >>> struct_list = pds4_tools.read('/path/to/Example_Label.xml')
   
+
 
    All below documentation assumes that the above outlined label,
+
        >>> # Remote URL
    containing an array that does not have a name indicated in the label,
+
        >>> struct_list = pds4_tools.read('https://url.com/Example_Label.xml')
    and a table that has the name 'Observations' with 3 fields as shown,
+
 
    has been read-in.
+
        Example Label Outline::
   
+
 
    Accessing Example Structures:
+
          Array_2D_Image: unnamed
   
+
          Table_Binary: Observations
        To access the data structures in `StructureList`, which is returned
+
              Field: order
        by pds4_read(), you may use any combination of `dict` or `list`.
+
              Field: wavelength
   
+
              Group: unnamed
        >>> unnamed_array = struct_list[0]
+
                  Field: pos_vector
        >>>              or struct_list['ARRAY_0']
+
 
   
+
        All below documentation assumes that the above outlined label,
        >>> obs_table = struct_list[1]
+
        containing an array that does not have a name indicated in the label,
        >>>          or struct_list['Observations']
+
        and a table that has the name 'Observations' with 3 fields as shown,
   
+
        has been read-in.
    Label or Structure Overview:
+
 
   
+
        Accessing Example Structures:
        To see a summary of the data structures, which for Arrays shows the
+
 
        type and dimensions of the array, and for Tables shows the type
+
            To access the data structures in `StructureList`, which is returned
        and number of fields, you may use the info() method. Calling
+
            by `pds4_read()`, you may use any combination of ``dict``-like or
        info() on a specific `Structure` instead of `StructureList` will
+
            ``list``-like access.
        provide a more detailed summary, including all Fields for a table.
+
 
   
+
            >>> unnamed_array = struct_list[0]
        >>> struct_list.info()
+
            >>>              or struct_list['ARRAY_0']
        >>> unnamed_array.info()
+
 
        >>> obs_table.info()
+
            >>> obs_table = struct_list[1]
   
+
            >>>          or struct_list['Observations']
    Accessing Example Label data:
+
 
   
+
        Label or Structure Overview:
        To access the read-in data, as an array-like (either list,
+
 
        array.array or ndarray), you can use the data attribute for a
+
            To see a summary of the data structures, which for Arrays shows the
        PDS4 Array data structure, or the field() method to access a field
+
            type and dimensions of the array, and for Tables shows the type
        for a table.
+
            and number of fields, you may use the `StructureList.info()` method.
   
+
            Calling `Structure.info()` on a specific ``Structure`` instead will
        >>> unnamed_array.data
+
            provide a more detailed summary, including all Fields for a table.
        >>> obs_table.field('wavelength')
+
 
        >>> obs_table.field('pos_vector')
+
            >>> struct_list.info()
   
+
            >>> unnamed_array.info()
    Accessing Example Label meta data:
+
            >>> obs_table.info()
   
+
 
        You can access all meta data in the label for a given PDS4 data
+
        Accessing Example Label data:
        structure or field via the `OrderedDict` meta_data attribute. The
+
 
        below examples use the 'description' element.
+
            To access the read-in data, as an array-like (subclass of ``ndarray``),
   
+
            you can use the data attribute for a PDS4 Array data structure, or
        >>> unnamed_array.meta_data['description']
+
            list-like and the field() method to access a field for a table.
   
+
 
        >>> obs_table.field('wavelength').meta_data['description']
+
            >>> # PDS4 Arrays
        >>> obs_table.field('pos_vector').meta_data['description']
+
            >>> unnamed_array.data
   
+
 
    Accessing Example Label:
+
            >>> # PDS4 Table fields
   
+
            >>> obs_table['wavelength']
        The XML for a label is also accessible via the label attribute,
+
            >>> obs_table.field('wavelength')
        either the entire label or for each PDS4 data structure.
+
 
   
+
            >>> # PDS4 Table records
        Entire label:
+
            >>> obs_table[0:1000]
            >>> struct_list.label
+
 
   
+
        Accessing Example Label meta data:
        Part of label describing Observations table:
+
 
            >>> struct_list['Observations'].label
+
            You can access all meta data in the label for a given PDS4 data
            >>> struct_list[1].label
+
            structure or field via the ``OrderedDict`` meta_data attribute. The
   
+
            below examples use the 'description' element.
        The returned object is similar to an ElementTree instance. It is
+
 
        searchable via find() and findall() methods and XPATH. Consult
+
            >>> unnamed_array.meta_data['description']
        ElementTree manual for more details. For example,
+
 
   
+
            >>> obs_table.field('wavelength').meta_data['description']
        >>> struct_list.label.findall('.//disp:Display_Settings')
+
            >>> obs_table.field('pos_vector').meta_data['description']
   
+
 
        Will find all elements in the entire label named 'Display_Settings'
+
        Accessing Example Label:
        which are in the 'disp' namespace. You can additionally use the
+
 
        to_dict() and to_string() methods.
+
            The XML for a label is also accessible via the label attribute,
 +
            either the entire label or for each PDS4 data structure.
 +
 
 +
            Entire label:
 +
                >>> struct_list.label
 +
 
 +
            Part of label describing Observations table:
 +
                >>> struct_list['Observations'].label
 +
                >>> struct_list[1].label
 +
 
 +
            The returned object is similar to an ElementTree instance. It is
 +
            searchable via `Label.find()` and `Label.findall()` methods and XPATH.
 +
            Consult ``ElementTree`` manual for more details. For example,
 +
 
 +
            >>> struct_list.label.findall('.//disp:Display_Settings')
 +
 
 +
            Will find all elements in the entire label named 'Display_Settings'
 +
            which are in the 'disp' prefix's namespace. You can additionally use the
 +
            `Label.to_dict()` and `Label.to_string()` methods.
 +
</pre>
 +
 
 +
Usage is described above. A basic usage example is as follows:
 +
 
 +
<pre>
 +
""" Basic Reader example """
 +
 
 +
import pds4_tools
 +
 
 +
structures = pds4_tools.read('/path/to/label.xml')
 +
 
 +
structures.info()
 +
 
 +
0 - Array_3D_Spectrum 'table_name' (3 axes, 21 x 10 x 36)
 +
1 - Table_Binary 'array_name' (5 fields x 1000 records)
 +
 
 +
# Table data access
 +
table = structures['table_name'] # or
 +
table = structures[0]
 +
 
 +
table.info()
 +
 
 +
field_data = table.field('field_name') # or
 +
field_data = table.fields[0]
 +
 
 +
record_data = table[0:50]
 +
 
 +
# Array data access
 +
array = structures['array_name'] # or
 +
array = structures[1]
 +
 
 +
array_data = array.data
 +
 
 +
# Meta-data access
 +
field_meta = table.field('field_name').meta_data # or
 +
field_meta = table.fields[0].meta_data
 +
array_meta = array.meta_data
 +
 
 +
print field_meta['description']
 +
print field_meta['unit']
 +
print array_meta['local_identifier']
 +
 
 +
# Label access
 +
label = structures.label # Full label
 +
label = table.label      # Label section describing the table object
 +
 
 +
display_settings = label.findall('.//disp:Display_Settings')
 +
 
 +
display_dict = display_settings.to_dict()
 +
label_dict = label.to_dict()
 +
label_string = label.to_string()
 +
 
 
</pre>
 
</pre>
  
 
==== pds4_viewer ====
 
==== pds4_viewer ====
  
 +
Import via "<tt>import pds4_tools</tt>". To display the data structures (such as images, spectra, or tables) in a label you may then call <tt>pds4_tools.view()</tt> from the Python interpreter, with or without any arguments:
  
To display the data structures (such as images, spectra, or tables) in a label you may call <tt>pds4_viewer</tt> from the command line, or import it in the Python interpreter:
+
<pre>
 +
    Displays PDS4 compliant data in a GUI.
  
<pre>
 
    Displays PDS4 compliant data in a GUI
 
   
 
 
     Given a PDS4 label, displays PDS4 data described in the label and
 
     Given a PDS4 label, displays PDS4 data described in the label and
 
     associated label meta data in a GUI. By default all data structures described
 
     associated label meta data in a GUI. By default all data structures described
Line 273: Line 349:
 
     parameters, opening a GUI that has a File->Open function to select
 
     parameters, opening a GUI that has a File->Open function to select
 
     desired label to be read-in and displayed.
 
     desired label to be read-in and displayed.
   
+
 
     Parameters:
+
     Parameters
      
+
     ----------
        filename: str, optional
+
    filename : str or unicode, optional
            The filename, including full or relative path if necessary, of
+
        The filename, including full or relative path if necessary, of
            the PDS4 label describing the data to be viewed.
+
        the PDS4 label describing the data to be viewed.
        from_existing_structures: StructureList, optional
+
    from_existing_structures : StructureList, optional
            An existing StructureList, as returned by pds4_read(), to view. Takes
+
        An existing StructureList, as returned by pds4_read(), to view. Takes
            precedence if given together with filename.
+
        precedence if given together with filename.
        quiet: bool, optional
+
    lazy_load : bool, optional
            Suppresses all info/warnings from being output and displayed.
+
         Do not read-in data of each data structure until attempt to view said
         structure_num: integer, optional
+
         data structure. Defaults to True.
            Instead of reading all data structures, only read the n^th
+
    quiet : bool, int or str, optional
            structure, where n = structure_num.
+
        Suppresses all info/warnings from being output and displayed. Supports
         structure_name: str, optional
+
        log-level style options for more fine grained control. Defaults to False.
            Instead of reading all data structures, only read the structure
 
            with a name equal to structure_name.
 
        structure_lid: str, optional
 
            Instead of reading all data structures, only read the structure
 
            with a local identifier equal to structure_lid.
 
 
</pre>
 
</pre>
  
It is not necessary to include the filename parameter for <tt>pds4_viewer</tt>, you may simplify call it without any options or arguments and a GUI will open from which you can open labels.  
+
It is not necessary to include the filename parameter for <tt>pds4_tools.view</tt>, you may simplify call it without any options or arguments and a GUI will open from which you can open labels.  
  
You may also call <tt>pds4_viewer</tt> from another module or script. All the above arguments are available as optional named parameters. A basic example usage is as follows:
+
You may also call <tt>pds4_tools.view</tt> from another module or script. All the above arguments are available as optional named parameters. A basic example usage is as follows:
  
 
<pre>
 
<pre>
""" Basic pds4_viewer example """
+
""" Basic Viewer example """
  
from pds4_tools import pds4_read, pds4_viewer
+
import pds4_tools
  
pds4_viewer()
+
pds4_tools.view()
  
 
# or
 
# or
  
pds4_viewer('/path/to/label.xml')
+
pds4_tools.view('/path/to/label.xml')
  
 
# or  
 
# or  
  
struct_list = pds4_read('label.xml')
+
struct_list = pds4_tools.view('label.xml')
pds4_viewer(from_existing_structures=struct_list) # Won't re-read the data
+
pds4_tools.view(from_existing_structures=struct_list) # Won't re-read the data
 
</pre>
 
</pre>
 +
 +
 +
 +
[[Category:Python]]
 +
[[Category:Tools]]

Latest revision as of 06:28, 7 November 2021

Python and PDS4

This document describes the current status and usage of Python tools developed at PDS-SBN to read and visualize PDS4 data in Python. Please note that a PDS4 reader and visualizer for IDL is also available.

Reading and Displaying PDS4 Data

Introduction

This section describes a Python package that can read and display PDS4 data and meta data. In the future this tool is expected to support all PDS4 data structures, currently support is limited to structures given in the Supported Data Structures section. The package expects valid PDS4 labels formatted according to the PDS4 Standard.

Contact Lev Nagdimunov with questions or comments regarding this code or its description.

Requirements

Python 2.6+ or 3.3+

pds4_read: NumPy
pds4_viewer: NumPy, matplotlib

Supported Data Structures

PDS4 Data Standards >= v1.0 are supported.
PDS3 Data Standards are not supported.

The table below lists the main PDS4 data structures and the current status.

Read-in column indicates support by pds4_tools.read()
Display columns indicate support by pds4_tools.view().

Structure Read-in Display as Table Display as Image Display Columns as Plot
Header Yes No No No
Array Yes Yes Yes, N-dims Yes, 1-D only
Array_2D Yes Yes Yes No
Array_2D_* Yes Yes Yes No
Array_3D Yes Yes Yes No
Array_3D_* Yes Yes Yes No
Table_Character Yes Yes No Yes
Table_Binary Yes, except BitFields Yes No Yes
Table_Delimited Yes Yes No Yes
Composite_Structure No No No No

User Manual

Online usage documentation, both for scientists and for developers, is available.

Download

Download the ZIP file File:PDS4 tools-1.3.zip. Released on October 10, 2021.

Note: A distributable version of the viewer only, which does not require Python, is available.

Installation

Option 1

Use "pip install PDS4_tools-1.3.zip"

Option 2

Extract the downloaded file to a directory Python can find. To use it follow the instructions in Example Usage except with the following lines first,

import sys
sys.path.extend([r'/path/to/extraction_directory/'])

import pds4_tools

Example Usage

See also the User Manual.

pds4_read

Import via "import pds4_tools". You may then call pds4_tools.read() from your own code. The following is the docstring for pds4_tools.read():

        Reads PDS4 compliant data into a `StructureList`.

        Given a PDS4 label, reads the PDS4 data described in the label and
        associated label meta data into a `StructureList`, with each PDS4 data
        structure (e.g. Array_2D, Table_Binary, etc) as its own `Structure`. By
        default all data structures described in the label are immediately
        read into memory.

        Notes
        -----
        Python 2 v. Python 3: Non-data strings (label, meta data, etc)  in
        Python 2 will be decoded to ``unicode`` and in Python 3 they will
        be decoded to ``str``. The return type of all data strings is
        controlled by *decode_strings*.

        Remote URLs are downloaded into an on-disk cache which is cleared on
        Python interpreter exit.

        Parameters
        ----------
        filename : str or unicode
            The filename, including full or relative path, or a remote
            URL to the PDS4 label describing the data.
        quiet : bool, int or str, optional
            If True, suppresses all info/warnings from being output to stdout.
            Supports log-level style options for more fine grained control.
            Defaults to False.
        lazy_load : bool, optional
            If True, then the data of each PDS4 data structure will not be
            read-in to memory until the first attempt to access it. Additionally,
            for remote URLs, the data is not downloaded until first access.
            Defaults to False.
        no_scale : bool, optional
            If True, returned data will be exactly as written in the data file,
            ignoring offset or scaling values. Defaults to False.
        decode_strings : bool, optional
            If True, strings data types contained in the returned data will be
            decoded to the a unicode in Python 2, and to the str type in
            Python 3. If False, leaves string types as byte strings.
            Defaults to True.

        Returns
        -------
        StructureList
            Contains PDS4 data `Structure`'s, each of which contains the data,
            the meta data and the label portion describing that data structure.
            `StructureList` can be treated/accessed/used like a ``dict`` or
            ``list``.

        Examples
        --------

        Below we document how to read data described by an example label
        which has two data structures, an Array_2D_Image and a Table_Binary.
        An outline of the label, including the array and a table with 3
        fields, is given.

        >>> # Local file
        >>> struct_list = pds4_tools.read('/path/to/Example_Label.xml')

        >>> # Remote URL
        >>> struct_list = pds4_tools.read('https://url.com/Example_Label.xml')

        Example Label Outline::

           Array_2D_Image: unnamed
           Table_Binary: Observations
               Field: order
               Field: wavelength
               Group: unnamed
                   Field: pos_vector

        All below documentation assumes that the above outlined label,
        containing an array that does not have a name indicated in the label,
        and a table that has the name 'Observations' with 3 fields as shown,
        has been read-in.

        Accessing Example Structures:

            To access the data structures in `StructureList`, which is returned
            by `pds4_read()`, you may use any combination of ``dict``-like or
            ``list``-like access.

            >>> unnamed_array = struct_list[0]
            >>>              or struct_list['ARRAY_0']

            >>> obs_table = struct_list[1]
            >>>          or struct_list['Observations']

        Label or Structure Overview:

            To see a summary of the data structures, which for Arrays shows the
            type and dimensions of the array, and for Tables shows the type
            and number of fields, you may use the `StructureList.info()` method.
            Calling `Structure.info()` on a specific ``Structure`` instead will
            provide a more detailed summary, including all Fields for a table.

            >>> struct_list.info()
            >>> unnamed_array.info()
            >>> obs_table.info()

        Accessing Example Label data:

            To access the read-in data, as an array-like (subclass of ``ndarray``),
            you can use the data attribute for a PDS4 Array data structure, or
            list-like and the field() method to access a field for a table.

            >>> # PDS4 Arrays
            >>> unnamed_array.data

            >>> # PDS4 Table fields
            >>> obs_table['wavelength']
            >>> obs_table.field('wavelength')

            >>> # PDS4 Table records
            >>> obs_table[0:1000]

        Accessing Example Label meta data:

            You can access all meta data in the label for a given PDS4 data
            structure or field via the ``OrderedDict`` meta_data attribute. The
            below examples use the 'description' element.

            >>> unnamed_array.meta_data['description']

            >>> obs_table.field('wavelength').meta_data['description']
            >>> obs_table.field('pos_vector').meta_data['description']

        Accessing Example Label:

            The XML for a label is also accessible via the label attribute,
            either the entire label or for each PDS4 data structure.

            Entire label:
                >>> struct_list.label

            Part of label describing Observations table:
                >>> struct_list['Observations'].label
                >>> struct_list[1].label

            The returned object is similar to an ElementTree instance. It is
            searchable via `Label.find()` and `Label.findall()` methods and XPATH.
            Consult ``ElementTree`` manual for more details. For example,

            >>> struct_list.label.findall('.//disp:Display_Settings')

            Will find all elements in the entire label named 'Display_Settings'
            which are in the 'disp' prefix's namespace. You can additionally use the
            `Label.to_dict()` and `Label.to_string()` methods.

Usage is described above. A basic usage example is as follows:

""" Basic Reader example """

import pds4_tools

structures = pds4_tools.read('/path/to/label.xml')

structures.info()

0 - Array_3D_Spectrum 'table_name' (3 axes, 21 x 10 x 36)
1 - Table_Binary 'array_name' (5 fields x 1000 records)

# Table data access
table = structures['table_name'] # or
table = structures[0]

table.info()

field_data = table.field('field_name') # or
field_data = table.fields[0] 

record_data = table[0:50]

# Array data access
array = structures['array_name'] # or
array = structures[1]

array_data = array.data

# Meta-data access
field_meta = table.field('field_name').meta_data # or
field_meta = table.fields[0].meta_data
array_meta = array.meta_data

print field_meta['description']
print field_meta['unit']
print array_meta['local_identifier']

# Label access
label = structures.label # Full label
label = table.label      # Label section describing the table object

display_settings = label.findall('.//disp:Display_Settings')

display_dict = display_settings.to_dict()
label_dict = label.to_dict()
label_string = label.to_string()

pds4_viewer

Import via "import pds4_tools". To display the data structures (such as images, spectra, or tables) in a label you may then call pds4_tools.view() from the Python interpreter, with or without any arguments:

    Displays PDS4 compliant data in a GUI.

    Given a PDS4 label, displays PDS4 data described in the label and
    associated label meta data in a GUI. By default all data structures described
    in the label are read-in and displayed. Can be called without any
    parameters, opening a GUI that has a File->Open function to select
    desired label to be read-in and displayed.

    Parameters
    ----------
    filename : str or unicode, optional
        The filename, including full or relative path if necessary, of
        the PDS4 label describing the data to be viewed.
    from_existing_structures : StructureList, optional
        An existing StructureList, as returned by pds4_read(), to view. Takes
        precedence if given together with filename.
    lazy_load : bool, optional
        Do not read-in data of each data structure until attempt to view said
        data structure. Defaults to True.
    quiet : bool, int or str, optional
        Suppresses all info/warnings from being output and displayed. Supports
        log-level style options for more fine grained control. Defaults to False.

It is not necessary to include the filename parameter for pds4_tools.view, you may simplify call it without any options or arguments and a GUI will open from which you can open labels.

You may also call pds4_tools.view from another module or script. All the above arguments are available as optional named parameters. A basic example usage is as follows:

""" Basic Viewer example """

import pds4_tools

pds4_tools.view()

# or

pds4_tools.view('/path/to/label.xml')

# or 

struct_list = pds4_tools.view('label.xml')
pds4_tools.view(from_existing_structures=struct_list) # Won't re-read the data