Difference between revisions of "Filling Out the Packed Data Fields Sub-Structure"

From The SBN Wiki
Jump to navigation Jump to search
m
(Update for IM 1.14.0.0)
 
Line 1: Line 1:
 
The '''''<Packed_Data_Fields>''''' class is used to define multiple sub-fields within a single ''<Field_Binary>'' value.  These values do not have to observe byte boundaries, and so are defined in terms of bits.
 
The '''''<Packed_Data_Fields>''''' class is used to define multiple sub-fields within a single ''<Field_Binary>'' value.  These values do not have to observe byte boundaries, and so are defined in terms of bits.
  
{| class="wikitable" style="background-color: thistle"
+
{| class="wikitable" style="background-color: Yellow"
| '''''Note:''''' ''The whole ''Packed_Data_Field'' scheme is poorly defined.  The class is not mentioned in the Standards Reference at all. The Data Dictionary does not (and can't, really) put constraints - including a byte order constraint - on the containing ''<Field_Binary>'', nor does it specify bit order (although "most significant bit first" is almost certainly correct).''
+
| '''''Note:''''' Packed Data ''is prohibited in observational archive files. Consequently, SBN data preparers should not be using the ''<Packed_Data_Fields>'' structure for any observational data products, and probably not anywhere else, either.  Any packed data should be extracted into standard data types before archiving.  If you think you have a data set that should be an exception to this rule, contact your SBN consultant.''
 
 
''SBN data preparers should not use ''<Packed_Data_Fields>''.  Any packed data should be extracted into standard data types before archiving.  If you think you have a data set that should be an exception to this rule, contact your SBN consultant.''
 
 
|}
 
|}
 
  
 
For additional explanation, see the PDS4 ''Standards Reference'', or contact your PDS node consultant.
 
For additional explanation, see the PDS4 ''Standards Reference'', or contact your PDS node consultant.
Line 42: Line 39:
 
''OPTIONAL''
 
''OPTIONAL''
  
{| class="wikitable" style="background-color: thistle"
+
{| class="wikitable" style="background-color: yellow"
| ''The definition for this attribute is not sufficiently specific for it to be used reliably.  It is not clear, for example, whether the ''field_numbers'' within a ''Packed_Data_Field'' always start at one, or continue numbering started at a higher level.  If you must label packed data before its various issues are resolved, just do something reasonably with this and document what you've done in the ''description'' attribute of the ''<Packed_Data_Fields>'' class.''
+
| ''The definition for this attribute is not sufficiently specific for it to be used reliably.  It is not clear, for example, whether the ''field_numbers'' within a ''Packed_Data_Field'' always start at one, or continue numbering started at a higher level.''
 
|}
 
|}
  
 
+
=== <start_bit_location> ===
=== <start_bit> ===
 
 
 
  
 
''REQUIRED''
 
''REQUIRED''
  
The first bit comprising the value in the bit field.  Numbering begins at 1.   
+
The first bit comprising the value in the bit field, relative to the start of the string of bytes containing the field.  Numbering begins at 1.   
  
{| class="wikitable" style="background-color: thistle"
+
Note that the sequence of bytes containing packed data must be read in physical sequence from the file (i.e., as if they were characters). Then bit 1 is the most significant bit in the sequentially first byte in the field, and the bits are numbered sequentially across byte boundaries for the length of the field.
| It does not appear to be explicitly stated, but the only rational way to approach the bit-parsing problem is to require that the containing field be treated as character data - that is, the bytes are left in their file storage order for parsing. Then bit 1 is the most significant bit in the sequentially first byte in the field, and the bits are numbered sequentially across byte boundaries for the length of the field.
 
|}
 
  
=== <stop_bit> ===
+
=== <stop_bit_location> ===
  
 
''REQUIRED''
 
''REQUIRED''
  
The last bit comprising the value in the bit field.
+
The last bit comprising the value in the bit field, also relative to the start of the string of bytes containing the field. Numbering begins at 1.
  
 
=== <data_type> ===
 
=== <data_type> ===
Line 68: Line 61:
 
''REQUIRED''
 
''REQUIRED''
  
This must be one of two standard values: '''SignedBitString''' or '''UnsignedBitString'''.
+
This must be one of two standard values: '''SignedBitString''' or '''UnsignedBitString'''. These data types are described in section 5C.4 of the ''Standards Reference''.  What it boils down to is this:
{| class="wikitable" style="background-color: thistle"
+
:* '''UnsignedBitString''' values should be interpreted as unsigned integers, and so should be zero-extended on the left as they are transferred into standard integer fields.
| ''Fair enough, but I don't see how this helps anythingIt seems like what is needed is an indication of the data type the bit string should be converted into, since no modern programming language that I know of can manipulate fractional bytes directly.''
+
:* '''SignedBitString''' values are two's complement integers, and should be sign-extended into standard integer fields.
|}
 
{| class="wikitable" style="background-color: thistle"
 
| '''''Note:''''' ''These standard values do not follow the syntax pattern of most standard values, and in particular of other ''data_type'' values.''
 
|}
 
  
 
=== <field_format> ===
 
=== <field_format> ===
  
 
''OPTIONAL''
 
''OPTIONAL''
{| class="wikitable" style="background-color: thistle"
+
 
| I don't see how you can possibly use this attribute without a programmatic data type in the corresponding ''data_type'' field.
+
The only rational formats to have here would be for integers, although possibly for a non-decimal integer (hex codes, for example).  This, however, is not validated.
|}
 
  
 
=== <unit> ===
 
=== <unit> ===
 +
 
''OPTIONAL''
 
''OPTIONAL''
 +
 
If there is a unit of measurement associated with the extracted value, here's where it goes.
 
If there is a unit of measurement associated with the extracted value, here's where it goes.
 +
 
=== <scaling_factor> ===
 
=== <scaling_factor> ===
 +
 
''OPTIONAL''
 
''OPTIONAL''
 +
 
If the original value was scaled to fit into the bit field, this attribute is the place to put the scaling factor to be applied when parsing the data into program memory.  The value extracted from the packed data field is multiplied by this value before any ''value_offset'' is applied.
 
If the original value was scaled to fit into the bit field, this attribute is the place to put the scaling factor to be applied when parsing the data into program memory.  The value extracted from the packed data field is multiplied by this value before any ''value_offset'' is applied.
 +
 
=== <value_offset> ===
 
=== <value_offset> ===
 +
 
''OPTIONAL''
 
''OPTIONAL''
 +
 
If a value was applied to the original value to fit it into the bit field, this attribute is the place to put that offset value for use when parsing the data into program memory.  The value extracted from the packed data field is first multiplied by the ''scaling_factor'', then this ''value_offset'' is added.
 
If a value was applied to the original value to fit it into the bit field, this attribute is the place to put that offset value for use when parsing the data into program memory.  The value extracted from the packed data field is first multiplied by the ''scaling_factor'', then this ''value_offset'' is added.
 +
 
=== <description> ===
 
=== <description> ===
 +
 
''OPTIONAL''
 
''OPTIONAL''
 +
 
This attribute holds free-format text containing a description of the extracted bit field value.
 
This attribute holds free-format text containing a description of the extracted bit field value.
 +
 
=== <Special_Constants> ===
 
=== <Special_Constants> ===
 +
 
''OPTIONAL''
 
''OPTIONAL''
  

Latest revision as of 18:36, 3 August 2020

The <Packed_Data_Fields> class is used to define multiple sub-fields within a single <Field_Binary> value. These values do not have to observe byte boundaries, and so are defined in terms of bits.

Note: Packed Data is prohibited in observational archive files. Consequently, SBN data preparers should not be using the <Packed_Data_Fields> structure for any observational data products, and probably not anywhere else, either. Any packed data should be extracted into standard data types before archiving. If you think you have a data set that should be an exception to this rule, contact your SBN consultant.

For additional explanation, see the PDS4 Standards Reference, or contact your PDS node consultant.

Following are the attributes and subclasses you'll find in the <Packed_Data_Fields> class, in label order.

Note that in the PDS4 master schema, all classes have capitalized names; attributes never do.

<bit_fields>

REQUIRED

This attribute contains the number of <Field_Bit> classes in this <Packed Data Field>.

<description>

OPTIONAL

This attribute provides space for a free-format text description of the packed data.

<Field_Bit>

REQUIRED

This class, analogous to the <Field_Binary> class, defines a binary field in terms of bits rather than bytes. There must be at least one of these in the Packed_Data_Fields class, and there may be many.

<name>

REQUIRED

The name of the bit field.

<field_number>

OPTIONAL

The definition for this attribute is not sufficiently specific for it to be used reliably. It is not clear, for example, whether the field_numbers within a Packed_Data_Field always start at one, or continue numbering started at a higher level.

<start_bit_location>

REQUIRED

The first bit comprising the value in the bit field, relative to the start of the string of bytes containing the field. Numbering begins at 1.

Note that the sequence of bytes containing packed data must be read in physical sequence from the file (i.e., as if they were characters). Then bit 1 is the most significant bit in the sequentially first byte in the field, and the bits are numbered sequentially across byte boundaries for the length of the field.

<stop_bit_location>

REQUIRED

The last bit comprising the value in the bit field, also relative to the start of the string of bytes containing the field. Numbering begins at 1.

<data_type>

REQUIRED

This must be one of two standard values: SignedBitString or UnsignedBitString. These data types are described in section 5C.4 of the Standards Reference. What it boils down to is this:

  • UnsignedBitString values should be interpreted as unsigned integers, and so should be zero-extended on the left as they are transferred into standard integer fields.
  • SignedBitString values are two's complement integers, and should be sign-extended into standard integer fields.

<field_format>

OPTIONAL

The only rational formats to have here would be for integers, although possibly for a non-decimal integer (hex codes, for example). This, however, is not validated.

<unit>

OPTIONAL

If there is a unit of measurement associated with the extracted value, here's where it goes.

<scaling_factor>

OPTIONAL

If the original value was scaled to fit into the bit field, this attribute is the place to put the scaling factor to be applied when parsing the data into program memory. The value extracted from the packed data field is multiplied by this value before any value_offset is applied.

<value_offset>

OPTIONAL

If a value was applied to the original value to fit it into the bit field, this attribute is the place to put that offset value for use when parsing the data into program memory. The value extracted from the packed data field is first multiplied by the scaling_factor, then this value_offset is added.

<description>

OPTIONAL

This attribute holds free-format text containing a description of the extracted bit field value.

<Special_Constants>

OPTIONAL

This class defines flag values used to indicate that a particular field value is unknown for one reason or another. It is identical to the <Special_Constants> class used in the Array classes. For details, check the Filling Out the Array 2D Data Structure - <Special_Constants> page. Here is a quick list of the special constants available in this class:

  • saturated_constant
  • missing_constant
  • error_constant
  • invalid_constant
  • unknown_constant
  • not_applicable_constant