GruanToolRS92 (gt92)

Command line tool for handling radiosonde data files

Details

ID

gt92

Title

GruanToolRs92

Author

Michael Sommer (GRUAN Lead Centre)

Brief description

Command line tool which can be used to handle a couple of radiosounding-specific file formats. Files can be tested, converted, extracted and more.

Version

0.5.5_01 (2023-05-04)

Status

Stable

 

Description

  • GRUAN-TN-11: M. Sommer: Brief Description of GruanToolRs92 (gt92), GRUAN Technical Note 11 (GRUAN-TN-11), v1.0 (2020-10-01). PDF

History

  • 0.5.5_01 (2023-05-04)
    • improve handling of version 2.3 of XDATA type 'PCFH_3C' packets (e.g. golden points)
  • 0.5.5 (2023-03-15)
    • optimised dependencies (Maven)

  • 0.5.4_05 (2023-03-13)
    • bug fixed: loading config file within DC3DB file reader
    • bug fixed: time column of gridding table within DC3DB file reader
  • 0.5.4_04 (2023-01-27)
    • add a hidden option for MWX
  • 0.5.4_03 (2023-01-17)
    • update calculation of SKYDEW xData columns (HS and CB)
  • 0.5.4_02 (2022-05-05)
    • add option --special-case in combination with -f of GSFZ
  • 0.5.4_01 (2022-02-09)
    • add warning in case of unknown special format cases
    • bug fixed: title of column 'time_stamp' will be formatted correctly
  • 0.5.4 (2022-02-09)
    • project and dependencies switched to Maven

  • 0.5.3_04 (2021-12-02)
    • rename option --event to --special-event
    • add special format cases 'UAII2022Laboratory' and 'UAII2022Sounding'
    • add option --csv-compress to use in combination with --special-case
    • allow grid maps with double grid base (time steps) columns (original and copy) with different data type, e.g. DOUBLE and DATETIME
  • 0.5.3_03 (2021-12-01)
    • add option --event to use in combination with --special-case
  • 0.5.3_02 (2021-11-26)
    • add option --special-case in combination with -f of MWX
  • 0.5.3_01 (2021-08-04)
    • repair calculating of final T, U and Ts values of JMAFMT data file format
  • 0.5.3 (2021-07-23)
    • add handling of JMAFMT data file format (calculating p,T,U,Ts is incorrect at moment)

  • 0.5.2_02 (2021-06-11)
    • optimise calculation and checking calculated raw values of DFM-17 within GSFZ
    • add column 'RawTempCorrGraw' to export maps 'RAW_ALL_GRIDDED' and 'RAW_TU_GRIDDED' of converted GSFZ
  • 0.5.2_01 (2021-06-02)
    • bug fixed: crash if no XData table is available within GSFZ
  • 0.5.2 (2021-02-18)
    • add handling of XDATA type 'POPS_38' packets
    • add handling of XDATA type 'ASLWC_29' packets
    • add handling of version 2.1 of XDATA type 'PCFH_3C' packets
    • add decoding of XDATA within GSFZ files
    • bug fixed: crash if GSFZ file contains 'xdata.gsf'

  • 0.5.1_04 (2020-10-30)
    • change handling of final version of DFM-17 within GSFZ
  • 0.5.1_03 (2020-10-21)
    • bug fixed: crash if option --meta-out-file is used and no files/metadata are available
    • bug fixed: 'null' was printed as delimiter in metadata collection files if no delimiter was given
  • 0.5.1_02 (2020-09-22)
    • bug fixed: more tolerant detection of STRATO-ZIP files
    • bug fixed: option --no-meta-group was always switched on
    • change: small changes of descriptions of options
  • 0.5.1_01 (2020-08-18)
    • change: improve MWX file definition for MW41 v2.15
  • 0.5.1 (2020-08-04)
    • change: update file format GSFZ to GrawMet v5.15.02.02
    • add handling of DFM-17 within GSFZ

  • 0.5.0_05 (2020-04-29)
    • change: update file format IGNC-RAW to v1.2.0
  • 0.5.0_04 (2020-04-28)
    • bug fixed: crash using MwxFile because not inited localConfigMap in AbstractBaseDataFile
  • 0.5.0_03 (2020-04-22)
    • optimised and restructured 'DataFile' classes which allows easier adding of new config properties
    • add option --no-meta-out-units
  • 0.5.0_02 (2020-04-20)
    • add options --meta-out-file and --meta-out-delimiter
      • implemented for MWX, GNC-RAW (v1, v2), GNC-DATA, DC3DB, IGNC-RAW, STRATO-ZIP, GSFZ
  • 0.5.0_01 (2019-12-11)
    • bug fixed: wrong calculation of OIF411 PumpTemperature
  • 0.5.0 (2019-11-06)
    • new compiled only
    • first release version 0.5

  • 0.4.11_08 (2019-10-23)
    • bug fixed: skip gridding part of DC3DB in case of --itemize
    • change: extract SystemEvents as meta-data using last value in case of multiple events of same type
  • 0.4.11_07 (2019-09-12)
    • bug fixed: crash during CSV export
  • 0.4.11_06 (2019-08-28)
    • add options --export-col-filter and --reverse-col-filter and --csv-delimiter
  • 0.4.11_05 (2019-07-31)
    • change: update MWX file definition for MW41 v2.15
  • 0.4.11_04 (2019-05-07)
    • bug fixed: handling of wrong date format in GSFZ files
  • 0.4.11_03 (2019-04-02)
    • bug fixed: add 'missing_value' to several columns of XDATA type 'SKYDEW_3F'
  • 0.4.11_02 (2019-03-29)
    • bug fixed: repaired column names of XDATA type 'SKYDEW_3F'
    • rename file type SGNC-RAW to IGNC-RAW
  • 0.4.11_01 (2019-03-28)
    • bug fixed: decode abnormal package length 35 of XDATA type 'SKYDEW_3F'
  • 0.4.11 (2019-03-28)
    • add handling of XDATA type 'SKYDEW_3F' (SKYDEW) packets
    • compiled with Java 8

  • 0.4.10_09 (2019-03-19)
    • add handling of file type SGNC-RAW
  • 0.4.10_08 (2019-02-07)
    • better handling of maybe corrupt system-wide installed netcdf libraries (print messages)
    • compiled with Java SE 11
  • 0.4.10_07 (2018-12-17)
    • bug fixed: wrong XDATA match if both PCFH and CFH are flight together using one data stream (MWX file)
  • 0.4.10_06 (2018-09-27)
    • change: update MWX file definition for MW41 v2.11
  • 0.4.10_05 (2018-09-27)
    • change: SYNC_* columns added/activated for table EDT_XDATA_GRIDDED of file type MWX
  • 0.4.10_04 (2018-07-24)
    • bug fixed: correct global attribute key 'g.Instrument.PCFH.MirNum' to '...SerialNo' in case of handling of XDATA type 'PCFH_3C' (PCFH)
    • bug fixed: add columns PCFH_Trsurf1, PCFH_Trsurf2, PCFH_2_Trsurf1, PCFH_2_Trsurf2 to table 'EDT_XDATA_GRIDDED' in case of handling of XDATA type 'PCFH_3C' (PCFH)
  • 0.4.10_03 (2018-07-13)
    • change: handling of XDATA type `PCFH_3C' (PCFH) packets 0x01 to 0x07, add TimestampXX columns
  • 0.4.10_02 (2018-05-31)
    • add option --checksum-type
      • with following possibilities: CRC-32, MD2, MD5, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512
      • with CRC-32 as default (like before)
  • 0.4.10_01 (2018-05-31)
    • change: update MWX file definition for MW41 v2.9
  • 0.4.10 (2018-05-19)
    • change: handling of XDATA type 'PCFH_3C' (PCFH) packets 0x00 to 0x07 (follow doc v1.2, 2018-04-05)

  • 0.4.9_02 (2018-04-16)
    • bug fixed: handle 'NULL' value in column DataSrvTime, table AdditionalSensorData of MWX file
  • 0.4.9_01 (2018-03-06)
    • add two hidden options for MWX
  • 0.4.9 (2018-02-28)
    • add handling of XDATA type 'PCFH_3C' (PCFH) packets 0x00 to 0x07

  • 0.4.8_03 (2018-01-05)
    • change: option --no-info-key is now also related to test messages (use of -t)
    • change: option --no-info-key is now also related to identify messages (use of -y)
    • change: option --no-info-key is now also related to checksum messages (use of -k)
    • add option --no-source for combination with -x (extract) and file type GNC-RAW
  • 0.4.8_02 (2017-12-08)
    • change: improved file check possibilities (value options added to attribute description)
    • bug fixed: test messages with file path now (instead 'null')
  • 0.4.8_01 (2017-11-01)
    • bug fixed: better error handling during identify DC3DB (in case of corrupt files)
  • 0.4.8 (2017-10-23)
    • change: enable options --no-meta-root and --meta-sep for file type DC3DB

  • 0.4.7_03 (2017-03-17)
    • bug fixed: CFH optics temperature fixed
  • 0.4.7_02 (2017-02-15)
    • bug fixed: using attribute 'g_format_format' if type BYTE or SHORT during converting GNC-DATA to LDF (crash fixed)
  • 0.4.7_01 (2016-11-24) - bug fixed: decoding TINT and ILAMP of XDATA type 'FLASH_3D_0' (FLASH-B)
    • add option --no-csv-nan for MWX, DC3DB, GNC-RAW, GNC-DATA
  • 0.4.7 (2016-11-23)
    • add handling of non-valid xml-chars (e.g. �) for internal XML file 'AdditionalSensorData.xml' of MWX file

  • 0.4.6_13 (2016-11-23)
    • bug fixed: wrong frame length of XDATA type 'FLASH_3D_0' (FLASH-B)
  • 0.4.6_12 (2016-11-22)
    • change handling of XDATA type 'FLASH_3D_0' (FLASH-B)
  • 0.4.6_11 (2016-11-16)
    • add handling of XDATA type 'FLASH_3D_0' (FLASH-B)
  • 0.4.6_10 (2016-11-16)
    • add handling of XDATA type 'FLASH_3D' (FLASH-B)
  • 0.4.6_09 (2016-11-10)
    • bug fixed: handling of data type BYTE with file type MWX
    • change: update MWX file definition for MW41 v2.5
    • add option --no-info-key
  • 0.4.6_08 (2016-11-09)
    • bug fixed: handling of XDATA with file type STRATO-ZIP
  • 0.4.6_07 (2016-11-08)
    • bug fixed: wrong results with a list of INFO keys in combination with HISTORY, HISTORY_LAST, HISTORY_FULL
    • bug fixed: wrong results with -y in case of file types without version
    • change: include tool version in converted GNC-RAW files (gatts: history, g.Product.ProducerTool)
    • add handling of XDATA type 'COVRAD_05_0110' (Cover of radiation chamber)
    • change: improved automatical detection of number type for MWX columns (FLOAT, DOUBLE, LONG)
  • 0.4.6_06 (2016-09-21)
    • bug fixed: always add global attribute 'g.Aliases' during converting to GNC-RAW
    • add new INFO keys: HISTORY_LAST, HISTORY, HISTORY_FULL
    • bug fixed: convert STRATO-ZIP (SkySonde) to GNC-RAW
    • bug fixed: use correct data type INT for COBALD counters
    • add additional meta-data (time) for STRATO RAW decoding & gidding
  • 0.4.6_05 (2016-09-13)
    • bug fixed: handling GSFZ files with extension *.zip
    • bug fixed: handling GNC-DATA files with missing ALIASES
  • 0.4.6_04 (2016-09-12)
    • bug fixed: handling several XData sensors of same type with MWX (CFH, CFH_2, ...)
  • 0.4.6_03 (2016-09-07)
    • bug fixed: handling GSFZ files
  • 0.4.6_02 (2016-08-29)
    • add BOM handling for internal XML files of MWX file
  • 0.4.6_01 (2016-08-01)
  • bug fixed: extract first 'correct' value with calculation
  • 0.4.6 (2016-08-01)
    • change options
      • add --nc-old (enable old version, default before)
      • remove --nc4, --nc4-v2 (--nc4-v2 is default now)
      • add --no-meta-valid, --meta-osep, --no-meta-root, --no-meta-group (use defaults for each file type)

  • 0.4.5_11 (2016-07-29)
    • add calculation to table columns (-> store result as attribute)
    • change gridded table of MWX
      • add additional columns for GPS
      • remove 'meta-data' columns, e.g. CFH, OIF411
      • add calc to gridded table, e.g. CFH, OIF411 (-> store result as global attributes)
  • 0.4.5_10 (2016-07-26)
    • bug fixed: use XML version 1.1 for parsing internal MWX files
  • 0.4.5_09 (2016-06-28)
    • bug fixed: use correct base time for all columns to grid (DC3DB and MWX)
    • add a gridded table to DC3DB
  • 0.4.5_08 (2016-06-22)
    • bug fixed: use correct path to write LDF files
    • bug fixed: first try to handle too short formats for LDF converting (negative ascent speed)
  • 0.4.5_07 (2016-06-10)
    • bug fixed: static reading of file descriptions of MWX, DC3DB, STRATO, GSFZ
  • 0.4.5_06 (2016-05-23)
    • new compiled
  • 0.4.5_05 (2016-02-26)
    • add XData specific meta-data (MWX only)
      • g.Instrument.XData, g.Instrument.XDataSensors
    • rename XData instrument RSA411 to OIF411
  • 0.4.5_04 (2016-02-25)
    • change some internal MWX things
      • e.g. add correct positions to a couple of tables
    • change base of gridded table of MWX to RawPtu
  • 0.4.5_03 (2016-02-23)
    • change column names of XDATA columns of MWX
      • Cfh_... to CFH_...
      • several identical instruments possible, like CFH_..., CFH_2_...
    • bug fixed: rename meta-data keys correctly to original once (if required)
  • 0.4.5_02 (2016-02-23)
    • add functionality of '--json-result' to option --info / -i
    • change print result (without --json-result) to <file/name> = <result>
    • add result line for option --extract / -x (EXTRACTED or NOT_EXTRACTED)
  • 0.4.5_01 (2016-02-22)
    • bug fixed: data type changed from BYTE to SHORT of column 'PumpCurrent' of map 'RawOzone' of format 'MWX'
    • bug fixed: unit 'seconds since ...' of column 'RadioRxTimePk' is now correct in case of converting MWX to GNC-RAW
    • add argument --export-table-filter
    • add argument --import-table-filter
    • add argument --reverse-table-filter
    • mark argument --table-filter as 'deprecated'
  • 0.4.5 (2016-02-18)
    • some changes at file descriptions (MWX, DC3DB, STRATO-ZIP)
    • with option -t (--test) a file can be tested against a file description
      • this is very helpful to test created GDP files (GRUAN data product NetCDF file)
      • all test result messages are printed like other results (at console 'out' without/with JSON format)

  • 0.4.4_05 (2016-02-16)
    • add automatically extracting of relevant native libraries to sub-dir 'gt92-native-libs' of temp dir of system
      • in case of no rights to write in own program folder
  • 0.4.4_04 (2016-02-16)
    • add argument --test-description=<FILE> (for -t)
    • bug fixed: use now argument --no-xdata (true/false) with --convert
  • 0.4.4_03 (2016-02-03)
    • optimization of messages (reading MWX)
    • add new table 'StabilityIndex' to MWX file description
    • change start time for gridded MWX tables to Soundings.BeginTime or if not available Soundings.RadioResetTime
    • bug fixed: wrong name of one V7 value
  • 0.4.4_02 (2016-02-02)
    • bug fixed: optimize gridded data map EDT_XDATA_GRIDDED of MWX file
      • set grid base to AdditionalSensors
      • add columns from RawPtu, GSupport, SynchronizedSoundingData
      • optimize messages if columns/maps are missing
  • 0.4.4_01 (2016-02-01)
    • add gridded data map EDT_XDATA_GRIDDED to MWX extracting (including data from PtuResults & GpsResults & AdditionalSensors)
    • bug fixed: wrong decoding of XDATA instrument type (use HEX decoding now)
  • 0.4.4 (2016-01-13)
    • include native netcdf libraries
      • for os-arch: linux-x86, linux-x86-64, win32-x86, win32-x86-64
      • automatically extracting of relevant native libraries to sub-dir `lib' (in case of bin to '../lib')
    • optimise netcdf logging (bind to log4j)

  • 0.4.3 (2016-01-06)
    • add argument --itemize / -z
    • add argument --identify / -y
    • add argument --no-msg-header
    • add argument --no-silent (for -t, -z, -k, -y)
    • add argument --json-result
    • improved automatically silent mode for -t, -z, -k, -y
    • restructured code of processing routines

  • 0.4.2 (2015-12-16)
    • add argument --corrupt-dir=<DIR>
    • add argument --copy-corrupt
    • add argument --no-msg-error
    • add argument --no-msg-info

  • 0.4.1_01 (2015-12-15)
    • improve source documentation
    • bug fixed: remove tmp file in case of an error
    • add final info about corrupt files
  • 0.4.1 (2015-12-14)
    • add argument --skip-convert
    • add argument -i <KEY> / --info=<KEY> with a couple of possible keys:
      • VERSION, VERSION_NUMBER, VERSION_DATE,
      • NAME, NAME_SHORT, NAME_LONG,
      • COPYRIGHT,
      • AUTHOR, AUTHOR_NAME, AUTHOR_EMAIL,
      • FILE_TYPES
    • new error message in case of missing required option
    • add argument -w <DIR> / --work-dir=<DIR>
    • add argument --no-file-type-check

  • 0.4.0 (2015-12-11)
    • add argument --separate-dir
    • improve handling with DC3DB files (e.g. wrong SN, additional log info, ...)
    • use working directory to create NC files

References