Differences between revisions 1 and 44 (spanning 43 versions)
Revision 1 as of 2009-04-07 07:45:15
Size: 766
Editor: ?AlastairMcKinstry
Comment: Debian Meteorology page
Revision 44 as of 2014-06-25 14:37:37
Size: 8652
Editor: ?AlastairCMcKinstry
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
The [[http://cdd.alioth.debian.org/science/tasks/meteorology.html|Sentinel page]] gathers meteorology packages. The [[http://blends.alioth.debian.org/science/tasks/meteorology|Sentinel page]] gathers meteorology packages.
Line 8: Line 9:
 * [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=519184|cdo]] Climate Data Operators
 * [[http://packages.debian.org/sid/cdo|cdo]] ([[https://code.zmaw.de/projects/cdo | upstream]]) Climate Data Operators
 * [[http://packages.debian.org/sid/nco|nco]] command-line operators to analyze netCDF files
 * [[http://www.arpa.emr.it/dettaglio_documento.asp?id=1172&idlivello=64|arkimet]] Arkimet is a set of tools to archive, dispatch and distribute data files.
 * [[http://packages.qa.debian.org/d/dballe.html|DB-All.e]] ([[http://www.arpa.emr.it/dettaglio_documento.asp?id=514&idlivello=64|upstream]]) Database for working with punctual meteorological data. It can also be used only as a BUFR encoder and decoder library.
 * OASIS ([[http://www.prism.enes.org/PAEs/coupling_IO/software_OASIS3.php |upstream]]) climate model Coupler
 * GeoNetwork (DebianBug:529453) ([[http://geonetwork-opensource.org/|upstream]]) Data publication / search network.
 * [[http://www2-pcmdi.llnl.gov/cdat| cdat]] : Climate Data Analysis Tools
 * [[http://www2-pcmdi.llnl.gov/cmor| cmor]] : Climate Model Output Rewriter - rewrite output in a format suitable for the IPCC experiments.
 * [[http://ferret.wrc.noaa.gov/Ferret | Ferret ]] : Data visualization and analysis from NOAA. Note: There is a different [[http://packages.debian.org/lenny/ferret | GNU Ferret ]] in Debian already.
Line 11: Line 21:
 * [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=519184|gribapi]] GRIB API from ECMWF
 * [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=522775|emoslib]] Interpolation library from ECMWF
 * [[http://packages.debian.org/sid/libgrib-api-tools|grib_api]] ([[http://www.ecmwf.int/products/data/software/grib_api.html|upstream]]) GRIB API from ECMWF
 * [[http://packages.debian.org/source/sid/emoslib|emoslib]] ([[http://www.ecmwf.int/products/data/software/interpolation.html|upstream]]) Interpolation library from ECMWF
 * [[http://hdfeos.net/software.php | HDF-EOS]] : extension of HDF for Earth Observation Systems: The library supports the construction of new data structures: Grid, Point and Swath
 * [[http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/ | g2clib]] : GRIB2 library from NCEP. Present in fedora, needed by NCL.
 * ([[http://www.unidata.ucar.edu/software/udunits/ | upstream]]) library for handling physical units. used by NCL, etc.
 * [[http://packages.qa.debian.org/libt/libterralib.html | libterralib]] used by Magics++, etc.
 * [[http://packages.qa.debian.org/libd/libdap.html | OpenDAP]] ([[http://www.opendap.org/|upstream]]) : Network Data Access Protocol
 * [[http://packages.qa.debian.org/libg/libgctp.html | gctp ]] ([[http://gcmd.nasa.gov/records/USGS-GCTP.html | upstream]]) General Cartographic Transformation Package. Used by Met packages.
 * [[http://packages.qa.debian.org/libt/libterralib.html | terralib ]] ([[http://www.terralib.org/|upstream]]) GIS classes and functions library
 * [[http://www.cisl.ucar.edu/css/software/spherepack/ | Spherepack]]. Now released under BSD license.
 * [[http://liblas.org/ | libLAS]] ([[http://bugs.debian.org/432145 | ITP]]) : ASPRS LiDAR data translation toolset
 *[[http://packages.qa.debian.org/m/magics%2B%2B.html| magics++]] ([[http://www.ecmwf.int/products/data/software/magics++.html | upstream]]) meteorological plotting software from ECMWF

=== Analysis and Visualization ===
 * [[http://packages.debian.org/sid/gmt|gmt]] ([[http://gmt.soest.hawaii.edu/|upstream]]) Generic Mapping Tools.
 * [[http://bugs.debian.org/395573 | VisIT]] ([[https://wci.llnl.gov/codes/visit/ | upstream]]) visualisation suite
 * [[http://www.zygrib.org | zyGrib]] GRIB file viewer
 * NCL ([[http://www.ncl.ucar.edu/overview.shtml | upstream]]) NCAR Command language
 * ncview (DebianBug:536882) ([[http://meteora.ucsd.edu/~pierce/ncview_home_page.htm | upstream]]) netCDF file viewer
 * [[http://www.iges.org/grads/ | GrADS ]] Grid Analyis and Display System
 * [[http://lug.rose-hulman.edu/code/projects/aweather/wiki | aweather]] ( [[http://bugs.debian.org/604189 | ITP ]]) ( [[http://git.debian.org/?p=debian-science/packages/aweather.git | debian-science git repo]] ) A weather monitoring program (See [[http://www.mail-archive.com/debian-science@lists.debian.org/msg04196.html | this]])
 * [[http://trmm-fc.gsfc.nasa.gov/trmm_gv/software/rsl/ | rsl ]] ([[http://bugs.debian.org/604185 | ITP]]) ([[http://git.debian.org/?p=debian-science/packages/rsl.git | debian-science git repo]]) TRMM Radar Software Library
Line 15: Line 45:
 * [[http://www.prism.enes.org/PAEs/coupling_IO/software_OASIS4.php |oasis]] OASIS Coupler
 * [[https://wci.llnl.gov/codes/visit/ | VisIT]] visualisation suite
   * [[http://www.esmf.ucar.edu/ | ESMF ]] NCAR Earth System Modelling Framework, a coupler for Climate model. Similar to OASIS, used mainly by American models.
* [[http://www.mcs.anl.gov/research/projects/mct/ | MCT ]] Model Coupling Toolkit. Used by ESMF and now OASIS3-MCT
 * [[http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html | wgrib ]] GRIB1 file manipulation. Present in Fedora / Redhat / SuSE.
 * [[http://met.no/diana | DIANA ]] Open-source meteorological workstation from met.no
 * [[http://www.cfa.harvard .edu/hitran | HITRAN ]] : Database of Spectroscopic info for radiative transfers
 * [[ http://geosci.uchicago.edu/~rtp1/ PrinciplesPlanetaryClimate | Python interface to HITRAN]]
 * [[https://github.com/scitools/iris | IRIS package from the UK Met Office]]
 * [[http://bugs.debian.org/612897 | wreport - ibrary for working with weather reports]]
 * [[https://forge.ipsl.jussieu.fr/prodiguer/wiki/docs/synchro-data | synchro-data ]] For fetching CMIP5 data
== Issues: ==

=== Earth System Grid ===

Basically, package all the code used in ESG nodes. Its all FLOSS code, and needs proper management and tools.
To start with, this will need CDAT, Thredds and their dependencies.

=== CMOR tables ===

Currently there is a package '''cmip5-cmor-tables'''. There are a bunch more tables;
see upstream [[[http://www2-pcmdi.llnl.gov/cmor/tables/ | here]]. Replace the '''cmip5-cmor-tables''' package
with a '''cmor-tables''' package? I've created an [[https://alioth.debian.org/projects/pkg-cmor-tables/ | Alioth packaging project]] for this.

=== SZIP Compression ===

[[http://www.hdfgroup.org/doc_resource/SZIP/ | SZIP]] is a compression library for HDF and other scientific data. HDF which we ship, can be built against it, but SZIP is patented (US only?)
and is free for non-commercial use (ie. not DFSG-free). It is unlikely to change and we could not ship it (can it be shipped in
non-free ?). However it would be desirable for HDF on Debian to be able to use it if present: hence a patch could be written to dynamically open
it if present.
The patent is owned by NASA who have agreed not to enforce it; it expires in a year or so. A GNU GPL'd version of SZIP is being written so that SZIP can be included in WMO standards.

=== Python and NetCDF ===

See [[http://lists.debian.org/debian-science/2012/04/msg00056.html | this discussion]]. There are multiple packages
in Debian/ the python archives providing NetCDF interfaces; some pass netcdf-type objects around. This is
sub-optimal as they don't interoperate. This should be fixed.

Python packages / APIs include:

* pyCDF (circa 2007):
* Scientific.IO.NetCDF : from python-netcdf or pupynere
* scipy.io.netcdf : from scipy, or python-netcdf4

python-netcdf is built against recent netcdf4, and so reads/writes netcdf4 files, but does not provide
NC4 capabilities such as compression, etc. Similarly python-scientific, python-netcdf do not provide
python3 versions. Is python-scientific obsolete? Transition towards scipy ?


=== Python3 Transition ===

As part of the general transition in Debian/Ubuntu to Python3, it is my release goal for Jessie to get all the met. packages working with Python3.

See DebianScienceMetPython3 to see progress.


=== pkg-config pc files for all libraries ===

As a release goal, provide pkg-config files for all libraries provided by DebMet, and hence
dependencies. This is most useful when thinking of MPI.
See DebianSciencePkgConfig for details.

=== Spherepack ===

spherepack is packaged, but NCL uses a variant with double-precision parameters, preceded by 'd' prefix.
Implement this within spherepack, and apply as an external dependency.

=== Common GRIB tables ===

grib-api and ncl have Grib tables that could be factored out into a separate package.

=== GSHHS : Global Self-consistent Hierarchical High-resolution Shorelines ===

Several packages include their own copies of GSHHS maps and code. These should be merged, and one set of maps
provided. DebianScienceGshhsMaps describes the work in progress.

Meteorology in Debian Science

The Sentinel page gathers meteorology packages.

Meteorology tools in Debian / currently being packaged

Data Manipulation

  • cdo (upstream) Climate Data Operators

  • nco command-line operators to analyze netCDF files

  • arkimet Arkimet is a set of tools to archive, dispatch and distribute data files.

  • DB-All.e (upstream) Database for working with punctual meteorological data. It can also be used only as a BUFR encoder and decoder library.

  • OASIS (upstream) climate model Coupler

  • ?GeoNetwork (529453) (upstream) Data publication / search network.

  • cdat : Climate Data Analysis Tools

  • cmor : Climate Model Output Rewriter - rewrite output in a format suitable for the IPCC experiments.

  • Ferret : Data visualization and analysis from NOAA. Note: There is a different GNU Ferret in Debian already.

Libraries

  • grib_api (upstream) GRIB API from ECMWF

  • emoslib (upstream) Interpolation library from ECMWF

  • HDF-EOS : extension of HDF for Earth Observation Systems: The library supports the construction of new data structures: Grid, Point and Swath

  • g2clib : GRIB2 library from NCEP. Present in fedora, needed by NCL.

  • (upstream) library for handling physical units. used by NCL, etc.

  • libterralib used by Magics++, etc.

  • OpenDAP (upstream) : Network Data Access Protocol

  • gctp (upstream) General Cartographic Transformation Package. Used by Met packages.

  • terralib (upstream) GIS classes and functions library

  • Spherepack. Now released under BSD license.

  • libLAS (ITP) : ASPRS LiDAR data translation toolset

  • magics++ (upstream) meteorological plotting software from ECMWF

Analysis and Visualization

Software that could be included in Debian

  • ESMF NCAR Earth System Modelling Framework, a coupler for Climate model. Similar to OASIS, used mainly by American models.

* MCT Model Coupling Toolkit. Used by ESMF and now OASIS3-MCT

Issues:

Earth System Grid

Basically, package all the code used in ESG nodes. Its all FLOSS code, and needs proper management and tools. To start with, this will need CDAT, Thredds and their dependencies.

CMOR tables

Currently there is a package cmip5-cmor-tables. There are a bunch more tables; see upstream ?here. Replace the cmip5-cmor-tables package with a cmor-tables package? I've created an Alioth packaging project for this.

SZIP Compression

SZIP is a compression library for HDF and other scientific data. HDF which we ship, can be built against it, but SZIP is patented (US only?) and is free for non-commercial use (ie. not DFSG-free). It is unlikely to change and we could not ship it (can it be shipped in non-free ?). However it would be desirable for HDF on Debian to be able to use it if present: hence a patch could be written to dynamically open it if present. The patent is owned by NASA who have agreed not to enforce it; it expires in a year or so. A GNU GPL'd version of SZIP is being written so that SZIP can be included in WMO standards.

Python and NetCDF

See this discussion. There are multiple packages in Debian/ the python archives providing NetCDF interfaces; some pass netcdf-type objects around. This is sub-optimal as they don't interoperate. This should be fixed.

Python packages / APIs include:

* pyCDF (circa 2007): * Scientific.IO.NetCDF : from python-netcdf or pupynere * scipy.io.netcdf : from scipy, or python-netcdf4

python-netcdf is built against recent netcdf4, and so reads/writes netcdf4 files, but does not provide NC4 capabilities such as compression, etc. Similarly python-scientific, python-netcdf do not provide python3 versions. Is python-scientific obsolete? Transition towards scipy ?

Python3 Transition

As part of the general transition in Debian/Ubuntu to Python3, it is my release goal for Jessie to get all the met. packages working with Python3.

See DebianScienceMetPython3 to see progress.

pkg-config pc files for all libraries

As a release goal, provide pkg-config files for all libraries provided by ?DebMet, and hence dependencies. This is most useful when thinking of MPI. See DebianSciencePkgConfig for details.

Spherepack

spherepack is packaged, but NCL uses a variant with double-precision parameters, preceded by 'd' prefix. Implement this within spherepack, and apply as an external dependency.

Common GRIB tables

grib-api and ncl have Grib tables that could be factored out into a separate package.

GSHHS : Global Self-consistent Hierarchical High-resolution Shorelines

Several packages include their own copies of GSHHS maps and code. These should be merged, and one set of maps provided. DebianScienceGshhsMaps describes the work in progress.