Differences between revisions 39 and 40
Revision 39 as of 2013-09-11 08:50:13
Size: 7465
Editor: ?AlastairCMcKinstry
Comment: Add synchro-data
Revision 40 as of 2013-09-24 11:14:25
Size: 8304
Editor: ?AlastairCMcKinstry
Comment:
Deletions are marked like this. Additions are marked like this.
Line 57: Line 57:
==== SZIP Compression ==== === SZIP Compression ===
Line 65: Line 65:
==== NCL ==== === NCL ===
Line 73: Line 73:
==== GRIB2 ====
EUMETNET and member consortia have agreed that GRIB2 will be the interchange format between their models. Ensure that tools within Debian Met. work with GRIB2.
=== Python and NetCDF ===
Line 76: Line 75:
__Done.___ See [[http://lists.debian.org/debian-science/2012/04/msg00056.html | this discussion]]. There are multiple packages
in Debian/ the python archives providing NetCDF interfaces; some pass netcdf-type objects around. This is
sub-optimal as they don't interoperate. This should be fixed.
Line 78: Line 79:
==== Python3 Transition ==== Python packages / APIs include:

* pyCDF (circa 2007):
* Scientific.IO.NetCDF : from python-netcdf or pupynere
* scipy.io.netcdf : from scipy, or python-netcdf4

python-netcdf is built against recent netcdf4, and so reads/writes netcdf4 files, but does not provide
NC4 capabilities such as compression, etc. Similarly python-scientific, python-netcdf do not provide
python3 versions. Is python-scientific obsolete? Transition towards scipy ?


=== Python3 Transition ===
Line 84: Line 96:
==== GSHHS : Global Self-consistent Hierarchical High-resolution Shorelines ====
=== pkg-config pc files for all libraries ===

As a release goal, provide pkg-config files for all libraries provided by DebMet, and hence
dependencies. This is most useful when thinking of MPI. See DebianScienceMPIPkgCongig for details.

=== GSHHS : Global Self-consistent Hierarchical High-resolution Shorelines ===

Meteorology in Debian Science

The Sentinel page gathers meteorology packages.

Meteorology tools in Debian / currently being packaged

Data Manipulation

  • cdo (upstream) Climate Data Operators

  • nco command-line operators to analyze netCDF files

  • arkimet Arkimet is a set of tools to archive, dispatch and distribute data files.

  • DB-All.e (upstream) Database for working with punctual meteorological data. It can also be used only as a BUFR encoder and decoder library.

  • OASIS (upstream) climate model Coupler

  • ?GeoNetwork (529453) (upstream) Data publication / search network.

  • cdat : Climate Data Analysis Tools

  • cmor : Climate Model Output Rewriter - rewrite output in a format suitable for the IPCC experiments.

  • Ferret : Data visualization and analysis from NOAA. Note: There is a different GNU Ferret in Debian already.

Libraries

  • grib_api (upstream) GRIB API from ECMWF

  • emoslib (upstream) Interpolation library from ECMWF

  • HDF-EOS : extension of HDF for Earth Observation Systems: The library supports the construction of new data structures: Grid, Point and Swath

  • g2clib : GRIB2 library from NCEP. Present in fedora, needed by NCL.

  • (upstream) library for handling physical units. used by NCL, etc.

  • libterralib used by Magics++, etc.

  • OpenDAP (upstream) : Network Data Access Protocol

  • gctp (upstream) General Cartographic Transformation Package. Used by Met packages.

  • terralib (upstream) GIS classes and functions library

  • Spherepack. Now released under BSD license.

  • libLAS (ITP) : ASPRS LiDAR data translation toolset

  • magics++ (upstream) meteorological plotting software from ECMWF

Analysis and Visualization

Software that could be included in Debian

  • ESMF NCAR Earth System Modelling Framework, a coupler for Climate model. Similar to OASIS, used mainly by American models.

* MCT Model Coupling Toolkit. Used by ESMF and now OASIS3-MCT

Issues:

SZIP Compression

SZIP is a compression library for HDF and other scientific data. HDF which we ship, can be built against it, but SZIP is patented (US only?) and is free for non-commercial use (ie. not DFSG-free). It is unlikely to change and we could not ship it (can it be shipped in non-free ?). However it would be desirable for HDF on Debian to be able to use it if present: hence a patch could be written to dynamically open it if present. The patent is owned by NASA who have agreed not to enforce it; it expires in a year or so. A GNU GPL'd version of SZIP is being written so that SZIP can be included in WMO standards.

NCL

NCL Depends on a large number of packages, many in Debian. The required ones that could cause issues include:

  • Triangle - non-free license. Contact author ?

  • Vis5d - Has a plugin that is non-free outside NCL, but GTK replacement in progress? There has been no progress in recent years.

  • 2013-06-15 : Appears to be moving again. Will retry packaging.
  • 2013-07-04: packaging underway. No longer needs Triangle or Vis5d.

Python and NetCDF

See this discussion. There are multiple packages in Debian/ the python archives providing NetCDF interfaces; some pass netcdf-type objects around. This is sub-optimal as they don't interoperate. This should be fixed.

Python packages / APIs include:

* pyCDF (circa 2007): * Scientific.IO.NetCDF : from python-netcdf or pupynere * scipy.io.netcdf : from scipy, or python-netcdf4

python-netcdf is built against recent netcdf4, and so reads/writes netcdf4 files, but does not provide NC4 capabilities such as compression, etc. Similarly python-scientific, python-netcdf do not provide python3 versions. Is python-scientific obsolete? Transition towards scipy ?

Python3 Transition

As part of the general transition in Debian/Ubuntu to Python3, it is my release goal for Jessie to get all the met. packages working with Python3.

See DebianScienceMetPython3 to see progress.

pkg-config pc files for all libraries

As a release goal, provide pkg-config files for all libraries provided by ?DebMet, and hence dependencies. This is most useful when thinking of MPI. See DebianScienceMPIPkgCongig for details.

GSHHS : Global Self-consistent Hierarchical High-resolution Shorelines

Several packages include their own copies of GSHHS maps and code. These should be merged, and one set of maps provided. DebianScienceGshhsMaps describes the work in progress.