netcdfio#

netCDF4 functions to a copy netcdf file while doing some transformations on variables and dimensions.

This module was written by Matthias Cuntz while at Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement (INRAE), Nancy, France.

It borrows the idea of get_variable_definition from the netcdf4 thin layer of David Schaefer.

copyright:: Copyright 2020-2022 Matthias Cuntz, see AUTHORS.rst for details.
license:: MIT License, see LICENSE for details.

The following functions are provided

`copy_dimensions`(fi, fo[, removedim, ...])	Create dimensions in output file from dimensions in input file.
`copy_file`(ifile, ofile[, timedim, ...])	Copy variables from input file into output file.
`copy_global_attributes`(fi, fo[, add, remove])	Create global output file attributes from input global file attributes.
`copy_variables`(fi, fo[, time, timedim, ...])	Copy variables from input file into output file.
`create_new_variable`(invardef, fo[, izip, ...])	Create variable in output file from dictionary with variable attributes.
`create_variables`(fi, fo[, time, timedim, ...])	Create variables in output from variables in input file.
`get_fill_value_for_dtype`(dtype)	Get default _FillValue of netCDF4 for the given data type.
`get_variable_definition`(ncvar)	Collect information on input variable.
`set_output_filename`(ifile, ext)	Create output file name from input file name by adding ext before the file suffix.

History

Written Apr 2020 by Matthias Cuntz (mc (at) macu (dot) de)
Added get_fill_value_for_dtype, Mar 2021, Matthias Cuntz
Added create_new_variable, Mar 2021, Matthias Cuntz
flake8 compatible, Mar 2021, Matthias Cuntz
Remove keyword in copy_global_attributes, Mar 2021, Matthias Cuntz
Add timedim in create_variables, Dec 2021, Matthias Cuntz
Rename functions, e.g. create_dimensions -> copy_dimensions, May 2022, Matthias Cuntz
Add copy_variables, May 2022, Matthias Cuntz
Delete unnecessary HDF5 filters in variable definition for compatibility with netcdf4 > 1.6.0, Jun 2022, Matthias Cuntz
Make get_variable_definition public, Apr 2023, Matthias Cuntz
get_variable_definition returns _FillValue and only then missing_value, Jul 2023, Matthias Cuntz
create_variables sets missing_value attribute if present even if used for _FillValue, Jul 2023, Matthias Cuntz
addglobalatt in copy_file, Dec 2023, Matthias Cuntz

copy_dimensions(fi, fo, removedim=[], renamedim={}, changedim={}, adddim={})[source]#

Create dimensions in output file from dimensions in input file.

Parameters:

fi (file_handle) – File handle of opened netcdf input file
fo (file_handle) – File handle of opened netcdf output file
removedim (list of str, optional) – Do not create dimensions given in removedim in output file.
renamedim (dict, optional) – Rename dimensions in output file compared to input file. Dimension names in input file are given as dictionary keys, corresponding dimension names of output file are give as dictionary values.
changedim (dict, optional) – Change the size of the output dimension compared to the input file. Dimension names are given as dictionary keys, corresponding dimension sizes are given as dictionary values.
adddim (dict, optional) – Add dimension to output file. New dimension names are given as dictionary keys and new dimension sizes are given as dictionary values.

Returns:

The output file will have the altered and unaltered dimensions of the input file.

Return type:

nothing

Examples

copy_dimensions(fi, fo, removedim=['patch'],
                renamedim={'x': 'lon', 'y': 'lat'},
                changedim={'mland': 1})

copy_file(ifile, ofile, timedim='time', removevar=[], renamevar={}, replacevar={}, replaceatt={}, addglobalatt={}, noclose=False)[source]#

Copy variables from input file into output file.

Parameters:

ifile (str) – File name of netcdf input file
ofile (str) – File name of netcdf output file
timedim (str, optional) – Name of time dimension in input file (default: ‘time’).
removevar (list of str, optional) – Do not copy variables given in removevar to output file.
renamevar (dict, optional) – Copy variables from input file with different name in output file. Variable names in input file are given as dictionary keys, corresponding variable names of output file are give as dictionary values.
replacevar (dict, optional) – Replace existing variables with variables in dictionary. Variable names in input file are given as dictionary keys, dictionary values are also dictionaries where keys are the output variable name and values are the output variable values.
replaceatt (dict, optional) – Replace or set attributes of variables in dictionary keys (case sensitive). Dictionary values are also dictionaries with {‘attribute_name’: attribute_value}. Dictionary keys are the names of the output variables after renaming and replacing.
addglobalatt (dict, optional) – Create or add to global file attributes. dict values will be given to attributes given in dict keys. Attributes will be created if they do not exist yet.
noclose (bool, optional) – Return file handle of opened output file for further manipulation if True (default: False)

Returns:

The output file will have the altered or unaltered variables copied from the input file.

Return type:

nothing or file_handle

Examples

ovar = np.arange(100)
copy_variable('in.nc', 'out.nc',
              renamevar={'lon': 'longitude'},
              replacevar={'var1': {'arange': ovar}},
              replaceatt={'arange':
                          {'long_name': 'A range', 'unit': '-'}})

copy_global_attributes(fi, fo, add={}, remove=[])[source]#

Create global output file attributes from input global file attributes.

Parameters:

fi (file_handle) – File handle of opened netcdf input file
fo (file_handle) – File handle of opened netcdf output file
add (dict, optional) – dict values will be given to attributes given in dict keys. Attributes will be created if they do not exist yet.
remove (list, optional) – Do not create global attributes given in remove in the output file.

Returns:

Output will have global file attributes

Return type:

nothing

Examples

copy_global_attributes(
    fi, fo, add={'history': time.asctime()+': '+' '.join(sys.argv)})

copy_variables(fi, fo, time=None, timedim='time', removevar=[], renamevar={})[source]#

Copy variables from input file into output file.

Parameters:

fi (file_handle) – File handle of opened netcdf input file
fo (file_handle) – File handle of opened netcdf output file
time (None or bool, optional) – None: copy all variables (default). True: copy only variables having dimension timedim. False: copy only variables that do not have dimension timedim.
timedim (str, optional) – Name of time dimension (default: ‘time’).
removevar (list of str, optional) – Do not copy variables given in removevar to output file.
renamevar (dict, optional) – Copy variables from input file with different name in output file. Variable names in input file are given as dictionary keys, corresponding variable names of output file are give as dictionary values.

Returns:

The output file will have the altered or unaltered variables copied from the input file.

Return type:

nothing

Examples

copy_variable(fi, fo, fill=True, renamevar={'lon': 'longitude'})

create_new_variable(invardef, fo, izip=False, fill=None, chunksizes=True)[source]#

Create variable in output file from dictionary with variable attributes.

Parameters:

invardef (dict) – Dictionary with name and dtype plus further attributes used in netCDF4.Dataset.createVariable; all other entries are set as variable attributes: ‘dimensions’, ‘zlib’, ‘complevel’, ‘shuffle’, ‘fletcher32’, ‘contiguous’, ‘chunksizes’, ‘endian’, ‘least_significant_digit’, ‘fill_value’, ‘chunk_cache’
fo (file_handle) – File handle of opened netcdf output file
izip (bool, optional) – True: the data will be compressed in the netCDF file using gzip compression independent of ‘zlib’ entry in input dictionary invardef (default: False).
fill (float, bool or None, optional) – Determine the behaviour if variable has no _FillValue or missing_value. If None or False: no _FillValue will be set. If True: _FillValue will be set to default value of the Python netCDF4 package for this type. If number: _FillValue will be set to number.
chunksizes (bool, optional) – True: include possible chunksizes in output file (default). False: do not include chunksize information from input file in output file, even if given in input dictionary invardef.

Returns:

Handle to newly created variable in output file.

Return type:

variable handle

Examples

nvar = {'name': 'new_field',
        'dtype': np.dtype(np.float),
        'dimensions': ('time', 'y', 'x'),
        'units': 'kg/m2/s',
        }
ovar = create_new_variable(nvar, fo, fill=True, izip=True)

create_variables(fi, fo, time=None, timedim='time', izip=False, fill=None, chunksizes=True, removevar=[], renamevar={}, removedim=[], renamedim={}, replacedim={})[source]#

Create variables in output from variables in input file.

Parameters:

fi (file_handle) – File handle of opened netcdf input file
fo (file_handle) – File handle of opened netcdf output file
time (None or bool, optional) – None: create all variables (default). True: create only variables having dimension timedim. False: create only variables that do not have dimension timedim.
timedim (str, optional) – Name of time dimension (default: ‘time’).
izip (bool, optional) – True: the data will be compressed in the netCDF file using gzip compression (default: False).
fill (float, bool or None, optional) – Determine the behaviour if variable have no _FillValue or missing_value. If None or False: no _FillValue will be set. If True: _FillValue will be set to default value of the Python package netCDF4 for this type. If number: _FillValue will be set to number.
chunksizes (bool, optional) – True: include possible chunksizes in output file (default). False: do not include chunksize information from input file in output file. Set to False, for example, if dimension size gets changed because the chunksize on a dimension can not be greater than the dimension size.
removevar (list of str, optional) – Do not create variables given in removevar in output file.
renamevar (dict, optional) – Rename variables in output file compared to input file. Variable names in input file are given as dictionary keys, corresponding variable names of output file are give as dictionary values.
removedim (list of str, optional) – Remove dimensions from variable definitions in output file.
renamedim (dict, optional) – Rename dimensions for variables in output file. Dimension names in input file are given as dictionary keys, corresponding dimension names of output file are give as dictionary values.
replacedim (dict, optional) – Replace dimensions for variables in output file. Dimension names in input file are given as dictionary keys, corresponding dimension names of output file are given as dictionary values. The output names can be tuples or lists to extend dimensions of a variable.

Returns:

The output file will have the altered or unaltered variables of the input file defined.

Return type:

nothing

Examples

create_variable(fi, fo, fill=True, izip=True, removedim=['patch'],
                renamevar={'lon': 'longitude'},
                replacedim={'land': ('y', 'x')})

get_fill_value_for_dtype(dtype)[source]#

Get default _FillValue of netCDF4 for the given data type.

Parameters:: dtype (np.dtype) – numpy data type
Return type:: default _FillValue of given numpy data type

Examples

fill_value = get_fill_value_for_dtype(var.dtype)

get_variable_definition(ncvar)[source]#

Collect information on input variable.

Parameters:: ncvar (netcdf4 variable) – Variable of input file
Returns:: Containing information on input variable withkey/value pairs. The following keys are returned: ‘name’, ‘dtype’, ‘dimensions’, ‘fill_vallue’, ‘chunksizes’
Return type:: dict

Examples

get_variable_definition(fi.variables['GPP'])

set_output_filename(ifile, ext)[source]#

Create output file name from input file name by adding ext before the file suffix.

Parameters:

ifile (str) – input file name
ext (str) – string to add before file suffix

Returns:

output filename with ext before file suffix

Return type:

str

Examples

>>> set_output_filename('in.nc', '-no_patch')
in-no_patch.nc
>>> set_output_filename('in.nc', '.nop')
in.nop.nc

netcdfio

Contents

netcdfio#