FileList¶
Module to handle sources distributed over a list of files.
File lists ar specified using a TagExpression
.
-
class
Source
(expression=None, file_list=None, axes_order=None, shape=None, dtype=None, order=None, name=None)[source]¶ Bases:
ClearMap.IO.Source.VirtualSource
File list source.
Note
The full shape of the file list source is the shape of the expression and the shape of the data in each file, i.e. shape = file_list_shape + array_shape.
-
as_virtual
()[source]¶ Return virtual source without array data to pass in parallel processing.
Returns
- sourceSource class
The source class without array data.
-
tag_to_axes_order
()[source]¶ Map from the tag list from the file expression and the axes of this source.
-
property
array
¶
-
property
axes_file
¶ Source axes that constitute individual file dimensions in the full array.
-
property
axes_list
¶ Source axes that constitute the dimensions of the file list in the full array.
-
property
axes_order
¶ Optional ordering of the tag names.
Returns
- axes_orderlist of str
The ordered axis names.
-
property
dtype
¶ The data type of the source.
Returns
- dtypedtype
The data type of the source.
-
property
element_strides
¶
-
property
expression
¶ The underlying expression of this file list.
Returns
- expressionstr
The underlying expression of this source.
-
property
file_list
¶ The underlying file list.
Returns
- filelistlist
The underlying sources of this source.
-
property
name
¶ The name of this source.
Returns
- namestr
Name of this source.
-
property
ndim_file
¶ Source dimension of the individual files.
-
property
ndim_list
¶ Source dimensions of the file list.
-
property
shape
¶ The shape of the source.
Returns
- shapetuple
The shape of the source.
-
property
shape_file
¶ Source shape of the individual files.
-
property
shape_list
¶ Source shape of the file list.
-
-
class
VirtualSource
(expression=None, file_list=None, shape=None, dtype=None, order=None, axes_order=None, source=None, name=None)[source]¶ Bases:
ClearMap.IO.Source.VirtualSource
Virtual file list source.
-
as_virtual
()[source]¶ Return virtual source without array data to pass in parallel processing.
Returns
- sourceSource class
The source class without array data.
-
property
array
¶
-
property
axes_order
¶
-
property
dtype
¶ The data type of the source.
Returns
- dtypedtype
The data type of the source.
-
property
element_strides
¶
-
property
expression
¶ The underlying expression of this file list.
Returns
- expressionstr
The underlying expression of this source.
-
property
file_list
¶
-
property
name
¶ The name of this source.
Returns
- namestr
Name of this source.
-
property
shape
¶ The shape of the source.
Returns
- shapetuple
The shape of the source.
-
-
convert
(source, sink, processes=None, verbose=False)[source]¶ Converts list of files to a sink in parallel
Arguments
- sourceSource
File list source.
- sinkSource
A sink to write the source to.
Returns
- sinkSource
The sink the data was converted to.
-
dtype
(expression=None, file_list=None)[source]¶ Returns data type of the array stored in a file list.
Arguments
- expressionstr
The regular epression for the file list.
Returns
- dtypedtype
The data type of the file list.
-
is_file_list
(expression, exists=False, tag_names=None, n_tags=-1, verbose=False)[source]¶ Checks if the expression is a valid file list.
Arguments
- expressionstr
The regular expression to check.
- existsbool
If True, check if at least one file exists.
- tag_nameslist of str or None
List of tag names expected to be present in the expression.
- n_Tagsint or None
Number of tags to expect.
- verbosebool
If True, print reason why the epxression does not represent the desired file list.
Returns
- is_expressionbool
Returns True if the expression fullfills the desired criteria and at least one file matching the expression exists.
-
ndim
(expression=None, file_list=None)[source]¶ Calculates the dimension of the file list given by an expression.
Arguments
- expressionstr
The expression for the file_list.
- file_listlist
Optional file_list to speed up calculation.
Returns
- ndimint
The dimension of the file list given by the expression.
-
order
(expression=None, file_list=None)[source]¶ Returns order of the array stored in a file list.
Arguments
- expressionstr
The regular epression for the file list.
Returns
- dtypedtype
The data type of the file list.
-
shape
(expression=None, file_list=None, axes_order=None, axis=None)[source]¶ Calculates the shape of the data in a file list.
Arguments
- expressionstr or None
The regular epression for the file list.
- file_listlist or None
List of files.
- axisint or None
The shape along a specific axis. Can speed up the shape calculation.
- axes_orderlist or None
The names of how to order the different tag names in the expression. If None, use ordering of the tags in the expression.
Returns
- shapeint or tuple of ints
The shape of the array st ored in a file list.
-
shape_file
(expression=None, file_list=None)[source]¶ Calculates the shape of the data in a file list.
Arguments
- expressionstr or None
The regular epression for the file list.
- file_listlist or None
List of files.
Returns
- shapeint or tuple of ints
The shape of the array st ored in a file list.
-
shape_list
(expression=None, file_list=None, axes_order=None)[source]¶ Calculates the shape of the data in a file list.
Arguments
- expressionstr or None
The regular epression for the file list.
- file_listlist or None
List of files.
- axes_orderlist or None
The names of how to order the different tag names in the expression. If None, use ordering of the tags in the expression.
Returns
- shapeint or tuple of ints
The shape of the array along the dimensions created by the file list.