4.3. A collection of utility functions: flink.utils

4.3.1. flink.utils.pool — multiprocessing in FLinK activities

The module provides multiprocessing facilities for flink modules.

class workers_pool(ncpu)

A simple wrapper around multiprocessing.Pool.

ncpu is the maximum number of workers that can run at the same time.

The pool is automatically bypassed if ncpu is 1.

submit_job(func, kwargs)

Submit a new job to the pool.

func is the function that the worker will run, and kwargs is dictonary of keyword arguments for func.

get_results()

Return the list of results produced by the worker processes, in the order they were submitted to the pool.

get_elapsed_time()

Return CPU time (user + system) required to run all the jobs in the pool

4.3.2. flink.utils.logging — a very simple logging facility

Display formatted log messages.

Messages are actually printed or not depending on the value of the configuration property flink.config.logging_verbosity.

debug_msg(text)

Print a debug message.

info_msg(text)

Print an informative message.

result_msg(text)

Print some result or process output.

warning_msg(text)

Print a worning message.

error_msg(text, critical=False)

Print an error message. If critical is True, a RuntimeError is raised.

4.3.3. flink.utils.props — handle configuration properties

API to access/modify the value of configiration properties defined in the configuration file flink.config.

get_defined_properties()

Return a dictionary of the configuration properties defined in flink.config.

set_property(name, value)

Set the value of a configuration property.

The property must be already defined in the global configuration module.

In case of success, returns a tuple (old,new) with the old and the new value of the property.

Raises NoSuchProperty if the required property is not set, IncompatiblePropertyValue if the value provided does not match the type of the property to be overwritten.

get_property(name)

Return the value of the configuration property name.

Raise NoSuchProperty if a configuration property of the given name does not exist.

exception NoSuchProperty(name)

Raised when the undefined property name is accessed.

exception IncompatiblePropertyValue(prop, value)

Raised when trying to update the value prop of some property with the incompatible value value.

4.3.4. flink.utils.mathutils — mathematical functions

class dense_vector(initial={})

Dict-like dense representation for vectors.

Dictionary entries are in the form int: float and are meant to represent components and their values. A missing key is regarded as a component whose value is 0.

initial is a dictionary that can be used to initialize the value of some components of the vector.

static parse(vectstring)

Return a new dense vector by parsing the string vecstring, which is supposed to be in the form:

component_id:value [component_id:value ...]

multiply(val)

Multiply all the components of this vector by the value val.

sum(*dense_vectors)

Sum to this vector all the vectors in dense_vectors.

norm()

Calculate the norm of this dense vector.

4.3.5. flink.utils.fileutils — filesystem related functions

cat(filename, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Copy to the file descriptor stream (that must be open for writing) the contents of filename.

copy_file(src, dest)

Copy src to dest after making sure that the destination directory exists.

count_lines(inputfile)

Return the number of lines in inputfile.

join_files(splitfiles, outfile)

Concatenate all the files in splitfiles into the output file outfile.

make_folds(infile, num_folds, outdir)

Create a cross-fold validation set from the datafile infile.

The file is split into num_folds splits. For each split, assumed as test set, a corresponding training set is generated by concatenating all the other num_folds-1 folds.

Return a dictionary of training: test files for each fold.

makedirs(outdir)

Create the directory outdir and all the necessary intermediate directories.

move_file(src, dest)

Move src to dest after making sure that the destination directory exists.

open_for_writing(filename, mode='w')

Return a file descriptor for filename in mode mode after making sure that the destination directory has been created.

rmdir(dir)

Remove directory dir.

split_file(infile, num_splits, outdir, balanced=False)

Partition infile into num_splits partitions.

By default, the file is split sequentially, i.e. if num_splits equals 2, then the first half of the file is saved to a split and the second half to another split. If balanced is True, instead, the file is splitted so as to mantain the positive/negative example ratio in each split.

The partitions are created in outdir. Each partition has the same basename as infile, to which the extension .split_XXX_YYY is added. XXX is the zero padded identifier of the split, in the range [1-num_splits]. YYY is the zero padded num_splits.

Return the sequence of split file paths.

4.3.6. flink.utils.persistency — a facility for object persistency

A simple wrapper around pickle for object persistency.

save(object, filename)

Save object object to the path identified by filename. Create the necessary directory hierarchy in the process.

load(filename)

Unpickle the object saved at filename and return it.

4.3.7. flink.utils.tree — manage parenthesized trees

This module provides facilities for handling trees.

It allows to parse strings representing trees in parenthesized notation and to manipulate the resulting data structures.

class Node

A class to handle rooted ordered trees.

Each node in a tree is a tree itself.

label is the label of the node, and parent a reference to the parent node. If parent is None, then the node is the root of a tree.

Public fields:

• label: the label of the node
• parent: reference to the parent node (None if the node is the root of the tree)
• children: an ordered sequence of the children of this node. It is empty if the node is a leaf
• child_offset: if the node is not the root of a tree, it is the relative offset of the node with respect to the parent node (i.e. 0 if it is the first child, 1 if it is the second, and so on).

4.3.8. flink.utils.cli — command line parsing facilities

A wrapper around optparse.OptionParser to easily build command line parsers.

The module provides a higher level interface for command line parsing, implementing mandatory options and making easier to create groups of command line options.

The user invokes the cliparse() function, passing as arguments a variable number of cliopt objects in the order in which they should appear in the help message.

On success (all the required arguments and options were present) the function returns the same (options, args) pair as optparse:OptionParser.

A set of high-level factory functions (str_option(), int_option(), float_option(), flag(), list_option(), config_properties_option()) are used to ease the creation of typed options.

A special class, group, is used to easily create option groups. All the options added after creating a group are automatically included in that group. When a new group is added, all subsequent options will be added to the last group.

As an example, the following instructions:

from flink.utils.cli import cliparse, int_option, \
   float_option, group, str_option

options,args = cliparse(
   
   group("First group"),
   
   int_option("a", "option_a", 
      help = "This option requires an integer argument", 
      default = 1),
      
   float_option("b", "option_b",
      help = "This non-mandatory option requires a float argument",
      mandatory = False),
   
   group("Second group"),
   
   str_option("c", "option_c", 
      help = "A mandatory string option for my parser"),
      
   cli_num_args = 2, # there must be exactly 2 positional arguments
   usage = "%prog [options] <1st required arg> <2nd required arg>")

would produce the following help message:

Usage: test.py [options] <1st required arg> <2nd required arg>

Options:
  -h, --help            show this help message and exit

  First group:
    -a OPTION_A, --option_a=OPTION_A
                        This option requires an integer argument [default:1]
    -b OPTION_B, --option_b=OPTION_B
                        This non-mandatory option requires a float argument

  Second group:
    -c OPTION_C, --option_c=OPTION_C
                        A mandatory string option for my parser  (*)

Options marked with (*) are mandatory.

See also

The command line parsers in flink.linearize.ksl and flink.linearize.binary for richer examples.

cliparse(*options, cli_num_args = None, *optparser_options)

Parse command line arguments.

options is a variable number of positional arguments, each of them being either an instance of cliopt or an instance of group.

Every instance of cliopt is added to the last added group. This behavior makes extremely easy to design the command line in a visual way, as exemplified above.

After the sequence of command line options, the programmer can also specify the °cli_num_args* keyword argument to set the number of positional arguments that the user is supposed to provide when running the program.

cli_num_args can be either an integer or a string representing and integer value. In this case, the command line parser will require the user to provide exactly the given number of positional arguments when invoking the program.

If cli_num_args is a string in the form <num>+ (<num> being a valid integer number), then the parser will require the user to provide at least <num> positional arguments.

Finally, if cli_num_args is a string in the form <num>- (<num> being a valid integer number), then the parser will require the user to provide at most <num> positional arguments.

optparser_options is a dictionary of keyword arguments that are forwarded to the constructor of optparse.OptionParser.

If the arguments are successfully parsed (after applying all the checks for mandatory options and required arguments), the function returns the return value of optparse.OptionParser.parse_args(), i.e. a pair (options, args) in which options is an object with the values of command line options and args is the list of user provided positional arguments.

class cliopt(short, dest, mandatory=True, group=None, **option_args)

A wrapper class around optparse.Option to build command line options.

• short is a one-character string to be used as short flag for the option.

  Note

  Unlike optparse.OptionParser, the trailing - must not be included!

• dest is the name of the attribute where the option value will be stored to, and is also used for the long name of the command line flag.

• If mandatory is True, then the user of the program will be required to provide a value for the option, unless a default is provided by the programmer (via option_args["default"]).

• If group is not None, then the option is added to the group titled group.

• option_args is a set of keyword arguments that will be forwarded to the constructor of optparse.Option.

Note

Users should not use cliopt directly, instead they should use one of the higher level facilities like str_option(), int_option(), depending on the kind of option they want to create.

config_properties_option(short='D', dest='config_property', **kwargs)

Add an option to override the values of configuration properties.

See also

flink.config and flink.utils.props.

The arguments short and dest are the same as for cliopt.

Adding this option to the command line parser, allows the user of a program to override the values of one or more configuration properties by using a syntax like:

-D first_prop_name=value1 -D second_prop_name=value2

(assuming that D was used as short flag for the option).

When parsing the command line, the parser will also check the consistency of the user’s assignments, verifying that the required properties exist and that the values they are given are compatible with each property’s type.

list_option(short, dest, **kwargs)

Create an option for which users can specify multiple values, using a syntax like:

-x first_value -x second_value

assuming that short is set to x.

The arguments short and dest are the same as for cliopt.

int_option(short, dest, **kwargs)

Creating a command line option to store an integer parameter.

The arguments short and dest are the same as for cliopt.

float_option(short, dest, **kwargs)

Create a command line option to store a floating point parameter.

The arguments short and dest are the same as for cliopt.

str_option(short, dest, **kwargs)

Create a command line option to store a string valued parameter.

The arguments short and dest are the same as for cliopt.

flag(short, dest, **kwargs)

Create a boolean flag.

By default, the value of the flag is set to False. If the user adds the flag to the command line, then its value is set to True.

The arguments short and dest are the same as for cliopt.

class group(title)

Create a group of command line options.

title is the title for the group of options.