Internal API¶

Bricks
Extensions
Utils

Bricks¶

class blocks.bricks.base.Application(application_function)¶

Bases: object

An application method belonging to a particular type of brick.

The application methods of each Brick class are automatically replaced by an instance of Application. This allows us to store metadata about particular application methods (such as their in- and outputs) easily.

application¶

callable

The original (unbounded) application function defined on the Brick.

delegate_function¶

callable

A function that takes a Brick instance as an argument and returns a BoundApplication object to which attribute requests should be routed.

properties¶

dict (str, callable)

A dictionary of property getters that should be called when an attribute with the given name is requested.

instances¶

dict (Brick, BoundApplication)

A record of bound application instances created by the descriptor protocol.

call_stack¶

list of Brick

The call stack of brick application methods. Used to check whether the current call was made by a parent brick.

brick¶

type

The brick class to which this instance belongs.

Raises:	`ValueError` – If a brick’s application method is applied by another brick which does not list the former as a child. `ValueError` – If the application method’s inputs and/or outputs don’t match with the function signature or the values returned (respectively).

Notes

When a Brick is instantiated and its application method (i.e. an instance of this class) requested, the descriptor protocol (through the __get__() method) automatically instantiates a BoundApplication class and returns this. This bound application class can be used to store application information particular to a brick instance. Any attributes unknown to the bounded application are automatically routed to the application that instantiated it.

__get__(instance, owner)¶: Instantiate BoundApplication for each Brick.

application_function¶

apply(bound_application, *args, **kwargs)¶

call_stack = []

delegate(f)¶

Decorator to assign a delegate application.

An application method can assign a delegate application. Whenever an attribute is not available, it will be requested from the delegate instead.

Examples

>>> class Foo(Brick):
...     @application(outputs=['baz'])
...     def apply(self, x):
...         return x + 1
...
...     @apply.property('inputs')
...     def apply_inputs(self):
...         return ['foo', 'bar']
>>> class Bar(Brick):
...     def __init__(self, foo):
...         self.foo = foo
...
...     @application(outputs=['foo'])
...     def apply(self, x):
...         return x + 1
...
...     @apply.delegate
...     def apply_delegate(self):
...         return self.foo.apply
>>> foo = Foo()
>>> bar = Bar(foo)
>>> bar.apply.outputs
['foo']
>>> bar.apply.inputs
['foo', 'bar']

inputs¶

name¶

property(name)¶

Decorator to make application properties.

Parameters:	name (str) – The name the property should take.

Examples

>>> class Foo(Brick):
...     @application
...     def apply(self, x):
...         return x + 1
...
...     @apply.property('inputs')
...     def apply_inputs(self):
...         return ['foo', 'bar']
>>> foo = Foo()
>>> foo.apply.inputs
['foo', 'bar']

class blocks.bricks.base.ApplicationCall(application)¶

Bases: blocks.graph.Annotation

A link between the variable tags and bricks.

The application call can be used to attach to an apply call auxiliary variables (e.g. monitors or regularizers) that do not form part of the main computation graph.

The application call object is created before the call to the application method and can be accessed by specifying an application_call argument.

Also see Annotation.

Parameters:	application (`BoundApplication` instance) – The bound application (i.e. belong to a brick instance) object being called

Examples

>>> class Foo(Brick):
...     @application
...     def apply(self, x, application_call):
...         application_call.add_auxiliary_variable(x.mean())
...         return x + 1
>>> x = tensor.vector()
>>> y = Foo().apply(x)
>>> from blocks.filter import get_application_call
>>> get_application_call(y) 
<blocks.bricks.base.ApplicationCall object at ...>

add_auxiliary_variable(variable, roles=None, name=None)¶

class blocks.bricks.base.BoundApplication(application, brick)¶

Bases: object

An application method bound to a Brick instance.

name¶

class blocks.bricks.base.Brick(name=None)¶

Bases: blocks.graph.Annotation

A brick encapsulates Theano operations with parameters.

A brick goes through the following stages:

Construction: The call to __init__() constructs a Brick instance with a name and creates any child bricks as well.
Allocation of parameters:
1. Allocation configuration of children: The push_allocation_config() method configures any children of this block.
2. Allocation: The allocate() method allocates the shared Theano variables required for the parameters. Also allocates parameters for all children.
The following can be done in either order:
1. Application: By applying the brick to a set of Theano variables a part of the computational graph of the final model is constructed.
2. The initialization of parameters:
  1. Initialization configuration of children: The push_initialization_config() method configures any children of this block.
  2. Initialization: This sets the initial values of the parameters by a call to initialize(), which is needed to call the final compiled Theano function. Also initializes all children.

Not all stages need to be called explicitly. Step 3(a) will automatically allocate the parameters if needed. Similarly, step 3(b.2) and 2(b) will automatically perform steps 3(b.1) and 2(a) if needed. They only need to be called separately if greater control is required. The only two methods which always need to be called are an application method to construct the computational graph, and the initialize() method in order to initialize the parameters.

At each different stage, a brick might need a certain set of configuration settings. All of these settings can be passed to the __init__() constructor. However, by default many bricks support lazy initialization. This means that the configuration settings can be set later.

Note

Some arguments to __init__() are always required, even when lazy initialization is enabled. Other arguments must be given before calling allocate(), while others yet only need to be given in order to call initialize(). Always read the documentation of each brick carefully.

Lazy initialization can be turned off by setting Brick.lazy = False. In this case, there is no need to call initialize() manually anymore, but all the configuration must be passed to the __init__() method.

Parameters:	name (str, optional) – The name of this brick. This can be used to filter the application of certain modifications by brick names. By default, the brick receives the name of its class (lowercased).

name¶

str

The name of this brick.

print_shapes¶

bool

False by default. If True it logs the shapes of all the input and output variables, which can be useful for debugging.

parameters¶

list of TensorSharedVariable and None

After calling the allocate() method this attribute will be populated with the shared variables storing this brick’s parameters. Allows for None so that parameters can always be accessed at the same index, even if some parameters are only defined given a particular configuration.

children¶

list of bricks

The children of this brick.

allocated¶

bool

False if allocate() has not been called yet. True otherwise.

initialized¶

bool

False if allocate() has not been called yet. True otherwise.

allocation_config_pushed¶

bool

False if allocate() or push_allocation_config() hasn’t been called yet. True otherwise.

initialization_config_pushed¶

bool

False if initialize() or push_initialization_config() hasn’t been called yet. True otherwise.

Notes

To provide support for lazy initialization, apply the lazy() decorator to the __init__() method.

Brick implementations must call the __init__() constructor of their parent using super(BlockImplementation, self).__init__(**kwargs) at the beginning of the overriding __init__.

The methods _allocate() and _initialize() need to be overridden if the brick needs to allocate shared variables and initialize their values in order to function.

A brick can have any number of methods which apply the brick on Theano variables. These methods should be decorated with the application() decorator.

If a brick has children, they must be listed in the children attribute. Moreover, if the brick wants to control the configuration of its children, the _push_allocation_config() and _push_initialization_config() methods need to be overridden.

Examples

Most bricks have lazy initialization enabled.

>>> import theano
>>> from blocks.initialization import IsotropicGaussian, Constant
>>> from blocks.bricks import Linear
>>> linear = Linear(input_dim=5, output_dim=3,
...                 weights_init=IsotropicGaussian(),
...                 biases_init=Constant(0))
>>> x = theano.tensor.vector()
>>> linear.apply(x)  # Calls linear.allocate() automatically
linear_apply_output
>>> linear.initialize()  # Initializes the weight matrix

_abc_cache = <_weakrefset.WeakSet object at 0x7f42910e1d10>¶

_abc_negative_cache = <_weakrefset.WeakSet object at 0x7f42910e1d90>¶

_abc_negative_cache_version = 28¶

_abc_registry = <_weakrefset.WeakSet object at 0x7f42910e1c90>¶

_allocate()¶

Brick implementation of parameter initialization.

Implement this if your brick needs to allocate its parameters.

Warning

This method should never be called directly. Call initialize() instead.

_initialize()¶

Brick implementation of parameter initialization.

Implement this if your brick needs to initialize its parameters.

Warning

This method should never be called directly. Call initialize() instead.

_push_allocation_config()¶

Brick implementation of configuring child before allocation.

Implement this if your brick needs to set the configuration of its children before allocation.

Warning

This method should never be called directly. Call push_allocation_config() instead.

_push_initialization_config()¶

Brick implementation of configuring child before initialization.

Implement this if your brick needs to set the configuration of its children before initialization.

Warning

This method should never be called directly. Call push_initialization_config() instead.

allocate()¶

Allocate shared variables for parameters.

Based on the current configuration of this Brick create Theano shared variables to store the parameters. After allocation, parameters are accessible through the parameters attribute.

This method calls the allocate() method of all children first, allowing the _allocate() method to override the parameters of the children if needed.

Raises:	`ValueError` – If the configuration of this brick is insufficient to determine the number of parameters or their dimensionality to be initialized.

Notes

This method sets the parameters attribute to an empty list. This is in order to ensure that calls to this method completely reset the parameters.

children

get_dim(name)¶

Get dimension of an input/output variable of a brick.

Parameters:	name (str) – The name of the variable.

get_dims(names)¶

Get list of dimensions for a set of input/output variables.

Parameters:	names (list) – The variable names.
Returns:	dims – The dimensions of the sources.
Return type:	list

get_unique_path()¶: Returns unique path to this brick in the application graph.

initialize()¶

Initialize parameters.

Intialize parameters, such as weight matrices and biases.

Notes

If the brick has not allocated its parameters yet, this method will call the allocate() method in order to do so.

parameters

print_shapes = False: See Brick.print_shapes

push_allocation_config()¶

Push the configuration for allocation to child bricks.

Bricks can configure their children, based on their own current configuration. This will be automatically done by a call to allocate(), but if you want to override the configuration of child bricks manually, then you can call this function manually.

push_initialization_config()¶

Push the configuration for initialization to child bricks.

Bricks can configure their children, based on their own current configuration. This will be automatically done by a call to initialize(), but if you want to override the configuration of child bricks manually, then you can call this function manually.

class blocks.bricks.base.Children(brick, *args, **kwargs)¶

Bases: blocks.utils.containers.AnnotatingList

Adds the brick to the list of parents of its children.

_abc_cache = <_weakrefset.WeakSet object at 0x7f42910e1a10>¶

_abc_negative_cache = <_weakrefset.WeakSet object at 0x7f42910e1a90>¶

_abc_negative_cache_version = 28¶

_abc_registry = <_weakrefset.WeakSet object at 0x7f42910e1950>¶

_delitem(key)¶

_setitem(key, value)¶

class blocks.bricks.base.LazyNone(name)¶: Bases: object

class blocks.bricks.base.Parameters(brick, *args, **kwargs)¶

Bases: blocks.utils.containers.AnnotatingList

Adds the PARAMETER role to parameters automatically.

_abc_cache = <_weakrefset.WeakSet object at 0x7f42910e18d0>¶

_abc_negative_cache = <_weakrefset.WeakSet object at 0x7f42910e1910>¶

_abc_negative_cache_version = 28¶

_abc_registry = <_weakrefset.WeakSet object at 0x7f42910e1890>¶

_setitem(key, value)¶

class blocks.bricks.base._Brick¶

Bases: abc.ABCMeta

Metaclass which attaches brick instances to the applications.

In addition picklability of Application objects is ensured. This means that Application objects can not be added to a brick class after it is created. To allow adding application methods programatically, the following hook is supported: the class namespace is searched for decorators attribute, which can contain a list of functions to be applied to the namespace of the class being created. These functions can arbitratily modify this namespace.

blocks.bricks.base._variable_name(brick_name, application_name, name)¶

blocks.bricks.base.application(*args, **kwargs)¶

Decorator for methods that apply a brick to inputs.

Parameters:	optional (kwargs,) – The application method to wrap. optional** – Attributes to attach to this application.

Notes

This decorator replaces application methods with Application instances. It also sets the attributes given as keyword arguments to the decorator.

Note that this decorator purposely does not wrap the original method using e.g. wraps() or update_wrapper(), since that would make the class impossible to pickle (see notes at Application).

Examples

>>> class Foo(Brick):
...     @application(inputs=['x'], outputs=['y'])
...     def apply(self, x):
...         return x + 1
...     @application
...     def other_apply(self, x):
...         return x - 1
>>> foo = Foo()
>>> Foo.apply.inputs
['x']
>>> foo.apply.outputs
['y']
>>> Foo.other_apply 
<blocks.bricks.base.Application object at ...>

blocks.bricks.base.args_to_kwargs(args, f)¶

blocks.bricks.base.create_unbound_method(func, cls)¶

Create an unbounded method from a function and a class.

Notes

See https://bitbucket.org/gutworth/six/pull-request/64.

blocks.bricks.base.lazy(allocation=None, initialization=None)¶

Makes the initialization lazy.

This decorator allows the user to define positional arguments which will not be needed until the allocation or initialization stage of the brick. If these arguments are not passed, it will automatically replace them with a custom None object. It is assumed that the missing arguments can be set after initialization by setting attributes with the same name.

Parameters:	allocation (list) – A list of argument names that are needed for allocation. initialization (list) – A list of argument names that are needed for initialization.

Examples

>>> class SomeBrick(Brick):
...     @lazy(allocation=['a'], initialization=['b'])
...     def __init__(self, a, b, c='c', d=None):
...         print(a, b, c, d)
>>> brick = SomeBrick('a')
a NoneInitialization c None
>>> brick = SomeBrick(d='d', b='b')
NoneAllocation b c d

blocks.bricks.base.rename_function(function, new_name)¶

class blocks.bricks.Activation(name=None)¶

Bases: blocks.bricks.base.Brick

Elementwise application of activation function.

_abc_cache = <_weakrefset.WeakSet object at 0x7f42910ec290>¶

_abc_negative_cache = <_weakrefset.WeakSet object at 0x7f42910ec310>¶

_abc_negative_cache_version = 28¶

_abc_registry = <_weakrefset.WeakSet object at 0x7f42910ec210>¶

class blocks.bricks.ActivationDocumentation¶

Bases: blocks.bricks.base._Brick

Dynamically adds documentation to activations.

Notes

See http://bugs.python.org/issue12773.

Extensions¶

class blocks.extensions.predicates.OnLogRecord(record_name)¶

Bases: object

Trigger a callback when a certain log record is found.

Parameters:	record_name (str) – The record name to check.

class blocks.monitoring.evaluators.AggregationBuffer(variables, use_take_last=False)¶

Bases: object

Intermediate results of aggregating values of Theano variables.

Encapsulates aggregators for a list of Theano variables. Collects the respective updates and provides initialization and readout routines.

Parameters:	variables (list of `TensorVariable`) – The variable names are used as record names in the logs. Hence, all the variable names must be different. use_take_last (bool) – When `True`, the `TakeLast` aggregation scheme is used instead of `_DataIndependent` for those variables that do not require data to be computed.

initialization_updates¶

list of tuples

Initialization updates of the aggregators.

accumulation_updates¶

list of tuples

Accumulation updates of the aggregators.

readout_variables¶

dict

A dictionary of record names to TensorVariable representing the aggregated values.

inputs¶

list of TensorVariable

The list of inputs needed for accumulation.

_compile()¶: Compiles Theano functions.

Todo

The current compilation method does not account for updates attached to ComputationGraph elements. Compiling should be out-sourced to ComputationGraph to deal with it.

_create_aggregators()¶: Create aggregators and collect updates.

get_aggregated_values()¶: Readout the aggregated values.

initialize_aggregators()¶: Initialize the aggregators.

class blocks.monitoring.evaluators.DatasetEvaluator(variables, updates=None)¶

Bases: object

A DatasetEvaluator evaluates many Theano variables or other quantities.

The DatasetEvaluator provides a do-it-all method, evaluate(), which computes values of variables on a dataset.

Alternatively, methods initialize_aggregators(), process_batch(), get_aggregated_values() can be used with a custom loop over data.

The values computed on subsets of the given dataset are aggregated using the AggregationScheme`s provided in the `aggregation_scheme tags. If no tag is given, the value is averaged over minibatches. However, care is taken to ensure that variables which do not depend on data are not unnecessarily recomputed.

Parameters:

variables (list of TensorVariable and) –
MonitoredQuantity The variable names are used as record names in the logs. Hence, all the names must be different.

Each variable can be tagged with an AggregationScheme that specifies how the value can be computed for a data set by aggregating minibatches.
updates (list of tuples or OrderedDict or None) – TensorSharedVariable updates to be performed during evaluation. This parameter is only for Theano variables. Be careful not to update any model parameters as this is not intended to alter your model in any meaningfullway. A typical use case of this option arises when the theano function used for evaluation contains a call to:function:~theano.scan which might have returned shared variable updates.

_compile()¶: Compiles Theano functions.

Todo

The current compilation method does not account for updates attached to ComputationGraph elements. Compiling should be out-sourced to ComputationGraph to deal with it.

evaluate(data_stream)¶

Compute the variables over a data stream.

Parameters:	data_stream (instance of `DataStream`) – The data stream. Only the first epoch of data is used.
Returns:	A mapping from record names to the values computed on the provided dataset.

get_aggregated_values()¶

initialize_aggregators()¶

process_batch(batch)¶

class blocks.monitoring.evaluators.MonitoredQuantityBuffer(quantities)¶

Bases: object

Intermediate results of aggregating values of monitored-quantity.

Accumulate results for a list of monitored-quantity for every single batch. Provides initialization and readout routines to initialize each quantity and capture its accumulated results.

Parameters:	quantities (list of `MonitoredQuantity`) – The quantity names are used as record names in the logs. Hence, all the quantity names must be different.

requires¶

list of TensorVariable

Needed to calculate monitored-quantities.

quantity_names¶

list of str

Names of quantities.

inputs¶

list of TensorVariable

The list of inputs needed for variables in requires.

accumulate_quantities(numerical_values)¶: Accumulate the results for every batch.

get_aggregated_values()¶: Readout the accumulated values.

initialize()¶: Initialize the quantities.

Utils¶

class blocks.utils.containers.AnnotatingList(items=None)¶

Bases: _abcoll.MutableSequence

Mutable sequence performing operations on inserted/removed items.

Parameters:	items (iterable, optional) – An iterable of items to initialize the sequence with.

_abc_cache = <_weakrefset.WeakSet object at 0x7f42910e1790>¶

_abc_negative_cache = <_weakrefset.WeakSet object at 0x7f42910e1810>¶

_abc_negative_cache_version = 28¶

_abc_registry = <_weakrefset.WeakSet object at 0x7f42910e1710>¶

_delitem(key)¶: The operation to perform when an item is deleted.

_setitem(key, value)¶: The operation to perform when an item is inserted/appended.

insert(key, value)¶

class blocks.utils.profile.Profile¶

Bases: object

A profile of hierarchical timers.

Keeps track of timings performed with Timer. It also keeps track of the way these timings were nested and makes use of this information when reporting.

enter(name)¶

exit(t)¶

report(f=<open file '<stderr>', mode 'w' at 0x7f42a399b1e0>)¶

Print a report of timing information to standard output.

Parameters:	f (object, optional) – An object with a `write` method that accepts string inputs. Can be a file object, `sys.stdout`, etc. Defaults to `sys.stderr`.

class blocks.utils.profile.Timer(name, profile)¶

Bases: object

A context manager to time the execution time of code within it.

This timer is attached to a Profile object that it reports timings to. The Profile object accumulates the timings. Timers can be nested, which the Profile will automatically keep track of and use in its reporting.

Parameters:	name (str) – The name of this section. Expected to adhere to variable naming styles. profile (`Profile`) – The profile of the main loop. This is the object this context manager will report the execution time to. The accumulation and processing of timing information is handled by this object.

Notes

Timings are reported using timeit.default_timer().

class blocks.serialization.PersistentParameterID(zip_file, allow_unnamed=True, allow_duplicates=True)¶

Bases: theano.misc.pkl_utils.PersistentSharedVariableID

Persist the names of parameter arrays in the zip file.

Only Theano shared variables are persisted to the zip file using this method. Names are determined using the brick hierarchy, or the shared variable name.

Parameters:	allow_unnamed (bool, optional) – Allow shared variables without a name to be persisted. Defaults to `True`. allow_duplicates (bool, optional) – Allow multiple shared variables to have the same name, in which case they will be numbered e.g. x, x_2, x_3, etc. Defaults to `True`.
Raises:	`ValueError` – If an unnamed shared variable is encountered and allow_unnamed is `False`, or if two shared variables have the same name, and allow_duplicates is `False`.

class blocks.serialization.PicklerWithWarning(file, protocol=None)¶

Bases: pickle.Pickler

dispatch = {<type 'long'>: <function save_long at 0x7f42a11b0b18>, <type 'instance'>: <function save_inst at 0x7f42a11c3050>, <type 'float'>: <function save_float at 0x7f42a11b0b90>, <type 'classobj'>: <function save_global at 0x7f42902f55f0>, <type 'tuple'>: <function save_tuple at 0x7f42a11b0cf8>, <type 'NoneType'>: <function save_none at 0x7f42a11b09b0>, <type 'function'>: <function save_global at 0x7f42902f55f0>, <type 'str'>: <function save_string at 0x7f42a11b0c08>, <type 'type'>: <function save_global at 0x7f42902f55f0>, <type 'bool'>: <function save_bool at 0x7f42a11b0a28>, <type 'builtin_function_or_method'>: <function save_global at 0x7f42902f55f0>, <type 'list'>: <function save_list at 0x7f42a11b0de8>, <type 'int'>: <function save_int at 0x7f42a11b0aa0>, <type 'unicode'>: <function save_unicode at 0x7f42a11b0c80>, <type 'dict'>: <function save_dict at 0x7f42a11b0ed8>}¶

save_global(obj, name=None, **kwargs)¶

blocks.serialization.continue_training(path)¶

Continues training using checkpoint.

Parameters:	path (str) – Path to checkpoint.

Notes

Python picklers can unpickle objects from global namespace only if they are present in namespace where unpickling happens. Often global functions are needed for mapping, filtering and other data stream operations. In a case if the main loop uses global objects and this function fails with a message like ` AttributeError: 'module' object has no attribute '...' ` it means that you need to import these objects.

Examples

This function can be used in two ways: in your script where a main loop defined or in a different script. For later options see Notes section.

blocks.serialization.dump(obj, file_handler, protocol=2, persistent_id=<class 'blocks.serialization.PersistentParameterID'>, use_cpickle=False)¶

Pickles an object to a zip file using external persistence.

Parameters:

obj (object) – The object to pickle.
file_handler (file) – The file handle to save the object to.
protocol (int, optional) – The pickling protocol to use. Unlike Python’s built-in pickle, the default is set to 2 instead of 0 for Python 2. The Python 3 default (level 3) is maintained.
persistent_id (callable) – The callable that persists certain objects in the object hierarchy to separate files inside of the zip file. For example, PersistentNdarrayID saves any numpy.ndarray to a separate NPY file inside of the zip file.
use_cpickle (bool) – This enables the use of C-version of pickle (known as cPickle in Python 2). Note that this disables warnings about trying to pickle objects in the __main__ namespace.

Notes

The final file is simply a zipped file containing at least one file, pkl, which contains the pickled object. It can contain any other number of external objects. Note that the zip files are compatible with NumPy’s numpy.load() function.

>>> import numpy
>>> from blocks.bricks import MLP, Identity
>>> from blocks.initialization import Constant
>>> mlp = MLP([Identity()], [10, 10], weights_init=Constant(0.),
...           biases_init=Constant(0.))
>>> mlp.initialize()
>>> with open('model.zip', 'wb') as f:
...     dump(mlp, f)
>>> 'mlp-linear_0.W' in numpy.load('model.zip').keys()
True
>>> 'mlp-linear_0.b' in numpy.load('model.zip').keys()
True
>>> numpy.load('model.zip')['mlp-linear_0.W'].shape
(10, 10)
>>> with open('model.zip', 'rb') as f:
...     mlp2 = load(f)
>>> mlp2  
<blocks.bricks.MLP object at ...: name=mlp>

blocks.serialization.load_parameter_values(path)¶

Load parameter values saved by dump().

This is a thin wrapper over numpy.load(). It changes the names of the arrays to ones compatible with Model.set_param_values().

Parameters:	path (str or file) – The source for loading from.
Returns:
Return type:	A dictionary of (parameter name, numpy array) pairs.

blocks.serialization.secure_dump(object_, path, dump_function=<function dump at 0x7f42902f5848>, **kwargs)¶

Robust serialization - does not corrupt your files when failed.

Parameters:	object (object) – The object to be saved to the disk. path (str) – The destination path. dump_function (function) – The function that is used to perform the serialization. Must take an object and file object as arguments. By default, `dump()` is used. An alternative would be `pickle.dump()`. **kwargs – Keyword arguments to be passed to dump_function.