Computational graph¶

class blocks.graph.Annotation¶

Bases: object

Annotations on Theano variables in a graph.

In Blocks annotations are automatically attached to variables created using bricks. One form of annotation is that many variables are assigned a role (see VariableRole). A second form of annotation comes in the form of attaching a Annotation instance to the variable’s tag attribute, with auxiliary variables and/or updates.

For example, we might be interested in the mean activation of certain application of a Linear brick. The variable representing the mean activation is attached as an auxiliary variable to the annotations of the input and output variables of this brick. Using the ComputationGraph class (the variables, auxiliary_variables, etc. attributes in particular) we can retrieve these Theano variables to pass on to the monitor, use as a regularizer, etc.

In most cases, annotations are added on a brick level (e.g. each brick will assign the weight norm of its weights as an auxiliary value) or on an application level (e.g. each time a brick is applied, its mean activation will become an auxiliary variable). However, you can also add annotations manually, by setting the annotation value of a variable’s tag field.

Examples

>>> from theano import tensor
>>> x = tensor.vector()
>>> annotation = Annotation()
>>> annotation.add_auxiliary_variable(x + 1, name='x_plus_1')
>>> add_annotation(x, annotation)
>>> y = x ** 2
>>> from blocks.graph import ComputationGraph
>>> cg = ComputationGraph([y])
>>> cg.auxiliary_variables
[x_plus_1]

add_auxiliary_variable(variable, roles=None, name=None)¶

Attach an auxiliary variable to the graph.

Auxiliary variables are Theano variables that are not part of a brick’s output, but can be useful nonetheless e.g. as a regularizer or to monitor during training progress.

Parameters:	variable (`TensorVariable`) – The variable you want to add. roles (list of `VariableRole` instances, optional) – The roles of this variable. The `AUXILIARY` role will automatically be added. Other options are `COST`, `WEIGHT`, etc. name (str, optional) – Name to give to the variable. If the variable already has a name it will be overwritten.

Examples

>>> from blocks.bricks.base import application, Brick
>>> from blocks.roles import COST
>>> from blocks.utils import shared_floatx_nans
>>> class Foo(Brick):
...     def _allocate(self):
...         W = shared_floatx_nans((10, 10))
...         self.add_auxiliary_variable(W.mean(), name='mean_W')
...     @application
...     def apply(self, x, application_call):
...         application_call.add_auxiliary_variable(
...             x - 1, name='x_minus_1')
...         application_call.add_auxiliary_variable(
...             x.mean(), roles=[COST], name='mean_x')
...         return x + 1
>>> from theano import tensor
>>> x = tensor.vector()
>>> y = Foo().apply(x)
>>> from blocks.filter import VariableFilter
>>> cg = ComputationGraph([y])
>>> var_filter = VariableFilter(roles=[AUXILIARY])
>>> var_filter(cg.variables)  
{x_minus_1, mean_W, mean_x}
>>> var_filter = VariableFilter(roles=[COST])
>>> var_filter(cg.variables)  
{mean_x}

class blocks.graph.ComputationGraph(outputs)¶

Bases: object

Encapsulates a managed Theano computation graph.

This implies that it not only contains the variables required to compute the given outputs, but also all the auxiliary variables and updates that were attached to these variables through the annotation system.

All variables are presented in topologically sorted order according to the apply nodes that they are an input to.

Parameters:	outputs ((list of) `TensorVariable`) – The output(s) of the computation graph.

inputs¶

list of TensorVariable

The inputs of the computation graph. This does not include shared variables and constants.

shared_variables¶

list of TensorSharedVariable

All the shared variables in the graph.

parameters¶

list of TensorSharedVariable

All the shared variables which have the PARAMETER role.

outputs¶

list of TensorVariable

The outputs of the computations graph (as passed to the constructor).

auxiliary_variables¶

list of TensorVariable

All variables which have the AUXILIARY role.

intermediary_variables¶

list of TensorVariable

Any variable that is not part of inputs or outputs.

variables¶

list of TensorVariable

All variables (including auxiliary) in the managed graph.

scans¶

list of Scan

All Scan ops used in this computation graph.

scan_variables¶

list of TensorVariable

All variables of the inner graphs of Scan ops.

updates¶

TensorSharedVariable updates

All the updates found attached to the annotations.

auxiliary_variables

dict_of_inputs()¶: Return a mapping from an input name to the input.

get_snapshot(data)¶

Evaluate all role-carrying Theano variables on given data.

Parameters:	data (dict of (data source, data) pairs) – Data for input variables. The sources should match with the names of the input variables.
Returns:
Return type:	Dictionary of (variable, variable value on given data) pairs.

get_theano_function(additional_updates=None, **kwargs)¶

Create Theano function from the graph contained.

Parameters:	*kwargs (dict*) – Keyword arguments to theano.function. Useful for specifying compilation modes or profiling.

has_inputs(variable)¶

Check if a variable depends on input variables.

Returns:	`True` if the given variable depends on input variables, `False` otherwise.
Return type:	bool

inputs: Inputs to the graph, excluding constants and shared variables.

intermediary_variables

parameters

replace(replacements)¶

Replace certain variables in the computation graph.

Parameters:	replacements (dict) – The mapping from variables to be replaced to the corresponding substitutes.

Examples

>>> import theano
>>> from theano import tensor, function
>>> x = tensor.scalar('x')
>>> y = x + 2
>>> z = y + 3
>>> a = z + 5

Let’s suppose we have dependent replacements like

>>> replacements = {y: x * 2, z: y * 3}
>>> cg = ComputationGraph([a])
>>> theano.pprint(a)  
'(((x + TensorConstant{2}) + TensorConstant{3}) +
TensorConstant{5})'
>>> cg_new = cg.replace(replacements)
>>> theano.pprint(
...     cg_new.outputs[0])  
'(((x * TensorConstant{2}) * TensorConstant{3}) +
TensorConstant{5})'

First two sums turned into multiplications

>>> float(function(cg_new.inputs, cg_new.outputs)(3.)[0])
23.0

scan_variables: Variables of Scan ops.

shared_variables

blocks.graph.add_annotation(var, annotation)¶

blocks.graph.apply_dropout(computation_graph, variables, drop_prob, rng=None, seed=None)¶

Returns a graph to variables in a computational graph.

Parameters:

computation_graph (instance of ComputationGraph) – The computation graph.
variables (list of TensorVariable) – Variables to be dropped out.
drop_prob (float) – Probability of dropping out. If you want to apply the dropout with different probabilities for different layers, call it several times.
rng (MRG_RandomStreams) – Random number generator.
seed (int) – Random seed to be used if rng was not specified.

Notes

For more information, see [DROPOUT].

[DROPOUT]

Hinton et al. Improving neural networks by preventing co-adaptation of feature detectors, arXiv:1207.0580.

Examples

>>> import numpy
>>> from theano import tensor, function
>>> from blocks.bricks import MLP, Identity
>>> from blocks.filter import VariableFilter
>>> from blocks.initialization import Constant
>>> from blocks.roles import INPUT
>>> linear = MLP([Identity(), Identity()], [2, 10, 2],
...              weights_init=Constant(1), biases_init=Constant(2))
>>> x = tensor.matrix('x')
>>> y = linear.apply(x)
>>> cg = ComputationGraph(y)

We are going to drop out all the input variables

>>> inputs = VariableFilter(roles=[INPUT])(cg.variables)

Here we apply dropout with default setting to our computation graph

>>> cg_dropout = apply_dropout(cg, inputs, 0.5)

Dropped out variables have role DROPOUT and are tagged with replacement_of tag. Let’s filter these variables and check if they have the links to original ones.

>>> dropped_out = VariableFilter(roles=[DROPOUT])(cg_dropout.variables)
>>> inputs_referenced = [var.tag.replacement_of for var in dropped_out]
>>> set(inputs) == set(inputs_referenced)
True

Compiling theano functions to forward propagate in original and dropped out graphs

>>> fprop = function(cg.inputs, cg.outputs[0])
>>> fprop_dropout = function(cg_dropout.inputs, cg_dropout.outputs[0])

Initialize an MLP and apply these functions

>>> linear.initialize()
>>> fprop(numpy.ones((3, 2),
...       dtype=theano.config.floatX))  
array([[ 42.,  42.],
       [ 42.,  42.],
       [ 42.,  42.]]...
>>> fprop_dropout(numpy.ones((3, 2),
...               dtype=theano.config.floatX))  
array([[ 0.,  0.],
       [ 0.,  0.],
       [ 0.,  0.]]...

And after the second run answer is different

>>> fprop_dropout(numpy.ones((3, 2),
...               dtype=theano.config.floatX))  
array([[   0.,   52.],
       [ 100.,    0.],
       [   0.,    0.]]...

blocks.graph.apply_noise(computation_graph, variables, level, seed=None)¶

Add Gaussian noise to certain variable of a computation graph.

Parameters:	computation_graph (instance of `ComputationGraph`) – The computation graph. variables (`TensorVariable`) – Variables to add noise to. level (float) – Noise level. seed (int, optional) – The seed with which `MRG_RandomStreams` is initialized, is set to 1 by default.

blocks.graph.collect_parameters(computation_graph, parameters)¶

Replace parameters with a single shared variable.

This can be useful if you need to calculate the full Hessian of a computational graph. It replaces parameters with slices of a single large vectors like

>>> from blocks.utils import shared_floatx
>>> W1 = shared_floatx(numpy.random.rand(10, 10))
>>> W2 = shared_floatx(numpy.random.rand(10, 10))
>>> all_parameters = shared_floatx(numpy.concatenate(
...     [W1.get_value().flatten(), W2.get_value().flatten()]))
>>> W1 = all_parameters[:W1.size]
>>> W2 = all_parameters[W1.size:]

Parameters:	computation_graph (`ComputationGraph` instance) – The managed Theano graph in which to collect parameters. parameters (list of Theano shared variables) – The parameters whose values should be collected.
Returns:	A new Theano graph which has all the given parameters collected into a single large shared variable.
Return type:	ComputationGraph instance

Notes

Note that this replacement makes the training of the model significantly slower because of the large amount of Theano’s set_subtensor calls needed to train the model.

Examples

>>> from blocks.bricks import MLP, Logistic
>>> from blocks.bricks.cost import SquaredError
>>> from theano import tensor
>>> x = tensor.matrix()
>>> mlp = MLP(activations=[Logistic(), Logistic()],
...           dims=[784, 100, 784])
>>> cost = SquaredError().apply(x, mlp.apply(x))
>>> cg = ComputationGraph(cost)
>>> new_cg = collect_parameters(cg, cg.shared_variables)

The new graph only has a single shared variable. This variable receives the COLLECTOR role.

>>> new_cg.shared_variables
[collected_parameters]

The bricks’ variables have been replaced with reshaped segments of this single shared variable. These replacements are given the COLLECTED role.

>>> from blocks.filter import VariableFilter
>>> from blocks.roles import PARAMETER
>>> var_filter = VariableFilter(roles=[COLLECTED])
>>> var_filter(new_cg.variables)  
[Reshape{1}.0, Reshape{1}.0, Reshape{2}.0, Reshape{2}.0]