Recurrent Layers

[source]

SimpleRNN

polyaxon.layers.recurrent.SimpleRNN(mode, num_units, activation='sigmoid', dropout=None, num_layers=1, shared_layers=False, bias=True, weights_init=None, return_seq=False, return_state=False, initial_state=None, dynamic=False, trainable=True, restore=True, name='SimpleRNN')

Simple RNN (Simple Recurrent Layer.)

  • Output: if return_seq: 3-D Tensor [samples, timesteps, output dim].

    • else: 2-D Tensor [samples, output dim].
  • Args:

    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • num_units: int, number of units for this layer.
    • activation: str (name) or function (returning a Tensor). Default: 'sigmoid'.
    • dropout: tuple of float: (1 - input_keep_prob, 1 - output_keep_prob). The input and output keep probability.
    • num_layers: int how many times to stack the cell.
    • shared_layers: bool To share the same parameters in all created layers.
    • bias: bool. If True, a bias is used.
    • weights_init: str (name) or Tensor. Weights initialization.
    • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
    • return_state: bool. If True, returns a tuple with output and
      • states: (output, states).
    • initial_state: Tensor. An initial state for the RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
    • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
    • trainable: bool. If True, weights will be trainable.
    • restore: bool. If True, this layer weights will be restored when loading a model.
    • name: str. A name for this layer (optional).

[source]

LSTM

polyaxon.layers.recurrent.LSTM(mode, num_units, activation='tanh', inner_activation='sigmoid', dropout=None, num_layers=1, shared_layers=False, bias=True, weights_init=None, forget_bias=1.0, return_seq=False, return_state=False, initial_state=None, dynamic=False, trainable=True, restore=True, name='LSTM')

LSTM (Long Short Term Memory Recurrent Layer).

  • Output: if return_seq: 3-D Tensor [samples, timesteps, output dim].

    • else: 2-D Tensor [samples, output dim].
  • Args:

    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • num_units: int, number of units for this layer.
    • activation: str (name) or function (returning a Tensor). Default: 'tanh'.
    • inner_activation: str (name) or function (returning a Tensor). LSTM inner activation. Default: 'sigmoid'.
    • dropout: tuple of float: (1 - input_keep_prob, 1 - output_keep_prob). The input and output keep probability.
    • num_layers: int how many times to stack the cell.
    • shared_layers: bool To share the same parameters in all created layers.
    • bias: bool. If True, a bias is used.
    • weights_init: str (name) or Tensor. Weights initialization.
    • forget_bias: float. Bias of the forget gate. Default: 1.0.
    • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
    • return_state: bool. If True, returns a tuple with output and
      • states: (output, states).
    • initial_state: Tensor. An initial state for the RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
    • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
    • trainable: bool. If True, weights will be trainable.
    • restore: bool. If True, this layer weights will be restored when loading a model.
    • name: str. A name for this layer (optional).
  • References: Long Short Term Memory, Sepp Hochreiter & Jurgen Schmidhuber, Neural Computation 9(8): 1735-1780, 1997.

  • Links:

    • [http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf] (http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf)

[source]

GRU

polyaxon.layers.recurrent.GRU(mode, num_units, activation='tanh', inner_activation='sigmoid', dropout=None, num_layers=1, shared_layers=False, bias=True, weights_init=None, return_seq=False, return_state=False, initial_state=None, dynamic=False, trainable=True, restore=True, name='GRU')

GRU (Gated Recurrent Unit Layer).

  • Output: if return_seq: 3-D Tensor [samples, timesteps, output dim].

    • else: 2-D Tensor [samples, output dim].
  • Args:

    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • num_units: int, number of units for this layer.
    • activation: str (name) or function (returning a Tensor). Default: 'tanh'.
    • inner_activation: str (name) or function (returning a Tensor). GRU inner activation. Default: 'sigmoid'.
    • dropout: tuple of float: (1 - input_keep_prob, 1 - output_keep_prob). The input and output keep probability.
    • num_layers: int how many times to stack the cell.
    • shared_layers: bool To share the same parameters in all created layers.
    • bias: bool. If True, a bias is used.
    • weights_init: str (name) or Tensor. Weights initialization.
    • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
    • return_state: bool. If True, returns a tuple with output and
      • states: (output, states).
    • initial_state: Tensor. An initial state for the RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
    • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
    • trainable: bool. If True, weights will be trainable.
    • restore: bool. If True, this layer weights will be restored when loading a model.
    • name: str. A name for this layer (optional).
  • References: Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, K. Cho et al., 2014.

  • Links:


[source]

BidirectionalRNN

polyaxon.layers.recurrent.BidirectionalRNN(mode, rnncell_fw, rnncell_bw, return_seq=False, return_states=False, initial_state_fw=None, initial_state_bw=None, dynamic=False, name='BiRNN')

Bidirectional RNN.

Build a bidirectional recurrent neural network, it requires 2 RNN Cells to process sequence in forward and backward order. Any RNN Cell can be used i.e. SimpleRNN, LSTM, GRU... with its own parameters. But the two cells number of units must match.

  • Output: if return_seq: 3-D Tensor [samples, timesteps, output dim].

    • else: 2-D Tensor Layer [samples, output dim].
  • Args:

    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • rnncell_fw: RNNCell. The RNN Cell to use for foward computation.
    • rnncell_bw: RNNCell. The RNN Cell to use for backward computation.
    • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
    • return_states: bool. If True, returns a tuple with output and
      • states: (output, states).
    • initial_state_fw: Tensor. An initial state for the forward RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
    • initial_state_bw: Tensor. An initial state for the backward RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
    • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
    • name: str. A name for this layer (optional).

[source]

BasicRNNCell

polyaxon.layers.recurrent.BasicRNNCell(mode, num_units, activation='tanh', bias=True, weights_init=None, trainable=True, restore=True, name='BasicRNNCell')

The most basic RNN cell with custom params.

  • Args:
    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • num_units: int, number of units for this layer.
    • activation: str (name) or function (returning a Tensor). Default: 'tanh'.
    • bias: bool. If True, a bias is used.
    • weights_init: str (name) or Tensor. Weights initialization.
    • trainable: bool. If True, weights will be trainable.
    • restore: bool. If True, this layer weights will be restored when loading a model.
    • name: str. A name for this layer (optional).

[source]

GRUCell

polyaxon.layers.recurrent.GRUCell(mode, num_units, activation='tanh', inner_activation='sigmoid', bias=True, weights_init=None, trainable=True, restore=True, name='GRUCell')

Gated Recurrent Unit cell with custom params.

  • Args:
    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • num_units: int, number of units for this layer.
    • activation: str (name) or function (returning a Tensor). Default: 'tanh'.
    • inner_activation: str (name) or function (returning a Tensor). GRU inner activation. Default: 'sigmoid'.
    • bias: bool. If True, a bias is used.
    • weights_init: str (name) or Tensor. Weights initialization.
    • trainable: bool. If True, weights will be trainable.
    • restore: bool. If True, this layer weights will be restored when loading a model.
    • name: str. A name for this layer (optional).

[source]

BasicLSTMCell

polyaxon.layers.recurrent.BasicLSTMCell(mode, num_units, forget_bias=1.0, state_is_tuple=True, activation='tanh', inner_activation='sigmoid', bias=True, weights_init=None, batch_norm=False, trainable=True, restore=True, name='BasicLSTMCell')

Basic LSTM recurrent network cell with custo\m params.

The implementation is based on: http://arxiv.org/abs/1409.2329.

We add forget_bias (default: 1) to the biases of the forget gate in order to reduce the scale of forgetting in the beginning of the training.

It does not allow cell clipping, a projection layer, and does not use peep-hole connections: it is the basic baseline.

For advanced models, please use the full LSTMCell that follows.

  • Args:
    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • num_units: int, number of units for this layer.
    • forget_bias: float. Bias of the forget gate. Default: 1.0.
    • state_is_tuple: If True, accepted and returned states are n-tuples, where n = len(cells). If False, the states are all concatenated along the column axis. This latter behavior will soon be deprecated.
    • activation: str (name) or function (returning a Tensor). Default: 'tanh'.
    • inner_activation: str (name) or function (returning a Tensor). GRU inner activation. Default: 'sigmoid'.
    • bias: bool. If True, a bias is used.
    • weights_init: str (name) or Tensor. Weights initialization.
    • batch_norm: bool. If True, use batch normalization for this cell.
    • trainable: bool. If True, weights will be trainable.
    • restore: bool. If True, this layer weights will be restored when loading a model.
    • name: str. A name for this layer (optional).

[source]

DropoutWrapper

polyaxon.layers.recurrent.DropoutWrapper(mode, cell, input_keep_prob=1.0, output_keep_prob=1.0, seed=None, name='DropoutWrapper')

Operator adding dropout to inputs and outputs of the given cell.

Creates a cell with added input and/or output dropout.

Dropout is never used on the state.

  • Args:

    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • cell: an RNNCell, a projection to output_size is added to it.
    • input_keep_prob: unit Tensor or float between 0 and 1, input keep probability; if it is float and 1, no input dropout will be added.
    • output_keep_prob: unit Tensor or float between 0 and 1, output keep probability; if it is float and 1, no output dropout will be added.
    • seed: (optional) integer, the randomness seed.
  • Raises:

  • TypeError: if cell is not an RNNCell.
  • ValueError: if keep_prob is not between 0 and 1.

[source]

MultiRNNCell

polyaxon.layers.recurrent.MultiRNNCell(mode, cells, state_is_tuple=True, name='MultiRNNCell')

RNN cell composed sequentially of multiple simple cells.

Create a RNN cell composed sequentially of a number of RNNCells.

  • Args:

    • mode: str, Specifies if this training, evaluation or prediction. See Modes.
    • cells: list of RNNCells that will be composed in this order.
    • state_is_tuple: If True, accepted and returned states are n-tuples, where n = len(cells). If False, the states are all concatenated along the column axis. This latter behavior will soon be deprecated.
  • Raises:

    • ValueError: if cells is empty (not allowed), or at least one of the cells returns a state tuple but the flag state_is_tuple is False.