Class FeedforwardDependencyTrainer

java.lang.Object
opennlp.tools.depparse.FeedforwardDependencyTrainer

public final class FeedforwardDependencyTrainer extends Object
Trains the FeedforwardDependencyModel: oracle-derived transition examples, minibatch AdaGrad over a softmax cross-entropy loss, cube activation, and inverted dropout on the hidden layer, the training recipe of Chen and Manning (2014).

The hyperparameters are read from TrainingParameters: the keys of this class plus Parameters.ITERATIONS_PARAM for the number of epochs, Parameters.CUTOFF_PARAM for the minimum word frequency, QNTrainer.L2COST_PARAM for the L2 penalty on the dense weights (default 1.0E-8), and BeamSearch.BEAM_SIZE_PARAMETER for the refinement beam. Other keys are ignored.

Words below the frequency cutoff share the reserved unknown row, which is therefore trained only when such words exist; the unknown rows of the tag and label vocabularies keep their random initialization. Absent template positions share a trained padding row. Non-projective samples have no arc-standard derivation and are skipped. Training is deterministic for a fixed SEED_PARAM.

Since:
3.0.0
  • Field Details

    • EMBEDDING_SIZE_PARAM

      public static final String EMBEDDING_SIZE_PARAM
      The training parameter key of the embedding dimensionality; default 50.
      See Also:
    • HIDDEN_SIZE_PARAM

      public static final String HIDDEN_SIZE_PARAM
      The training parameter key of the hidden layer width; default 200.
      See Also:
    • BATCH_SIZE_PARAM

      public static final String BATCH_SIZE_PARAM
      The training parameter key of the minibatch size; default 256.
      See Also:
    • LEARNING_RATE_PARAM

      public static final String LEARNING_RATE_PARAM
      The training parameter key of the AdaGrad step size; default 0.02.
      See Also:
    • DROPOUT_PARAM

      public static final String DROPOUT_PARAM
      The training parameter key of the hidden dropout probability; default 0.5.
      See Also:
    • SEED_PARAM

      public static final String SEED_PARAM
      The training parameter key of the random seed, read as an int; default 17.
      See Also:
    • DEFAULT_EMBEDDING_SIZE

      public static final int DEFAULT_EMBEDDING_SIZE
      The default embedding dimensionality.
      See Also:
    • DEFAULT_HIDDEN_SIZE

      public static final int DEFAULT_HIDDEN_SIZE
      The default hidden layer width.
      See Also:
    • DEFAULT_EPOCHS

      public static final int DEFAULT_EPOCHS
      The default number of epochs.
      See Also:
    • DEFAULT_BATCH_SIZE

      public static final int DEFAULT_BATCH_SIZE
      The default minibatch size.
      See Also:
    • DEFAULT_LEARNING_RATE

      public static final double DEFAULT_LEARNING_RATE
      The default AdaGrad step size.
      See Also:
    • DEFAULT_L2

      public static final double DEFAULT_L2
      The default L2 penalty.
      See Also:
    • DEFAULT_DROPOUT

      public static final double DEFAULT_DROPOUT
      The default hidden dropout probability.
      See Also:
    • DEFAULT_WORD_CUTOFF

      public static final int DEFAULT_WORD_CUTOFF
      The default minimum word frequency for an embedding row.
      See Also:
    • DEFAULT_SEED

      public static final int DEFAULT_SEED
      The default random seed.
      See Also:
    • DEFAULT_BEAM_SIZE

      public static final int DEFAULT_BEAM_SIZE
      The default refinement beam size.
      See Also:
  • Method Details

    • train

      public static FeedforwardDependencyModel train(String languageCode, opennlp.tools.util.ObjectStream<opennlp.tools.depparse.DependencySample> samples, opennlp.tools.util.TrainingParameters parameters) throws IOException
      Trains a model from dependency samples.
      Parameters:
      languageCode - The ISO language code of the training data. Must not be null.
      samples - The training samples. Must not be null.
      parameters - The TrainingParameters. Must not be null.
      Returns:
      A trained FeedforwardDependencyModel. Never null.
      Throws:
      IOException - Thrown if reading the samples fails.
      IllegalArgumentException - Thrown if a parameter is null, a hyperparameter is out of range, no trainable example can be derived from the samples, a vocabulary or the three vocabularies together exceed ResourceLimits.MAX_ENTRIES entries or a symbol as many UTF-8 bytes, or the network would store more than ResourceLimits.MAX_MATRIX_CELLS values.
      IllegalStateException - Thrown if an epoch produces a non-finite parameter.
    • refine

      public static FeedforwardDependencyModel refine(FeedforwardDependencyModel model, opennlp.tools.util.ObjectStream<opennlp.tools.depparse.DependencySample> samples, opennlp.tools.util.TrainingParameters parameters) throws IOException
      Refines a locally trained model with beam search and the early-update method of Collins and Roark (2004). The loss is conditional likelihood over candidate paths scored by summed log-probabilities.

      Refinement trains a copy with per-sentence AdaGrad steps and no dropout. Parameters.ITERATIONS_PARAM sets the number of passes and BeamSearch.BEAM_SIZE_PARAMETER the beam width, which the manifest of the result names as the beam size to parse it with. The vocabularies and the transition inventory of model are not extended, so the refined model carries the word cutoff of model.

      Parameters:
      model - The locally trained model. Not modified. Must not be null.
      samples - The training samples. Must not be null.
      parameters - The TrainingParameters; the epochs, learning rate, L2 penalty, seed, and beam size apply. Must not be null.
      Returns:
      A new refined model, distinct from model. Never null.
      Throws:
      IOException - Thrown if reading the samples fails.
      IllegalArgumentException - Thrown if a parameter is null, a hyperparameter is out of range, the beam size is below 2, no trainable sample can be derived, or a sample requires a transition model does not know.
      IllegalStateException - Thrown if an epoch produces a non-finite parameter, or the network produces a non-finite transition score while a sentence is decoded.