Model Page

The model panel allows the management of models, i.e. their training and evaluation.

Training

This section describes the Training tab, the parameters available for training as well as the key insights displayed on the page.

Quick Models

The Quick Models are classification models from scikit-learn trained on pre-computed features (what are features?). They are quick to train and do not require GPUs. These models can be used for Active Learning in the Annotate page.

When launching a model training, the model will be trained on the subset of the train set where labels exist. For this reason, there must be a few label annotated for each category (around 10). Once trained, the model will predict the labels for the rest of the train set in order to be used in Active Learning (see Annotation page)

Models available include:

• Logistic model with L2 penalty: A linear classifier that combines features with weighted coefficients and applies an L2 (ridge) penalty to keep weights small and reduce overfitting. The parameters are:
  • Cost: Inverse of regularization strength (scikit-learn's C). Lower values mean stronger regularization (smaller weights); higher values fit the training data more closely.
• Logistic model with L1 penalty: Same as above but with an L1 (lasso) penalty, which tends to drive uninformative feature weights exactly to zero (sparse model). Useful for feature selection. The parameters are:
  • Cost: Same as for L2 — lower values mean stronger regularization.
• k-Nearest-Neighbors (KNN): Classifies input based on the label of the k-nearest neighbors from the trainset. The parameters are:
  • Number of neighbors: Number of neighbors to look at to choose the label.
• Random Forest: A collection of Decision Tree classifiers that split the input space to generate a label. The parameters are:
  • Number of estimators: The number of decision trees in the forest.
  • Max features: The number of columns from the embeddings to consider per tree.
• Multinomial Naive Bayes classifier: A probabilistic classifier that assumes features are conditionally independent given the label and follow a multinomial distribution. Works well on count-based features (DFM, n-grams). The parameters are:
  • Alpha: Additive (Laplace/Lidstone) smoothing parameter. 0 means no smoothing; higher values smooth probability estimates and reduce the effect of unseen feature/label combinations.
  • Fit prior: If True, learn class prior probabilities from the data; if False, use a uniform prior.

On top of model-specific parameters, additional parameters can be set:

• Name for the model: The name used as a reference in the interface. It must be unique within a project.
• Automatically balance labels: If set to True, the model will be trained on a subset of the train dataset² constructed by picking an equal number of text inputs across labels.
• 10-fold cross validation: Computes performance test using the 10-fold cross validation technique.
• Labels to ignore: Labels selected will be ignored. You need at least two labels to start training a model. Models with ignored labels are available for Active Learning (see Annotation page).

BERTmodel

The BERT Models are pre-trained encoders models with a classification layer trained with the huggingface framework. They can take a dozen minutes to train and require GPUs¹. These models can be used for Active Learning in the Annotate page.

When launching a model training, the model will be trained on the subset of the train set where labels exist. For this reason, there must be a few label annotated for each category (around 10). After training, the best model (across epochs / checkpoints) will be used to predict the labels on the whole trainset to be used in Active Learning (see Annotation page).

The parameters for the model are:

• Name for the model The name used as a reference in the interface.
• Model base: The pretrained model used for generating the embeddings. Models are loaded through the HuggingFace interface (which models can I train?).
• Context window size: The number of tokens per entry. After tokenization each input is truncated/padded to match this size. If selecting Auto adjust Max context window size the window size will be calculated as the minimum between the maximum window size of the model and the length of the longest entry of the dataset.
• Epochs: The number of times the model will be trained on the train dataset² (more on choosing the hyperparameters).
• Learning Rate: The initial learning rate value used during training (more on choosing the hyperparameters).
• Weight Decay: The probability for a weight to not be updated during the backward propagation (more on choosing the hyperparameters).
• Use GPU: If set to True, the model will be trained on GPU — if available.
• Batch size: The number of inputs provided to the model at once. Large batch size makes training faster but requires higher tier hardware.
• Gradient Accumulation: The number of batches used to update the weights during the backward propagation. @JULIEN
• Eval: This is the number of evaluations — scoring performance on the train-eval dataset — performed during training.
• Train-eval split size: The ratio of elements in the train dataset² that will be used to assess the model performance during training.
• Label threshold: The minimum number of annotations per label required.
• Balance labels: If set to True, the train dataset² will be constructed using the same number of text inputs per label.
• Loss: The loss function used to optimise the model weights for training.
• Keep the best model: If set to True, the best model (i.e. the one with the lowest evaluation loss) across epochs/checkpoints is kept instead of the final one.
• Labels to ignore: Text inputs with these labels will be dropped out of the train dataset².

Trained BERT models are downloadable from the Export page. Users can download the model files as well as the predictions on the dataset of the project or an external dataset.

Key insights

For both classifiers, the application displays the Macro F1-Score (what metrics to use) as well as the confusion matrix and scores per label (Recall, Precision and F1)

• Show false predictions to display the text inputs where the classification failed.
• Download as JSON to download the confusion matrix, scores per label and false predictions as a JSON file.
• Parameters to show the parameters of the model and the training arguments.
• Rename to rename the model.

For BERT models only, the application displays the loss curve (how to read the loss curve?).

Evaluation

The Evaluation tab shows similar components as the Training tab. After selecting a classifier, Compute statistics on current annotations to compute the performance of the model on the full train, test and validation set³. The evaluation is computed on the different available dataset : the train dataset (for indication), the validation dataset if exists and the test dataset if exists.

The rest of the layout is similar to the one in the Training tab.