Main workflows

There are different ways of using a software. This page presents possible ways of using ActiveTigger for your research.

Annotate a corpus collaboratively

ActiveTigger makes collaborative annotation seamless through the following features:

• Creation of a scheme) and a codebook for the project.
• Annotation of the text inputs by each user.
• Comparison of annotations and measurement of agreement.
• Export the annotated corpus for analysis.

Why choose ActiveTigger for this task:

• User friendly interface, on any device.
• Seamless integration of collaborative work.

Here is an overview of the steps necessary to set up and undertake an annotation project with manual annotation:

• Step 0 — Prepare your corpus: Prepare your corpus as a CSV, XLSX or parquet file. More information here
• Step 1 — Create your project: On the Project Creation page, give the project a name, upload the data file and enter the columns you require for the annotation (ID, text, contextual). Set the number of rows in the train set to the size of your dataset. You may consider setting secondary parameters.
• Step 2 — Create your scheme and write down your codebook: On the Codebook page, create a scheme that fits your study (multiclass, multilabel or span — more info on the available schemes) and start writing the instructions for labelling the corpus. All users can see the schemes, labels and codebook of your project, but only managers and contributors can edit them.
• Step 3 — Give your collaborators access to your project: Go to the Users page and authorize your collaborators to access and annotate text inputs from your project.
• Step 4 — Each collaborator starts annotating the corpus: Go to the Annotate page and start the annotation process.
• Step 5 — Curate the annotation (Only available to projects with more than one user): On the Annotate page, choose "Curate". You will be able to see annotations differences between schemes or between users and choose which annotation to keep (as well as agreement metrics).
• Step 6 — Export your annotated corpus: On the Export page, choose "Annotations" and All annotations/schemes to download your corpus as a CSV file, or XLSX or parquet.

In practice, the annotation process is not so linear. It is normal to move back and forth between step 3, step 4 and step 5 to refine your codebook and change your annotation strategy. You may also want to use exploration tools (see below), or use machine learning to find the most challenging text inputs and test your codebook on rare or ambiguous cases.

Accelerate annotation with active learning and BERT fine-tuning

ActiveTigger makes machine learning methods easily available to social scientists, in particular by implementing the full pipeline for finetuning classifier models (what are BERT models? How to finetune BERT models? ). These models can automatically extend your human-developed annotation scheme on a larger dataset. This workflow consists of:

• Creation of a scheme and a codebook for the project.
• Annotation of the text inputs.
• Training classifier models.
• Inferring labels on non-annotated data.
• Export human and inferred annotations for analysis.

Why choose ActiveTigger for this task:

• User friendly interface and model training without any coding skills required.
• Safer and reproducible use of machine learning tools for annotating a corpus.
• An easy introduction to good practices in machine learning.

Here is an overview of the steps to complete this workflow:

• Step 0 — Prepare your corpus: Prepare your corpus as a CSV, XLSX or parquet file. More information here
• Step 1 — Create your project: On the Project Creation page, give the project a name, upload the data file and enter the columns you require for the annotation (ID, text, contextual). Set the number of rows in the train set to about 70% of your dataset, 15% to the test set and 15% to the validation set (more info). You may consider setting secondary parameters.
• Step 2 — Create your scheme and write down your codebook: On the Codebook page, create a scheme that fits your study (for now, only multiclass schemes can be used for training classifier models — more info on the available schemes) and start writing the instructions for labelling the corpus.
• Step 3 — Start annotating your corpus: Go to the Annotate page and start the annotation process. You can:
  • Annotate text inputs (add, remove or skip) and leave comments
  • Choose which text inputs to annotate or the order in which you wish to see them (more info).
• Step 4 — Train a quick model to enable active learning: Once you have annotated a small initial batch (a few dozen examples covering each label), go to the Model page and Create new quick model in the Quick Models section. Quick models are lightweight classifiers trained on pre-computed features (be sure to have at least one available) — they train in seconds and run on CPU. Once trained, you can return to the Annotate page and turn on Active Learning to surface the text inputs the model is most uncertain about (more info). Retrain the quick model periodically as you accumulate more annotations.
• Step 5 — Train a BERT model: Once you have enough annotations for the quick model to plateau, switch to a BERT classifier for higher quality. On the Model page and in the Training tab, Create new BERT model and set hyperparameters (more info on setting hyperparameters). You can also use the BERT model itself to drive active learning from this point on.
• Step 6 — Evaluate the model on validation set: On the Model page and in the Evaluation tab, select your model and Compute predictions to classify text inputs from the validation and test set to compute metrics on these sets.
  • Once you have annotated your training set, you will probably have a better understanding of your data, and stabilized the definitions of your labels.
  • You can now annotate your validation set, and check whether your model performs as well as on the training set.
  • Again, iterate between annotation and validation quality, until you are satisfied with the quality scores.
  • If you have trained several models, choose the model maximizing the quality scores on the validation set.

• Step 7 — Annotate the test set: this is the final step, which should only be used to evaluate the quality of your final model. ⚠️ Optimising the quality scores on the test set will lead to biased performance estimators!

• Step 8 — Export your annotated corpus and model: On the Export page and "Models tab", select the bert model:

  • Prediction on complete dataset infer labels on the full dataset as imported upon project creation.
  • Prediction on external dataset to import a dataset and infer the labels.
  • Export fine-tuned model (large file) download the weights of the fine-tuned model (good practice for reproducibility).

Once again, this workflow tends to be much less linear in practice. You will likely move back and forth between step 3 and step 4 — annotating a batch, retraining the quick model, annotating more with active learning — until you have enough data to train a BERT model in step 5, and then between step 5 and step 6 to refine your hyperparameters. This workflow is compatible with the collaborative annotation workflow as well as the exploration workflow.

Explore your data

Sometimes, with a large textual dataset, it is difficult to get an overview of what your dataset contains. ActiveTigger provides tools to explore your corpus (either as an independent step or as part of an annotation process), helping you understand its structure, identify clusters of similar content, and make more informed annotation decisions. You can use of the tabular oversight functionality of the interface, or make use of topic models.

For instance, you have a large dataset and you want to identify the sample on which you will really work.

This workflow consists of:

• Browsing and filtering your data in a tabular view.
• Computing a topic model to map thematic structures in your corpus.
• Using topics to guide annotation or as a starting point for a scheme.
• Exporting topic outputs for further analysis.

Why choose ActiveTigger for this task:

• No-code access to state-of-the-art topic modelling (BERTopic).
• Visual, interactive exploration of your corpus.
• A seamless bridge between exploration and annotation.

Here is an overview of the steps to explore your data with BERTopic:

• Step 0 — Prepare your corpus: Prepare your corpus as a CSV, XLSX or parquet file. More information here
• Step 1 — Create your project: On the Project Creation page, give the project a name, upload your data file and enter the columns you require (ID, text, contextual). You may consider setting secondary parameters.
• Step 2 — Browse your data (optional): On the Explore page, open the Tabular View tab to filter and browse your corpus. You can filter by dataset (train, valid, test), by label, or by regex query. Click on any ID to go directly to the Annotate page for that text input.
• Step 3 — Compute a BERTopic model: On the Explore page, go to the Topic model tab and click Compute new BERTopic.
• Step 4 — Explore the topic model results: Once computed, the Topic model tab displays:
  • A visual map of your corpus with each text input coloured by topic.
  • A topic table summarising each topic's ID, size, name, representative keywords, and representative documents.
  • export the results in different ways. See here

Using topic models for exploring your dataset can also be integrated into the above workflow pipelines. We recommend complementing a topic model approach with familiarizing yourself with your corpus by simply reading it, or through an annotation process. Confiding too much in an automated synthesis of your text data could risk missing important aspects of your corpus.

However, active learning could still be very helpful: just follow step 1 of the following section, and iterate until you are confident you have found all the positive cases in your dataset.

The only difference: use the complete dataset as your training set.