Attributing Multilingual MT Models

Inseq supports attribution of multilingual MT models such as mBART, M2M-100 and NLLB.

These models differ from standard encoder-decoder systems in that you will have to specify the source and target languages, which are used to include a flag in the input to the model. In the following example we attribute a pair of inputs using M2M-100:

import inseq
from inseq.data.aggregator import SubwordAggregator

model = inseq.load_model(
    "facebook/m2m100_418M",
    "input_x_gradient",
    # The tokenizer_kwargs are used to specify the source and target languages upon initialization
    tokenizer_kwargs={"src_lang": "en", "tgt_lang": "it"},
)

out = model.attribute(
    "Did you know? The Inseq library is very flexible!",
    # Step the correct BOS language token
    generation_args={"forced_bos_token_id": model.tokenizer.lang_code_to_id["it"]},
    attribute_target=True,
    step_scores=["probability"],
)
# Aggregate the attribution scores at subword level
out.aggregate().show(aggregator=SubwordAggregator)

Probability scores for the target language id should be disregarded, since the token is manually set before generation. The language ids are model-specific and can be found in the Hugging Face Hub repositories of the models.