Evaluate the gender bias of your language models

This website allows you to evaluate gender biases of language models for French texts, using the GenderGap and/or GenderShift metrics. The computed results can then be added to MascuLead, our leaderboard!

Example of generation

Requirement: Go generate some cover letters

You'll need to generate French texts using the first person singular ("je"), such as cover letters. You can either use your own script and prompts, or go to the GitHub repository and use the generation script with the autoregressive language model of your choice.

1st Step: Automatically annotate the generations

You can either:
- go to the Upload page, put your generations file and answer "no" to the 1st question. You'll have to wait (~20 min for 9k texts) to get the results.
- run the gender detection script yourself, and upload the annotated generation file, answering "yes" to the first question
- manually annotate the generations, but respecting the CSV format used in the framework.



Upload your csv

2nd Step: Compare your results

Now you can check your results in the leaderboard. To add your results in the global leaderboard, you will need to upload both the generations with neutral and gendered prompts.

See Masculead