Hate-Speech Classifier Comparison
Naive Bayes, Gradient Boosting, XGBoost and TabNet compared on a hate-speech dataset with stratified cross-validation.
Team project.
What was built
- Exploratory analysis of the dataset and its engineered features.
- Four models: Multinomial Naive Bayes, Gradient Boosting, XGBoost and TabNet.
- Evaluation with accuracy, precision, recall, F1, ROC-AUC and confusion matrices.
- Stratified 5-fold cross-validation for several of the models.
- A final comparison of the four, with conclusions.
We compared four classifiers that label short messages as hate speech or not, for a fictional company. The notebook starts with exploratory analysis of a provided, preprocessed dataset. Each model is trained and scored on a hold-out set of 2,000 messages with accuracy, precision, recall, F1 and ROC-AUC. Stratified 5-fold cross-validation checks that the results are stable.
Gradient Boosting, XGBoost and TabNet are within half a point of each other (F1 0.981, 0.982 and 0.977). Naive Bayes is lower at 0.939. The dataset came from the course and is not included.
Facts
| Item | Value |
|---|---|
| Hold-out test set | 2,000 messages, 1,000 per class |
| F1 on the test set | NB 0.939, GB 0.981, XGBoost 0.982, TabNet 0.977 |
| Gradient Boosting, 5-fold CV F1 | 0.979 (std 0.002) |