sot

sotoxic

top 1% solution to toxic comment classification challenge on Kaggle.

Showing:

Popularity

Downloads/wk

0

GitHub Stars

182

Maintenance

Last Commit

3yrs ago

Contributors

1

Package

Dependencies

0

License

Categories

Readme

DeepToxic

This is part of 27th solution for the toxic comment classification challenge. For easy understanding, I only uploaded what I used in the final stage, and did not attach any experimental or deprecated codes.

Dataset and External pretrained embeddings

You can fetch the dataset here. I used 3 kind of word embeddings:

Overview

Preprocessing

We trained our models on 3 datasets with different preprocessing:

  • original dataset with spellings correction: by comparing the Levenshtein distance and a lot of regular expressions.
  • original dataset with pos taggings: We generate the part of speech (POS) tagging for every comment by TextBlob and concatenate the word embedding and POS embedding as a single one. Since TextBlob drops some tokens and punctuations when generating the POS sequences, that gives our models another view.
  • Riad's dataset: with very heavily data-cleaning, spelling correction and translation

Models

In our case, the simpler, the better. I tried some complicated structures (RHN, DPCNN, HAN). Most of them had performed very well locally but got lower AUC on the leaderboard. The models I kept trying during the final stage are the following two:

Pooled RNN (public: 0.9862, private: 0.9858) pooledRNN

Kmax text CNN (public: 0.9856 , private: 0.9849) kmaxCNN

As many competitors pointed out, dropout and batch-normalization are the keys to prevent overfitting. By applying the dropout on the word embedding directly and behind the pooling does great regularization both on train set and test set. Although model with many dropouts takes about 5 more epochs to coverage, it boosts our scores significantly. For instance, my RNN boosts from 0.9853 (private: 0.9850) to 0.9862 (private: 0.9858) after adding dropout layers.

For maximizing the utility of these datasets, besides training on the original labels, we also add a meta-label "bad_comment". If a comment is labeled, then it's considered to be a bad comment. The hypothesizes between these two labels sets are slightly different but with almost the same LB score, which leaves us room for the ensemble.

In order to increase the diversity and to deal with some toxic typos, we trained the models both on char-level and word-level. The results of char-level perform a bit worse (for charRNN: 0.983 on LB, 0.982 on PB, charCNN: 0.9808 on LB, 0.9801 on PB) but it does have a pretty low correlation with word-level models. By simply bagging my char-level and word-level result, it is good enough to push me over 0.9869 in the private test set. By the way, the hyperparameters influence the performance hugely in the char-based models. A large batch size (256), very long sequence length (1000) would ordinarily get a considerable result even though it takes much time for the K-fold validation. (my char-based models usually converge after 60~70 epochs which is about 5 times more than my word-based models.)

Performance of Single models

Scored by AUC on the private testset.

Word level

ModelFasttextGloveTwitter
AVRNN0.98580.98550.9843
Meta-AVRNN0.98500.9849No data
Pos-AVRNN0.9850No data0.9841
AVCNN0.98460.98450.9841
Meta-AVCNN0.98440.9844No data
Pos-AVCNN0.9850No dataNo data
KmaxTextCNN0.98490.98450.9835
TextCNN0.9837No dataNo data
RCNN0.98470.98420.9832
RHN0.9842No dataNo data

Char level

ModelAUC
AVRNN0.9821
KmaxCNN0.9801
AVCNN0.9797

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100