The task is to classify the sentiment of potentially long texts for several aspects.
The key idea is to build a modern NLP package which supports explanations of model predictions.
The approximated decision explanations help you to infer how reliable predictions are.
The package is standalone, scalable, and can be freely extended to your needs.
We sum up thoughts in the article:
There are over 100 repositories on GitHub around sentiment analysis 1 2 3 4 5 6 7 8 9 . All of them are hard to commercialize and reuse open-source research projects. We clean up this excellent research. Please give a star if you like the project. This is important to keep this project alive.
The aim is to classify the sentiments of a text concerning given aspects. We have made several assumptions to make the service more helpful. Namely, the text being processed might be a full-length document, the aspects could contain several words (so may be defined more precisely), and most importantly, the service should provide an approximate explanation of any decision made, therefore, a user will be able to immediately infer the reliability of a prediction.
import aspect_based_sentiment_analysis as absa nlp = absa.load() text = ("We are great fans of Slack, but we wish the subscriptions " "were more accessible to small startups.") slack, price = nlp(text, aspects=['slack', 'price']) assert price.sentiment == absa.Sentiment.negative assert slack.sentiment == absa.Sentiment.positive
Above is an example of how quickly you can start to benefit from our open-source package.
All you need to do is to call the
load function which sets up the ready-to-use pipeline
You can explicitly pass the model name you wish to use (a list of available models is below), or a path to your model.
In spite of the simplicity of using fine-tune models, we encourage you to build a custom model which reflects your data.
The predictions will be more accurate and stable.
The pipeline provides an easy-to-use interface for making predictions. Even a highly accurate model will be useless if it is unclear how to correctly prepare the inputs and how to interpret the outputs. To make things clear, we have introduced a pipeline that is closely linked to a model. It is worth to know how to deal with the whole process, especially if you plan to build a custom model.
The diagram above illustrates an overview of the pipeline stages.
As usual, at the very beginning, we pre-process the inputs.
We convert the text and the aspects into a
task which keeps examples (pairs of a text and an aspect) that we can then further tokenize, encode and pass to the model.
The model makes a prediction, and here is a change.
Instead of directly post-processing the model outputs, we have added a review process wherein
the independent component called the
professor supervises and explains a model prediction.
The professor might dismiss a model prediction if the model internal states or outputs seem suspicious.
In the article [here], we discuss in detail how the model and the professor work.
import aspect_based_sentiment_analysis as absa name = 'absa/classifier-rest-0.2' model = absa.BertABSClassifier.from_pretrained(name) tokenizer = absa.BertTokenizer.from_pretrained(name) professor = absa.Professor(...) # Explained in detail later on. text_splitter = absa.sentencizer() # The English CNN model from SpaCy. nlp = absa.Pipeline(model, tokenizer, professor, text_splitter) # Break down the pipeline `call` method. task = nlp.preprocess(text=..., aspects=...) tokenized_examples = nlp.tokenize(task.examples) input_batch = nlp.encode(tokenized_examples) output_batch = nlp.predict(input_batch) predictions = nlp.review(tokenized_examples, output_batch) completed_task = nlp.postprocess(task, predictions)
Above is an example how to initialize the pipeline directly,
and we revise in code the process being discussed by exposing what calling the pipeline does under the hood.
We have omitted a lot of insignificant details but there's one thing we would like to highlight.
The sentiment of long texts tends to be fuzzy and neutral.
Therefore, you might want to split a text into smaller independent chunks, sometimes called spans.
These could include just a single sentence or several sentences.
It depends on how the
In this case, we are using the SpaCy CNN model, which splits a document into single sentences,
and, as a result each sentence can then be processed independently.
Note that longer spans have richer context information, so a model will have more information to consider.
Please take a look at the pipeline details here.
It's time to explain model reasoning, something which is extremely hard.
The key concept is to frame the problem of explaining a model decision as an independent task wherein
an aux. model, the
pattern recognizer, predicts patterns (weighted compositions of tokens, presented below) given model inputs, outputs, and internal states.
Due to time constraints, at first we did not want to research and build a trainable pattern recognizer.
Instead, we decided to start with a pattern recognizer that originates from our observations, prior knowledge.
The model, the aspect-based sentiment classifier, is based on the transformer architecture wherein self-attention layers hold the most parameters.
Therefore, one might conclude that understanding self-attention layers is a good proxy to understanding a model as a whole.
Accordingly, there are many articles that show how to explain a model decision
in simple terms, using attention values (internal states of self-attention layers) straightforwardly.
Inspired by these articles, we have also analyzed attention values (processing training examples) to search for any meaningful insights.
This exploratory study has led us to create the
BasicPatternRecognizer (details are here).
import aspect_based_sentiment_analysis as absa recognizer = absa.aux_models.BasicPatternRecognizer() nlp = absa.load(pattern_recognizer=recognizer) completed_task = nlp(text=..., aspects=['slack', 'price']) slack, price = completed_task.examples absa.summary(slack) absa.display(slack.review)
The explanations are only useful if they are correct. To form the basic pattern recognizer, we have made several assumptions (prior beliefs), therefore we should be careful about interpreting the explanations too literally. Even if the attention values have thought-provoking properties, for example, they encode rich linguistic relationships, there is no proven chain of causation. There are a lot of articles that illustrate various concerns why drawing conclusions about model reasoning directly from attentions might be misleading. In the article here, we validate and analyse explanations in detail.
In the table below, we present the State of the Art results on the SemEval 2014 evaluation dataset (dataset details are here). There are two available models for the restaurant and the laptop domains. The model implementation details here. The hyper-parameters optimization (with the explanation how to train a model) is here. You can easily reproduce our evaluations, look at the performance tests here.
|Model Name||Acc Rest||Acc Lapt||Release|
|LCF-ATEPC [code][paper]||90.18||82.29||Jan 2020|
|BERT-ADA [code][paper]||87.89||80.23||Nov 2019|
|BAT [code][paper]||86.03||79.35||Feb 2020|
You can use the pip:
pip install aspect-based-sentiment-analysis
Otherwise, clone the code and create the new environment via conda:
git clone firstname.lastname@example.org:ScalaConsultants/Aspect-Based-Sentiment-Analysis.git conda env create -f=environment.yml conda activate Aspect-Based-Sentiment-Analysis
The package works with the Python in the version 3.7 (the same as in Colab 2021).
How to use language models in the Aspect-Based Sentiment Analysis:
Introduction to the BERT interpretability:
The State of the Art results:
Developed by Scalac