Investigating Semantic Roles for Emotion Role Prediction

Tushar Dhyani, Maximilian Wegge

Institut für Maschinelle Sprachverarbeitung, University of Stuttgart

A detailed scientific report on this project is available on Researchgate. Please feel free to read and share your feedback with me over email

Emotion analysis primarily focuses on classifying, predicting and retrieving emotions and their related properties from text. However, only few research was conducted towards analyzing the semantic roles of emotions, i.e. who is experiencing which emotion, what caused it and what or whom is it directed towards. This project investigate the influence of semantic role labels on emotion role prediction. Building on top of previous approaches and resources, I've implemented a framework for predicting emotion roles using different features with co-researcher Maximilian Wegge. We find that semantic role label features have no significant influence on the task and identify two possible reasons for that.

This project was conducted under the supervision of Dr. Roman Klinger

What is Semantic Role Labelling?

In natural languages, we understand the who, why, which, what, etc. in a sentence by understanding the words and phrases and their semantic meaning. This semantic meaning helps a lot in understanding why emotion is triggered, towards whom it is directed, or what gets affected. The process of pointing these individual components from a sentence is called Semantic Role Labelling or is sometimes analogous to Emotion Role labeling.

Semantic role labeling provides us with fine-grained control over emotion classification or prediction as we get a clear understanding of it.

emotion_roles Here, who (experiencer) feels which emotion (indicated by cue), which object, person, or instance the emotion is directed towards (target), and what evoked the emotion in the feeler (cause).

Datasets used:

Methods

Metric

We evaluate our approach by calculating the Jaccard value and reporting the F1 score of the overlapping spans. The Jaccard index is defined as $$ J(A,B) = \frac{| A \cap B |}{| A \cup B |} = \frac{| A \cap B|}{|A| + |B| - |A \cap B|} $$

In our case, we take A as our prediction span and B as the target span. We calculate a span to be correct only if the predicted and the target span have 80% overlap.

Results

We run our experiments on the testing split of the following corpora and compare the performances using jaccard score. For our evaluation, we set the overlap threshold to 0.8 and consider only the spans that have atleast 80% overlap compared to the training spans.

Table 1: Demonstrate the results of our experiments on Reman corpus. Our model achieves higher scores compared to our baseline using HMM model.
Dataset	Method	Exp	Tar	Cue	Cause
REMAN	HMM	0.228	0.021	-	0.011
	biLSTM-emb	0.435	0.051	-	0.101
	biLSTM-emb+srl(all)	0.494	0.025	-	0.078
	biLSTM-emb+srl(slct)	0.465	0.115	-	0.139

Table 2: The results of our experiments on Good News Everyone corpus. Our model's performance still remains higher compared to baseline HMM model.
Dataset	Method	Exp	Tar	Cue	Cause
GNE	HMM	0.370	0.062	0.293	0.321
	biLSTM-emb	0.595	0.228	0.441	0.654
	biLSTM-emb+srl(all)	0.591	0.312	0.443	0.638
	biLSTM-emb+srl(slct)	0.553	0.278	0.408	0.613

Table 3: Results of our model on Emotion Stimuli dataset for *cause* only. The results show that SRL features do not really impact the performance on respective corpus.
Dataset	Method	Exp	Tar	Cue	Cause
ES	HMM	-	-	-	0.215
	biLSTM-emb	-	-	-	0.593
	biLSTM-emb+srl(all)	-	-	-	0.591
	biLSTM-emb+srl(slct)	-	-	-	0.553

Table 4: Results of our model on Electoral Tweet dataset. The results still suggests that the model does not achieve any improvement using the SRL features.
Dataset	Method	Tar	Cue	Cause
ET	HMM	0.228	0.122	0.124
	biLSTM-emb	0.383	0.170	0.047
	biLSTM-emb+srl(all)	0.434	0.134	0.085
	biLSTM-emb+srl(slct)	0.443	0.136	0.070

Table 5: Results demonstrate no improvement using the SRL all and selected features in improvement of emotion detection.
Dataset	Method	Exp	Tar	Cue	Cause
ECA	HMM	-	-	-	0.025
	biLSTM-emb	-	-	-	0.155
	biLSTM-emb+srl(all)	-	-	-	0.206
	biLSTM-emb+srl(slct)	-	-	-	0.152

A detailed scientific report on this project is available on Researchgate. Please feel free to read and share your feedback with me over email