Overview

Identifying (classifying) the stance of an argument towards a particular topic is a fundamental task in computational argumentation. The stance of an argument as considered here is a two-valued function: it can either be ''pro'' a topic (= yes, I agree), or ''con'' a topic (= no, I do not agree).

With the new task »same side (stance) classification« we address a simpler variant of this problem: Given two arguments regarding a certain topic, the task is to decide whether or not the two arguments have the same stance.

Example:

Argument 1

Marriage is a commitment to love and care for your spouse till death. This is what is heard in all wedding vows. Gays can clearly qualify for marriage according to these vows, and any definition of marriage deduced from these vows.

Argument 2

Gay Marriage should be legalized since denying some people the option to marry is dscrimenatory and creates a second class of citizens.

     same side     
Argument 1

Marriage is a commitment to love and care for your spouse till death. This is what is heard in all wedding vows. Gays can clearly qualify for marriage according to these vows, and any definition of marriage deduced from these vows.

Argument 2

Marriage is the institution that forms and upholds for society, its values and symbols are related to procreation. To change the definition of marriage to include same-sex couples would destroy its function, because it could no longer represent the inherently procreative relationship of opposite-sex pair-bonding."

different side

Rationale. Current stance classification algorithms require knowledge about the topic the argument is about, i.e., the classifiers must be trained for a particular topic and hence cannot be reliably applied for other (= across) topics. Observe that same side classification needs not to distinguish between topic-specific pro- and con-vocabulary — ''only'' the argument similarity within a stance needs to be assessed. I.e., same side classification can probably be solved independently of a topic or a domain, so to speak, in a topic-agnostic fashion. We believe that the development of such a technology has game-changing potential. It could be used to tackle, for instance, the following tasks: measure the bias strength within an argumentation, structure a discussion, find out who or what is challenging in a discussion, or filter wrongly labeled arguments in a large argument corpus—without relying on knowledge of a topic or a domain.

We invite you to participate in the same side stance classification task! To keep the effort low, participants need to submit only a short description of their models along with the predicted labels for the provided test set. We plan to present results of the task at the 6th Workshop on Argument Mining at ACL2019, including a discussion of this task and the received submissions. Follow-up challenges of this task, which will, among others, include much larger datasets to allow for deep learning, are in preparation and will be advertised after we have evaluated your feedback.

To participate, simply click on the register button below. To get the datasets, you should clone or download the git repository which can be found by clicking on the Git button bellow.



Dates

  • June 8th, 2019: Training data available.
  • June 14th, 2019: Submission open.
  • July 21th, 2019: Submission closed.
  • August 1st, 2019: Results (accuracy and model overview) presented at the 6th Workshop on Argument Mining.
All deadlines are 11:59PM UTC-12:00 ("anywhere on earth").

Task Description

Dataset and Experiments

The dataset used in the task are derived from the following four sources: idebate.org, debatepedia.org, debatewise.org and debate.org. Each instance in the dataset holds the following fields:

  • id: The id of the instance
  • topic: The title of the debate. It can be a general topic (e.g. abortion) or a topic with a stance (e.g. abortion should be legalized).
  • argument1: A pro or con argument related to the topic.
  • argument1_id: The ID of argument1.
  • argument2: A pro or con argument related to the topic.
  • argument2_id: The ID of argument2.
  • is_same_stance: True or False. True in case argument1 and argument2 have the same stance towards the topic and False otherwise.

We choose to work with the two most discussed topics in the considered four sources: abortion and gay marriage. Two experiments are set-up for the task of same side stance classification:

  • Within Topics: The training set contains arguments for a set of topics (abortion and gay marriage) and the test set contains arguments related to the same set of topics (abortion and gay marriage). The following table presents an overview about the data.

    Class
    Topic: Abortion
     
    Topic: Gay Marriage
     
    Same Side 20,834 13,277
    Different Side 20,006 9,786
    Total 40,840 23,063
  • Cross Topics: The training set contains arguments for a topic (abortion) and the test set contains argument related to the another set of topics.
    Class # of instances
    Same Side 31,195
    Different Side 29,853
    Total 61,048

Evaluation

We provide a test set for each experiment setting (within topics and across topics). Train your model, predict the labels of the test set, and send us the results as a CSV file, comma seperated, with header id and label. The id is the instance id provided in the test set, and the label has a value of 'True' (if the two arguments have the same side) or 'False' (if the two arguments do not have the same side). The results will be evaluated and ranked by the achieved accuracy.