Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials
Weitere Details
performance across various NLP tasks. However, investigations have shown that
these models tend to rely on shortcut features, leading to inaccurate
predictions and causing the models to be unreliable at generalization to
out-of-distribution (OOD) samples. For instance, in the context of relation
extraction (RE), we would expect a model to identify the same relation
independently of the entities involved in it. For example, consider the
sentence "Leonardo da Vinci painted the Mona Lisa" expressing the
created(Leonardo_da_Vinci, Mona_Lisa) relation. If we substiute "Leonardo da
Vinci" with "Barack Obama", then the sentence still expresses the created
relation. A robust model is supposed to detect the same relation in both cases.
In this work, we describe several semantically-motivated strategies to generate
adversarial examples by replacing entity mentions and investigate how
state-of-the-art RE models perform under pressure. Our analyses show that the
performance of these models significantly deteriorates on the modified datasets
(avg. of -48.5% in F1), which indicates that these models rely to a great
extent on shortcuts, such as surface forms (or patterns therein) of entities,
without making full use of the information present in the sentences.
Zitierstile
Nolano G, Blum M, Ell B, Cimiano P. Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. In: arXiv:2402.19076. 2024.
Nolano, G., Blum, M., Ell, B., & Cimiano, P. (2024). Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. arXiv:2402.19076
Nolano, G., Blum, M., Ell, B., and Cimiano, P. (2024). “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials” in arXiv:2402.19076.
Nolano, G., et al., 2024. Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. In arXiv:2402.19076.
G. Nolano, et al., “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”, arXiv:2402.19076, 2024.
Nolano, G., Blum, M., Ell, B., Cimiano, P.: Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. arXiv:2402.19076. (2024).
Nolano, Gennaro, Blum, Moritz, Ell, Basil, and Cimiano, Philipp. “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”. arXiv:2402.19076. 2024.
Download
RDF/XML-Format
JSON-LD-Format
Turtle-Format
N3-Format
performance across various NLP tasks. However, investigations have shown that
these models tend to rely on shortcut features, leading to inaccurate
predictions and causing the models to be unreliable at generalization to
out-of-distribution (OOD) samples. For instance, in the context of relation
extraction (RE), we would expect a model to identify the same relation
independently of the entities involved in it. For example, consider the
sentence "Leonardo da Vinci painted the Mona Lisa" expressing the
created(Leonardo_da_Vinci, Mona_Lisa) relation. If we substiute "Leonardo da
Vinci" with "Barack Obama", then the sentence still expresses the created
relation. A robust model is supposed to detect the same relation in both cases.
In this work, we describe several semantically-motivated strategies to generate
adversarial examples by replacing entity mentions and investigate how
state-of-the-art RE models perform under pressure. Our analyses show that the
performance of these models significantly deteriorates on the modified datasets
(avg. of -48.5% in F1), which indicates that these models rely to a great
extent on shortcuts, such as surface forms (or patterns therein) of entities,
without making full use of the information present in the sentences.
Zitierstile
Nolano G, Blum M, Ell B, Cimiano P. Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. In: arXiv:2402.19076. 2024.
Nolano, G., Blum, M., Ell, B., & Cimiano, P. (2024). Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. arXiv:2402.19076
Nolano, G., Blum, M., Ell, B., and Cimiano, P. (2024). “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials” in arXiv:2402.19076.
Nolano, G., et al., 2024. Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. In arXiv:2402.19076.
G. Nolano, et al., “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”, arXiv:2402.19076, 2024.
Nolano, G., Blum, M., Ell, B., Cimiano, P.: Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. arXiv:2402.19076. (2024).
Nolano, Gennaro, Blum, Moritz, Ell, Basil, and Cimiano, Philipp. “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”. arXiv:2402.19076. 2024.
Download
RDF/XML-Format
JSON-LD-Format
Turtle-Format
N3-Format