Close menu

SURE

Sunderland Repository records the research produced by the University of Sunderland including practice-based research and theses.

An Exploration of Dropout with RNNs for Natural Language Inference

Gajbhiye, Amit, Jaf, Sardar, Al-Moubayed, Noura, McGough, Stephen and Bradley, Steven (2018) An Exploration of Dropout with RNNs for Natural Language Inference. In: Artificial Neural Networks and Machine Learning – ICANN 2018. Springer Lecture Notes in Computer Science, III (11141). Springer, pp. 156-167. ISBN 9783030014247

Item Type: Book Section

Abstract

Dropout is a crucial regularization technique for the Recurrent Neural Network (RNN) models of Natural Language Inference (NLI). However, dropout has not been evaluated for the effectiveness at different layers and dropout rates in NLI models. In this paper, we propose a novel RNN model for NLI and empirically evaluate the effect of applying dropout at different layers in the model. We also investigate the impact of varying dropout rates at these layers. Our empirical evaluation on a large (Stanford Natural Language Inference (SNLI)) and a small (SciTail) dataset suggest that dropout at each feed-forward connection severely affects the model accuracy at increasing dropout rates. We also show that regularizing the embedding layer is efficient for SNLI whereas regularizing the recurrent layer improves the accuracy for SciTail. Our model achieved an accuracy 86.14% on the SNLI dataset and 77.05% on SciTail.

[img]
Preview
PDF
exporaion-of-droout.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives.

Download (431kB) | Preview

More Information

Additional Information: [Conference]: Artificial Neural Networks and Machine Learning – ICANN 2018 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018
Depositing User: Sardar Jaf

Identifiers

Item ID: 10448
Identification Number: https://doi.org/10.1007/978-3-030-01424-7_16
ISBN: 9783030014247
URI: http://sure.sunderland.ac.uk/id/eprint/10448
Official URL: https://link.springer.com/chapter/10.1007/978-3-03...

Users with ORCIDS

ORCID for Sardar Jaf: ORCID iD orcid.org/0000-0002-5620-0277

Catalogue record

Date Deposited: 11 Mar 2019 10:53
Last Modified: 26 Nov 2020 15:15

Contributors

Author: Sardar Jaf ORCID iD
Author: Amit Gajbhiye
Author: Noura Al-Moubayed
Author: Stephen McGough
Author: Steven Bradley

University Divisions

Faculty of Technology

Subjects

Computing > Data Science
Computing > Artificial Intelligence
Languages > Languages
Computing > Programming

Actions (login required)

View Item View Item