{"doi":"10.18653/v1/2020.acl-main.39","title":"Location Attention for Extrapolation to Longer Sequences","abstract":null,"journal":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","year":2020,"id":680585,"datarank":0.29188652235829704,"base_score":1.9459101490553132,"endowment":1.9459101490553132,"self_citation_contribution":0.29188652235829704,"citation_network_contribution":0.0,"self_endowment_contribution":0.29188652235829704,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":6,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":null,"is_data_producer":false,"deposit_databanks":null,"is_oa":false,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":null,"fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":1778193,"name":"Gautier Dagan","orcid":null,"position":1,"is_corresponding":false},{"id":1778194,"name":"Dieuwke Hupkes","orcid":null,"position":2,"is_corresponding":false},{"id":1778195,"name":"Elia Bruni","orcid":null,"position":3,"is_corresponding":false},{"id":1778192,"name":"Yann Dubois","orcid":null,"position":0,"is_corresponding":false}],"reference_count":0,"raw_metadata":{"has_enrichment":true,"resolved":true,"title":"Location Attention for Extrapolation to Longer Sequences","abstract":"Neural networks are surprisingly good at interpolating and perform remarkably well when the training set examples resemble those in the test set. However, they are often unable to extrapolate patterns beyond the seen data, even when the abstractions required for such patterns are simple. In this paper, we first review the notion of extrapolation, why it is important, and how one could hope to tackle it. We then focus on a specific type of extrapolation, which is especially useful for natural language processing: generalization to sequences longer than those seen during training. We hypothesize that models with a separate contentand location-based attention are more likely to extrapolate than those with common attention mechanisms. We empirically support our claim for recurrent seq2seq models with our proposed attention on variants of the Lookup Table This sheds light on some striking failures of neural models for sequences and on possible methods to approaching such issues.","is_dataset_classified":null,"base_score":1.0986122886681096,"endowment":1.0986122886681096,"datacite_reuse_total":0,"file_count":0,"downloads":0,"views":0,"has_version_chain":false,"is_dataset":false,"is_oa":false,"pmid":"26207759","pmcid":null,"openalex_id":"https://openalex.org/W2984272647","authors":[],"funders":[{"funder_name":"European Commission","grant_id":"790369","title":"Multimodal Agents Grounded via Interactive Communication"},{"funder_name":"Netherlands Organisation for Scientific Research (NWO)","grant_id":"024.001.006","title":"Language in Interaction"}],"total_grants":2,"fwci":0.1152,"citation_percentile":0.30354277,"influential_citations":0,"citation_trend":[{"year":2019,"count":1},{"year":2020,"count":1}],"oa_status":"gold","license":"cc-by","oa_locations":[{"url":"https://www.aclweb.org/anthology/2020.acl-main.39.pdf","host_type":""},{"url":"https://www.aclweb.org/anthology/2020.acl-main.39.pdf","host_type":""},{"url":"https://doi.org/10.18653/v1/2020.acl-main.39","host_type":""},{"url":"http://arxiv.org/abs/1911.03872","host_type":"repository"},{"url":"https://doi.org/10.48550/arxiv.1911.03872","host_type":"repository"},{"url":"https://arxiv.org/pdf/1911.03872","host_type":"repository"},{"url":"https://dx.doi.org/10.48550/arxiv.1911.03872","host_type":""},{"url":"https://dx.doi.org/10.18653/v1/2020.acl-main.39","host_type":""}],"fields_of_study":["Topic Modeling","Domain Adaptation and Few-Shot Learning","Natural Language Processing Techniques","0202 electrical engineering, electronic engineering, information engineering","02 engineering and technology","01 natural sciences","0105 earth and related environmental sciences"],"mesh_terms":[],"keywords":["Extrapolation","Generalization","Computer science","Set (abstract data type)","Task (project management)","Machine learning","Training set","Simple (philosophy)","Table (database)","Artificial intelligence","Artificial neural network","Focus (optics)","Test set","Deep neural networks","Data set","Natural language processing","Data mining","Mathematics","Statistics","FOS: Computer and information sciences","Computer Science - Machine Learning","Statistics - Machine Learning","Machine Learning (stat.ML)","Machine Learning (cs.LG)"],"sdg_mappings":[{"sdg_number":0,"sdg_label":"Quality Education"}],"linked_datasets":[],"clinical_trials":[],"software_tools":[],"database_accessions":[],"source":"live","citation_network_status":"fetched"},"created_at":"2026-08-17T15:32:32.043260Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}