{"doi":"10.21437/interspeech.2021-332","title":"WavBERT: Exploiting Semantic and Non-Semantic Speech Using Wav2vec and BERT for Dementia Detection","abstract":"In this paper, we exploit semantic and non-semantic information from patient's speech data using Wav2vec and Bidirectional Encoder Representations from Transformers (BERT) for dementia detection. We first propose a basic WavBERT model by extracting semantic information from speech data using Wav2vec, and analyzing the semantic information using BERT for dementia detection. While the basic model discards the non-semantic information, we propose extended WavBERT models that convert the output of Wav2vec to the input to BERT for preserving the non-semantic information in dementia detection. Specifically, we determine the locations and lengths of inter-word pauses using the number of blank tokens from Wav2vec where the threshold for setting the pauses is automatically generated via BERT. We further design a pre-trained embedding conversion network that converts the output embedding of Wav2vec to the input embedding of BERT, enabling the fine-tuning of WavBERT with non-semantic information. Our evaluation results using the ADReSSo dataset showed that the WavBERT models achieved the highest accuracy of 83.1% in the classification task, the lowest Root-Mean-Square Error (RMSE) score of 4.44 in the regression task, and a mean F1 of 70.91% in the progression task. We confirmed the effectiveness of WavBERT models exploiting both semantic and non-semantic speech.","journal":"PubMed","year":2021,"id":212962,"datarank":0.5955437870328184,"base_score":3.970291913552122,"endowment":3.970291913552122,"self_citation_contribution":0.5955437870328184,"citation_network_contribution":0.0,"self_endowment_contribution":0.5955437870328184,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":52,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":0.9576,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2021-01-01","fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":804833,"name":"Abdelrahman Obyat","orcid":null,"position":1,"is_corresponding":false},{"id":658316,"name":"Xiaohui Liang","orcid":"0000-0003-4064-2393","position":2,"is_corresponding":false},{"id":272507,"name":"John A. Batsis","orcid":"0000-0002-0845-4416","position":3,"is_corresponding":false},{"id":315571,"name":"Robert M. Roth","orcid":"0000-0003-4374-6569","position":4,"is_corresponding":false},{"id":658315,"name":"Youxiang Zhu","orcid":"0009-0000-7294-1596","position":0,"is_corresponding":true}],"reference_count":9,"raw_metadata":{"citation_network_status":"fetched"},"created_at":"2026-07-18T23:52:31.378994Z","pmid":"37063977","pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}