{"doi":"10.29375/25392115.5054","title":"A snapshot of parallelism in distributed deep learning training","abstract":"<jats:p>The accelerated development of applications related to artificial intelligence has generated the creation of increasingly complex neural network models with enormous amounts of parameters, currently reaching up to trillions of parameters. Therefore, it makes your training almost impossible without the parallelization of training. Parallelism applied with different approaches is the mechanism that has been used to solve the problem of training on a large scale. This paper presents a glimpse of the state of the art related to parallelism in deep learning training from multiple points of view.  The topics of pipeline parallelism, hybrid parallelism, mixture-of-experts and auto-parallelism are addressed in this study, which currently play a leading role in scientific research related to this area. Finally, we develop a series of experiments with data parallelism and model parallelism. The objective is that the reader can observe the performance of two types of parallelism and understand more clearly the approach of each one.</jats:p>","journal":"Revista Colombiana de Computación","year":2024,"id":588401,"datarank":0.0,"base_score":0.0,"endowment":0.0,"self_citation_contribution":0.0,"citation_network_contribution":0.0,"self_endowment_contribution":0.0,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":0,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":null,"is_data_producer":false,"deposit_databanks":null,"is_oa":false,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":null,"fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":532225,"name":"Gabriel Núñez","orcid":"0000-0002-2858-1789","position":1,"is_corresponding":false},{"id":1505307,"name":"Elvis Rojas","orcid":null,"position":2,"is_corresponding":false},{"id":1505306,"name":"Hairol Romero-Sandí","orcid":null,"position":0,"is_corresponding":false}],"reference_count":0,"raw_metadata":{"has_enrichment":true,"resolved":true,"title":"A snapshot of parallelism in distributed deep learning training","abstract":"<jats:p>The accelerated development of applications related to artificial intelligence has generated the creation of increasingly complex neural network models with enormous amounts of parameters, currently reaching up to trillions of parameters. Therefore, it makes your training almost impossible without the parallelization of training. Parallelism applied with different approaches is the mechanism that has been used to solve the problem of training on a large scale. This paper presents a glimpse of the state of the art related to parallelism in deep learning training from multiple points of view.  The topics of pipeline parallelism, hybrid parallelism, mixture-of-experts and auto-parallelism are addressed in this study, which currently play a leading role in scientific research related to this area. Finally, we develop a series of experiments with data parallelism and model parallelism. The objective is that the reader can observe the performance of two types of parallelism and understand more clearly the approach of each one.</jats:p>","is_dataset_classified":null,"base_score":0.0,"endowment":0.0,"datacite_reuse_total":0,"file_count":0,"downloads":0,"views":0,"has_version_chain":false,"is_dataset":false,"is_oa":false,"pmid":"20725694","pmcid":null,"openalex_id":"https://openalex.org/W4400951174","authors":[],"funders":[],"total_grants":0,"fwci":0.0,"citation_percentile":0.08499256,"influential_citations":0,"citation_trend":[],"oa_status":"gold","license":"cc-by-nc-sa","oa_locations":[{"url":"https://revistas.unab.edu.co/index.php/rcc/article/download/5054/3970","host_type":"journal"},{"url":"https://revistas.unab.edu.co/index.php/rcc/article/download/5054/3970","host_type":"GOLD"},{"url":"https://revistas.unab.edu.co/index.php/rcc/article/download/5054/3970","host_type":"publisher"},{"url":"http://dx.doi.org/10.29375/25392115.5054","host_type":"journal"},{"url":"https://dialnet.unirioja.es/servlet/oaiart?codigo=9714355","host_type":"repository"},{"url":"https://doaj.org/article/17b27ff6706641a79de8e9943a772a84","host_type":"repository"},{"url":"https://doaj.org/article/6d41a4d051c143cbafd2347be397a16d","host_type":"repository"},{"url":"https://doaj.org/article/9e333338bec145c38bf7194e4f25b455","host_type":"repository"},{"url":"https://dialnet.unirioja.es/descarga/articulo/9714355.pdf","host_type":"repository"}],"fields_of_study":["Advanced Neural Network Applications","Advanced Image and Video Retrieval Techniques","Neural Networks and Applications","Computer Science"],"mesh_terms":[],"keywords":["Data parallelism","Parallelism (grammar)","Computer science","Instruction-level parallelism","Task parallelism","Implicit parallelism","Parallel computing","Artificial intelligence","Deep learning","Artificial neural network"],"sdg_mappings":[{"sdg_number":0,"sdg_label":"Quality Education"}],"linked_datasets":[],"clinical_trials":[],"software_tools":[],"database_accessions":[],"source":"live","citation_network_status":"fetched"},"created_at":"2026-07-20T17:22:27.851173Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}