publications

2024

1. Preprint

   Instruction Fine-Tuning: Does Prompt Loss Matter?

   Mathew Huerta-Enochian

   2024

   Abs arXiv

   We present a study analyzing the effects of prompt loss weighting (PLW) on supervised instruction fine-tuning. We recreated Stanford’s Alpaca experiment with both LLaMA 1 and LLaMA 2 and multiple instruction datasets. We found that performance of models fine-tuned on our short-completion dataset had a statistically significant negative quadratic relationship with PLW, but performance of models fine-tuned on medium- and long-completion data did not show any relationship with PLW. I.e., prompt loss can be safely ignored for many datasets. For short-completion data, small values (0.01-0.1) of PLW were optimal for multiple-choice and short-generation tasks while large values (  1.0) of PLW were optimal for long-generation tasks. We concluded that low non-zero PLW encourages models to not diverge from pre-trained model weights during training and high PLW reduces overfitting. Finally, we present a rough guide for selecting PLW values based on the completion-prompt length ratio of fine-tuning data.

2. SignLang

   Shedding Light on the Underexplored: Tackling the Minor Sign Language Research Topics

   Jung-Ho Kim, Changyong Ko, Mathew Huerta-Enochian, and 1 more author

   In Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources , May 2024

   Abs

   In the past decade, sign language research has achieved remarkable results alongside the advancements in deep learning. However, there is a disconnect between the outcomes of these research efforts and the actual use of sign language by signers. In this position paper, we reviewed sign language papers related to deep learning published in the last ten years to explore reasons for this gap. We found many areas of research that are still underdeveloped, despite their linguistic importance. Based on an analysis of known corpora and methodologies, we identified the reasons for the lack of progress in these areas and propose directions for future research efforts.

3. LREC-COLING

   SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation

   Jung-Ho Kim*, Mathew John Huerta-Enochian*, Changyong Ko, and 1 more author

   In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , May 2024

   Abs

   Sign languages are multi-channel languages that communicate information through not just the hands (manual signals) but also facial expressions and upper body movements (non-manual signals). However, since automatic sign language translation is usually performed by generating a single sequence of glosses, researchers eschew non-manual and co-occurring manual signals in favor of a simplified list of manual glosses. This can lead to significant information loss and ambiguity. In this paper, we introduce a new task named multi-channel sign language translation (MCSLT) and present a novel metric, SignBLEU, designed to capture multiple signal channels. We validated SignBLEU on a system-level task using three sign language corpora with varied linguistic structures and transcription methodologies and examined its correlation with human judgment through two segment-level tasks. We found that SignBLEU consistently correlates better with human judgment than competing metrics. To facilitate further MCSLT research, we report benchmark scores for the three sign language corpora and release the source code for SignBLEU at https://github.com/eq4all-projects/SignBLEU.

2023

1. KAIC

   A Study on Korean to Korean Sign Language Translation using Large Language Models

   Jung-Ho Kim, Changyong Ko, Mathew Huerta-Enochian, and 2 more authors

   In Korea Artificial Intelligence Conference , May 2023
2. SLTAT

   Sign Language Avatar Animation Search: An Ani2Ani Search Application

   Mathew Huerta-Enochian, and Changyong Ko

   In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) , May 2023

   Abs Poster

   To improve sign language animation asset management, we developed a general animation search system supporting multiple input modalities. The system reads animations as pose sequences, embeds them into fixed-length representation vectors using engineered or learned features, and scores animation similarity using distance in the embedding space. We present an overview of the system, use-case scenarios, and feedback from deaf users.

2022

1. SLTAT

   KoSign Sign Language Translation Project: Introducing the NIASL2021 Dataset

   Mathew Huerta-Enochian, Du Hui Lee, Hye Jin Myung, and 2 more authors

   In Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives , May 2022

   Abs Poster

   We introduce a new sign language production (SLP) and sign language translation (SLT) dataset, NIASL2021, consisting of 201,026 Korean-KSL data pairs. KSL translations of Korean source texts are represented in three formats: video recordings, keypoint position data, and time-aligned gloss annotations for each hand (using a 7,989 sign vocabulary) and for eight different non-manual signals (NMS). We evaluated our sign language elicitation methodology and found that text-based prompting had a negative effect on translation quality in terms of naturalness and comprehension. We recommend distilling text into a visual medium before translating into sign language or adding a prompt-blind review step to text-based translation methodologies