HuTieuBERT

This repository implements a morpheme-aware Transformer architecture that enhances pretrained encoders with explicit morphological structure for isolating languages.

The model effectively captures compound cohesion and morpheme boundaries that standard Transformers often overlook, while remaining optimized for Vietnamese.

The design is portable to other isolating languages like Mandarin Chinese, consistently improving performance on syntactic tasks such as POS tagging and Named Entity Recognition (NER).

Adaptive Boundary-Token Fusion

Subword alignment syncs word-structure tags (BMES) with the smaller sub-units of text created during tokenization.

• B (Begin): Marks the first syllable or character of a multi-syllable word.
• M (Middle): Applied to the internal syllables or characters of a multi-syllable word.
• E (End): Marks the final syllable or character of a multi-syllable word.
• S (Single): Used for standalone words that consist of only one syllable or character.

By expanding these tags, the model ensures that multi-syllable words keep their linguistic meaning even when broken into pieces.

Adaptive Interpolation Layer

• Blending Information: Combines standard word data with boundary information that marks where words start and end.
• Smart Filtering: A gate automatically decides how much boundary information is needed for each word based on its context.
• Rich Representation: The result is a more complete representation of the text that respects the natural boundaries of the language.

Morpheme-Aware Attention Bias

This module guides the model's focus by injecting a fixed structural prior into the early self-attention layers. It ensures that the attention mass respects the natural boundaries of compounds rather than spreading too thin across unrelated words.

The bias is controlled by a matrix using four key parameters to modulate relationship scores:

• Alpha (α): Strengthens focus between tokens that belong to the same compound phrase.
• Beta (β): Penalizes or mutes attention between tokens that belong to different compounds.
• Gamma (γ): Highlights and adjusts the importance of single-word units.
• Delta (δ): Controls the strength of a token's focus on itself, or self-attention bias.

By reweighting these connections, the model maintains a stable internal geometry while gaining a clearer understanding of linguistic structure. This method not only work with Vietnamese but also other isolating languages like Mandarin Chinese and Thai.

We compare HuTieuBERT against widely used pretrained BERT-based encoders, including PhoBERT-base, XLM-RoBERTa-base, and mBERT. Although ViWordFormer is closely related in spirit, it is not included in the main comparison due to substantial architectural and pretraining differences: ViWordFormer is a segmentation-free model trained from scratch, whereas HuTieuBERT augments pretrained encoders.

All reported results are mean ± standard deviation across five independent trials with shuffled training data.

HuTieuBERT achieves the best POS results across UD_VTB, VnDT, and VLSP_13, and also leads on PhoNER and VLSP_16. On VietMed, XLM-RoBERTa performs best, likely due to the dataset's domain-specific English medical terminology and pronounced class imbalance. For sentence-level classification, HuTieuBERT leads on sentiment, topic classification, and toxic detection, while remaining competitive on constructiveness detection.

Early-layer injection at layers 1-2 generally performs best on POS tasks. Removing either structural attention bias or BMES fusion degrades performance, supporting the contribution of both components.

On ULNER, MAChineseBERT improves F1 over ChineseBERT by 1.31 percentage points. On GSDS, it slightly trails ChineseBERT in accuracy but improves macro-F1 by 5.00 percentage points, suggesting more balanced predictions across POS categories.

HuTieuBERT depends on VnCoreNLP for Vietnamese word segmentation. Before running any tokenizer or model example in this repository, please download and set up VnCoreNLP following the official repository: vncorenlp/VnCoreNLP.

pip install py_vncorenlp

import py_vncorenlp

py_vncorenlp.download_model(save_dir="/absolute/path/to/vncorenlp")

Example Usage

import torch
import torch.nn as nn
from transformers.models.roberta.modeling_roberta import RobertaModel, RobertaEncoder, RobertaLayer
from transformers import RobertaConfig

from model.tokenizer import MorphemeAwareTokenizer
from model.embeddings import BoundaryAwareEmbeddings
from model.model import MorphemeAwareRobertaModel, MorphemeAwareRobertaForSequenceClassification

tokenizer = MorphemeAwareTokenizer.from_pretrained(
    "ducanhdinh/HuTieuBert",
    vncorenlp_dir="/content/vncorenlp",
    return_tensors="pt"
)

config = RobertaConfig.from_pretrained("ducanhdinh/HuTieuBert")

# Applied Structural Bias Matrix to Layer 1 and 2 full 12 heads
target_heads = {
    1: list(range(config.num_attention_heads)),
    2: list(range(config.num_attention_heads)),
}

model = MorphemeAwareRobertaForSequenceClassification(
    config,
    num_labels=label_num,
    target_heads=target_heads,
    alpha=0.5,
    beta=-0.3,
    gamma=0.0,
    delta=0.0,
)

model.roberta = MorphemeAwareRobertaModel.from_pretrained(
    "ducanhdinh/HuTieuBert",
    target_heads=target_heads,
    alpha=0.5,
    beta=-0.3,
    gamma=0.0,
    delta=0.0,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

If you find this work useful, please consider citing our paper:

@inproceedings{dinh-etal-2026-morphology,
    title = "When Morphology Hides in Plain Sight: Breaking the Isolation in {V}ietnamese and Beyond",
    author = "Dinh, Anh Trac Duc  and
      Vo, Khang Hoang Nhat  and
      Ta, Tai Tien  and
      Doan, Vinh Cong  and
      Quan, Tho",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.472/",
    pages = "10377--10392",
    ISBN = "979-8-89176-390-6",
    abstract = "In isolating languages such as Vietnamese, core morphological structure is encoded not by inflection but by the composition and ordering of monosyllabic morphemes, yet standard Transformer encoders largely overlook this signal. We introduce HuTieuBERT, a morpheme-aware Transformer that augments a pretrained Vietnamese encoder with two lightweight inductive biases: (i) Adaptive Boundary-Token Fusion, which integrates BMES-based morpheme boundary embeddings into token representations via a learnable gate, and (ii) a Morpheme-Aware Attention Bias, which injects a fixed structural attention matrix into early self-attention layers while minimally perturbing the pretrained attention geometry. Across a suite of Vietnamese POS, NER, and sentence-level classification benchmarks, HuTieuBERT consistently outperforms strong baselines, with the largest gains on syntactic tasks. Hyperparameter ablations show a broad regime in which structural biases improve accuracy without destabilizing representations. Applying the same design to ChineseBERT (Chinese-BERT-wwm) yields MAChineseBERT, which improves $F_{1}$ and produces more balanced tag distributions on Chinese POS and NER, suggesting that explicit morpheme-aware attention is a portable and effective strategy for modeling isolating languages."
}

This work makes use of VnCoreNLP - a Vietnamese natural language processing toolkit.