Zeng Z and Xiong D, Unsupervised and Few-Shot Parsing from Pretrained Language Models (Extended Abstract), IJCAI, 2023. (顶级国际会议)

Abstract: In this article, we propose UPOA, an Unsupervised constituent Parsing model that calculates an Out Association score solely based on the self-attention weight matrix learned in a pretrained language model as the syntactic distance for span segmentation. We further propose an enhanced version, UPIO, which exploits both inside association and outside association scores for estimating the likelihood of a span. Experiments with UPOA and UPIO disclose that the linear projection matrices for the query and key in the self-attention mechanism play an important role in parsing. We therefore extend the unsupervised models to few-shot parsing models (FPOA, FPIO) that use a few annotated trees to learn better linear projection matrices for parsing. Experiments on the Penn Treebank demonstrate that our unsupervised parsing model UPIO achieves results comparable to the state of the art on short sentences (length <= 10). Our few-shot parsing model FPIO trained with only 20 annotated trees outperforms a previous few-shot parsing method trained with 50 annotated trees. Experiments on cross-lingual parsing show that both unsupervised and few-shot parsing methods are better than previous methods on most languages of SPMRL.

摘要：在这篇文章中，提出了一种无监督成分句法分析模型UPOA，它基于预训练语言模型中学习到的自注意力权重矩阵来计算句段间的句法距离，用于句段划分。进一步提出了一个增强版本UPIO，它利用内部关联和外部关联分数来估计句段的可能性。实验表明，自注意力机制中的查询和键的线性投影矩阵在句法分析中起着重要作用。因此，将无监督模型扩展到少样本解析模型（FPOA和FPIO），使用少量注释树来学习更好的线性投影矩阵进行句法分析。在Penn Treebank上的实验表明，无监督解析模型UPIO在短句（长度<=10）上的结果与最新技术相当。少样本解析模型FPIO仅用20个注释树训练就超过了以前用50个注释树训练的少样本解析方法。跨语言解析的实验显示，无监督和少样本解析方法在大多数SPMRL语言上都优于以前的方法。