• 中国期刊全文数据库
  • 中国学术期刊综合评价数据库
  • 中国科技论文与引文数据库
  • 中国核心期刊(遴选)数据库
杨璐, 钱艺, 文益民. 基于场景中物体位置关系的图像描述方法[J]. 桂林电子科技大学学报, xxxx, x(x): 1-8. DOI: 10.3969/1673-808X.202360
引用本文: 杨璐, 钱艺, 文益民. 基于场景中物体位置关系的图像描述方法[J]. 桂林电子科技大学学报, xxxx, x(x): 1-8. DOI: 10.3969/1673-808X.202360
YANG Lu, QIAN Yi, WEN Yimin. Learning positional relationship for Image Captioning[J]. Journal of Guilin University of Electronic Technology, xxxx, x(x): 1-8. DOI: 10.3969/1673-808X.202360
Citation: YANG Lu, QIAN Yi, WEN Yimin. Learning positional relationship for Image Captioning[J]. Journal of Guilin University of Electronic Technology, xxxx, x(x): 1-8. DOI: 10.3969/1673-808X.202360

基于场景中物体位置关系的图像描述方法

Learning positional relationship for Image Captioning

  • 摘要: 图像描述旨在将图像内容转化为语言表述,是一个亟待解决且具有挑战性的多模态生成任务。然而,现有方法缺少对图像中隐含位置信息的关注,导致物体位置关系难以得到准确描述。为解决该问题,提出一种基于场景中物体位置关系的图像描述方法。首先,使用图节点特征构建物体关系场景图,随后利用位置关系编码器对节点特征进行初次编码。其次,提出常识词典与推理模块,计算物体间比例失衡程度值,根据该程度值对物体关系节点进行二次编码。再次,设计联合解码器对已编码信息进行处理,通过擦除模块和偏置门控机制进一步优化图中的节点特征。最后,生成该图像对应的文字描述。提出的方法在公开数据集上进行实验验证,在各项评价指标上对比现有方法均有提升,并在CIDEr指标上取得显著效果。该方法源码可在https://gitee.com/ymw12345/PRCO获取。

     

    Abstract: Image captioning aims to transform visual content into language description, which is an urgent and challenging multimodal generation task. Due to the lack of attention to the implicit position information in the most image captioning methods, it is difficult to accurately describe the position relationship of the objects in the image. For solving this problem, the Position Relationship Encoder-Combine Decoder (RPCO) structure is proposed, which focus on and generate the objects positional relationships. A novel Position Relationship-Encoder get started with the object relationship scene graph using node features. Technically, common sense dictionary and reasoning module are created to calculate the degree of imbalance between objects, which are used to perform a secondary encoding of the object relationship nodes. Specifically, the Combine-Decoder is designed to process the encoded information, with an erasing module and bias gate to optimize the node features in the graph. Experiments are conducted on MSCOCO and Visual Genome image captioning dataset, and superior results in comparing to state-of-the-art approaches. More remarkably, PRCO achieves an increases CIDEr performance on Visual Genome testing set. Our code is publicly available on Gitee: https://gitee.com/ymw12345/PRCO.

     

/

返回文章
返回