Abstract

Retrieving charts from a large corpus is a fundamental task that can benefit numerous applications such as visualization recommendations.The retrieved results are expected to conform to both explicit visual attributes (e.g., chart type, colormap) and implicit user intents (e.g., design style, context information) that vary upon application scenarios. However, existing example-based chart retrieval methods are built upon non-decoupled and low-level visual features that are hard to interpret, while definition-based ones are constrained to pre-defined attributes that are hard to extend. In this work, we propose a new framework, namely WYTIWYR (What-You-Think-Is-What-You-Retrieve), that integrates user intents into the chart retrieval process. The framework consists of two stages: first, the Annotation stage disentangles the visual attributes within the bitmap query chart; and second, the Retrieval stage embeds the user’s intent with customized text prompt as well as query chart, to recall targeted retrieval result. We develop a prototype WYTIWYR system leveraging a contrastive language-image pre-training (CLIP) model to achieve zero-shot classification, and test the prototype on a large corpus with charts crawled from the Internet. Quantitative experiments, case studies, and qualitative interviews are conducted. The results demonstrate the usability and effectiveness of our proposed framework.

Schematic Diagram of the Proposed Method

The architecture of our model is presented below:

The annotation process is presented below:

The retrieval process is presented below:

For detail information, please refer to our paper.

Downloads

Full paper: click here

Project code: click here

Reference

@inproceedings{xiao2023WYTIWYR,
  title={WYTIWYR: A User Intent-Aware Framework with Multi-modal Inputs for Visualization Retrieval},
  author={Xiao, Shi-Shi and Hou, Yihan and Jin, Cheng and Zeng, Wei},
  booktitle={2023 Eurographics Conference on Visualization EuroVis},
  year={2023}
}