CreativeSeg: Semantic Segmentation of Creative Sketches

IEEE Transactions on Image Processing

Yixiao Zheng

PRIS Lab, BUPT, China

Kaiyue Pang

SketchX Lab, University of Surrey UK

Ayan Das

SketchX Lab, University of Surrey UK

Dongliang Chang

PRIS Lab, BUPT, China

Yi-Zhe Song

SketchX Lab, University of Surrey UK

Zhanyu Ma

PRIS Lab, BUPT, China


March 18, 2024

Paper Code


The problem of sketch semantic segmentation is far from being solved. Despite existing methods exhibiting near-saturating performances on simple sketches with high recognisability, they suffer serious setbacks when the target sketches are products of an imaginative process with high degree of creativity. We hypothesise that human creativity, being highly individualistic, induces a significant shift in distribution of sketches, leading to poor model generalisation. Such hypothesis, backed by empirical evidences, opens the door for a solution that explicitly disentangles creativity while learning sketch representations. We materialise this by crafting a learnable creativity estimator that assigns a scalar score of creativity to each sketch. It follows that we introduce CreativeSeg, a learning-to-learn framework that leverages the estimator in order to learn creativity-agnostic representation, and eventually the downstream semantic segmentation task. We empirically verify the superiority of CreativeSeg on the recent “Creative Birds” and “Creative Creatures” creative sketch datasets. Through a human study, we further strengthen the case that the learned creativity score does indeed have a positive correlation with the subjective creativity of human.


BibTeX citation:
  author = {Zheng, Yixiao and Pang, Kaiyue and Das, Ayan and Chang,
    Dongliang and Song, Yi-Zhe and Ma, Zhanyu},
  publisher = {IEEE},
  title = {CreativeSeg: {Semantic} {Segmentation} of {Creative}
  journal = {IEEE Transactions on Image Processing},
  pages = {2266 - 2278},
  date = {2024-03-18},
  url = {},
  doi = {10.1109/TIP.2024.3374196},
  langid = {en}
For attribution, please cite this work as:
Zheng, Yixiao, Kaiyue Pang, Ayan Das, Dongliang Chang, Yi-Zhe Song, and Zhanyu Ma. 2024. “CreativeSeg: Semantic Segmentation of Creative Sketches.” IEEE Transactions on Image Processing, March, 2266–78.