Computational Approaches for App-to-App Retrieval and Design Consistency Check

"Extracting semantic representations from mobile user interfaces (UI) for designers' decision-making processes has shown promise as an effective computational design support tool. Current methods rely on machine learning models trained on small mobile UI datasets to extract semantic vectors, using screenshot-to-screenshot comparisons to retrieve similar-looking UIs given query screenshots. However, the usability of these methods is limited because they are often not open-sourced and have complex training pipelines, making them difficult for practitioners to follow. Additionally, they are unable to perform screenshot set-to-set (i.e., app-to-app) retrieval. To address these issues, we (1) employ visual models trained on large web-scale images and test their ability to extract a UI representations in a zero-shot manner, and (2) utilize mathematically founded methods to enable app-level analysis. Our experiments demonstrate that our approaches not only outperform existing retrieval models but also enable multiple new applications, such as app-to-app retrieval and design consistency check. "