Text-Grounded LLM-Assisted Design Rationale Interfaces: Turning Advertising Layout Metadata into Explainable UI/UX Decision Cards
DOI:
https://doi.org/10.51903/ijgd.v3i1.3713Keywords:
UI/UX design, explainable AI, advertising layouts, OpenCOLE, LLM critiqueAbstract
Automatic graphic design generation has progressed from template retrieval to layered generative pipelines, but designers still need explainable review support that links generated layouts to practical UI/UX criteria. This paper presents a text-grounded, LLM-assisted rationale-card interface for advertising layouts using OpenCOLE annotations, including intention, description, keywords, background caption, object caption, heading, and subheading. The interface converts these fields into structured decision cards covering visual hierarchy, white space, call-to-action (CTA) prominence, brand tone, layout balance, and risk warning. An experimental evaluation was conducted on all 23,419 rows from the OpenCOLE train, validation, and test splits. Because expert UI critique labels are unavailable, the tasks are evaluated as weak-label schema-recovery tasks rather than professional design-quality judgments. A linear SVM achieved 0.888 accuracy and 0.872 macro-F1 for nine-way design-purpose recovery. The best CTA model reached 0.817 accuracy, 0.801 macro-F1, and 0.884 quadratic weighted kappa, while text-based layout-proxy prediction reached 0.734 accuracy and 0.706 macro-F1. Structured cards improved required-dimension coverage from 4.21 to 5.92 out of 6 and field-grounding from 0.681 to 0.918. The findings show that OpenCOLE text fields can support inspectable first-pass design review without replacing expert judgment or direct visual-layout evaluation.
References
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). Association for Computing Machinery. https://doi.org/10.1145/3290605.3300233
Burge, J. E. (2008). Design rationale: Researching under uncertainty. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 22(4), 311-324. https://doi.org/10.1017/S0890060408000235
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017). Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 845-854). Association for Computing Machinery. https://doi.org/10.1145/3126594.3126651
Duan, P., Warner, J., Li, Y., & Hartmann, B. (2024). Generating automatic feedback on UI mockups with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. https://doi.org/10.1145/3613904.3642782
Gupta, K., Lazarow, J., Achille, A., Davis, L. S., Mahadevan, V., & Shrivastava, A. (2021). LayoutTransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 1004-1014).
Inoue, N., Kikuchi, K., Simo-Serra, E., Otani, M., & Yamaguchi, K. (2023). LayoutDM: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10167-10176).
Inoue, N., Masui, K., Shimoda, W., & Yamaguchi, K. (2024). OpenCOLE: Towards reproducible automatic graphic design generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 8131-8135).
Jason Kuhn, Yushan Chen, & Evelyn Chan. (2024). AI-Driven Mobile UI Pattern Recognition and Design Topic Mining on RICO: Semantic Clustering and Screenshot-Based Topic Classification. Journal of Advanced Computing Systems , 4(5), 67-83. https://doi.org/10.69987/JACS.2024.40506
Jia, P., Li, C., Yuan, Y., Liu, Z., Shen, Y., Chen, B., Chen, X., Zheng, Y., Chen, D., Li, J., Xie, X., Zhang, S., & Guo, B. (2024). COLE: A hierarchical generation framework for multi-layered and editable graphic design. arXiv. https://arxiv.org/abs/2311.16974
Kikuchi, K., Simo-Serra, E., Otani, M., & Yamaguchi, K. (2021). Constrained graphic layout generation via latent optimization. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 88-96). Association for Computing Machinery. https://doi.org/10.1145/3474085.3475497
Kong, X., Jiang, L., Chang, H., Zhang, H., Hao, Y., Gong, H., & Essa, I. (2022). BLT: Bidirectional layout transformer for controllable layout generation. In European Conference on Computer Vision (pp. 474-490). Springer. https://doi.org/10.1007/978-3-031-19790-1_29
Kunz, W., & Rittel, H. W. J. (1970). Issues as elements of information systems (Working Paper No. 131). Center for Planning and Development Research, University of California, Berkeley.
Lee, J. (1991). What's in design rationale? Human-Computer Interaction, 6(3-4), 251-280. https://doi.org/10.1207/s15327051hci0603&4_3
Li, J., Yang, J., Hertzmann, A., Zhang, J., & Xu, T. (2019). LayoutGAN: Generating graphic layouts with wireframe discriminators. In International Conference on Learning Representations.
Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-15). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376590
Liao, Q. V., Pribic, M., Han, J., Miller, S., & Sow, D. (2021). Question-driven design process for explainable AI user experiences. arXiv. https://arxiv.org/abs/2104.03483
Moran, T. P., & Carroll, J. M. (Eds.). (1996). Design rationale: Concepts, techniques, and use. Lawrence Erlbaum Associates.
Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 152-158). Association for Computing Machinery. https://doi.org/10.1145/191666.191729
O'Donovan, P., Agarwala, A., & Hertzmann, A. (2015). DesignScape: Design with interactive layout suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1221-1224). Association for Computing Machinery. https://doi.org/10.1145/2702123.2702149
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135-1144). Association for Computing Machinery. https://doi.org/10.1145/2939672.2939778
Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe, and trustworthy. International Journal of Human-Computer Interaction, 36(6), 495-504. https://doi.org/10.1080/10447318.2020.1741118
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019). Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-15). Association for Computing Machinery. https://doi.org/10.1145/3290605.3300831
Yamaguchi, K. (2021). CanvasVAE: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5481-5489).
Yushan Chen, & Evelyn Chan. (2023). Multimodal UI Representation Learning: Ablation of Screenshot, Wireframe, and View-Hierarchy Proxies on an Uploaded 168-Screen Dataset. Journal of Advanced Computing Systems , 3(1), 1-15. https://doi.org/10.69987/JACS.2023.30101
Zhang, H. (2025). LLM-Driven CI Failure Diagnosis and Automated Repair: From GitHub Actions Logs to Patch Recommendation. Journal of Technology Informatics and Engineering, 4(1), 190-214. https://doi.org/10.51903/jtie.v4i1.484
Zheng, X., Qiao, X., Cao, Y., & Lau, R. W. H. (2019). Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics, 38(4), Article 133. https://doi.org/10.1145/3306346.3322971
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Qiyou Wu, Siquan Meng, Jacob Zhao
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
5.png)
