LLM-as-Design-Critic: Aligning AI-Generated UI Feedback with Human Graphic Design Judgment

Abstract

This paper evaluates whether AI-authored mobile user-interface critiques align with human graphic design judgment. The study uses the public UICrit CSV derived from RICO mobile screens, containing 2,981 annotator rows, 1,000 distinct UI screens, 11,344 source-indexed design critiques, normalized critique bounding boxes, and ratings for aesthetics, learnability, efficiency, usability, and overall design quality. We conducted a full reproducible empirical evaluation rather than reporting illustrative results. Seven models were compared on a group-disjoint split by RICO screen ID: a mean baseline, task-text TF-IDF Ridge, human-critique TF-IDF Ridge, LLM-critique TF-IDF Ridge, all-critique TF-IDF Ridge, topic-and-region Ridge, and a fused text-topic-region Ridge model. We also measured human-LLM critique alignment using TF-IDF cosine, character n-gram cosine, ROUGE-L F1, unigram F1, topic Jaccard, and best-match bounding-box IoU. The fused model achieved the strongest overall design-quality prediction on the held-out test set (MAE = 0.613, RMSE = 0.805, Spearman = 0.575), improving over the mean baseline MAE of 0.779. Human critiques alone were highly predictive (design-quality Spearman = 0.556), whereas LLM-inclusive critiques alone were much weaker (Spearman = 0.194). Human-LLM semantic alignment was low for exclusive human versus exclusive LLM comments (mean TF-IDF cosine = 0.046) and substantially higher when comments tagged as both were included (mean TF-IDF cosine = 0.390). Results show that design critiques encode measurable aesthetic and usability judgment, but LLM critiques still differ from human critique priorities unless shared comments and region evidence are incorporated