GUI Understanding | Yiqiao Jin CS PhD @ Georgia Techhttps://ahren09.github.io/tags/gui-understanding/GUI UnderstandingHugo Blox Builder (https://hugoblox.com)en-usWed, 30 Apr 2025 00:00:00 +0000https://ahren09.github.io/media/icon_hu_eee6347cbdb2cc3f.pngGUI Understandinghttps://ahren09.github.io/tags/gui-understanding/ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Predictionhttps://ahren09.github.io/publication/webconf25_screenllm/Wed, 30 Apr 2025 00:00:00 +0000https://ahren09.github.io/publication/webconf25_screenllm/<h2 id="abstract">Abstract</h2> <p>We introduce ScreenLLM, a specialized multimodal LLM for Graphical User Interface (GUI) understanding and action prediction. ScreenLLM proposes a stateful screen schema that represents dynamic user sessions as compact, time-aware textual summaries, and a high-efficiency key-frame extraction method based on second-order pixel changes to isolate significant UI transitions.</p> <h2 id="links">Links</h2> <ul> <li> </li> </ul>