VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications
Abstract
Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.
Community
We introduce VisualArtifactSpec 1.0 (VAS), a benchmark for evaluating agent-authored visual artifacts.
VAS includes 211 creation and modification tasks across 4 artifact families:
📊 Presentations: creating decks from scratch, working with templates, and editing existing slides.
🎨 Static graphics: posters, infographics, banners, and multi-image series.
🎬 Animations and videos: animated graphics, animated videos, and video editing.
🌐 Web interfaces: content browsing, data exploration, interactive tools, and workflows.
We evaluate two aspects separately:
① Task Fulfillment: Does the artifact satisfy explicit task requirements?
② Artifact Performance: How well do its content, organization, visuals, and delivery serve the intended context?
Across nine model–harness configurations, we find that meeting requirements does not necessarily make an artifact suitable for its intended context. For example, science cards may contain all the required information but differ substantially in visual hierarchy and readability on a phone.
We welcome feedback on the benchmark, evaluation protocol, and agent results!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models (2026)
- BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References (2026)
- EvoCowork: A Benchmark for Real-World Office Tasks (2026)
- Can Image Agents Complete Visual Commissions? A Benchmark and Modular Framework (2026)
- Pixels or Code? Benchmarking Instruction-Guided Visual Artifact Operation (2026)
- Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports (2026)
- JoyFrontBench: Evaluating Front-End Coding Agents through Goal-Oriented Acceptance Contracts (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.21382 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper