Benchmarking LLMs in File System Design and Implementation
The Bench framework evaluates LLM performance in file system development with 505 tasks, highlighting capabilities and failure mitigation.
Large Language Models (LLMs) are fundamentally transforming computer system research and development. Understanding their capabilities and limitations in file system development is crucial. This study introduces Bench, an LLM benchmarking framework tailored for file system-specific tasks, featuring six task types: basic understanding, implementation, performance modeling, debugging, optimization, and new feature development. With 505 tasks, the empirical study evaluates both open-source and proprietary LLMs, revealing model efficiency and strategies to mitigate failures in file system tasks.