Hasnain says:

“To expand on the above successes toward these next generation capabilities, the community of practitioners and researchers working in this topic would benefit from common benchmarks to help move the field towards practical engineering tasks. So far, benchmarks have been focused mostly around code generation (e.g., HumanEval). In an enterprise setting, however, benchmarks for a wider range of tasks could be particularly valuable, e.g., code migrations and production debugging. Some benchmarks, such as one for bug resolution (e.g., SWEBench), and prototypes targeting those benchmarks (e.g., from Cognition AI) have been published. We encourage the community to come together to suggest more benchmarks to span a wider range of software engineering tasks.”

Posted on 2024-06-07T03:22:19+0000