Bug-Reproducing Test cases (BRTs) are essential for developers to understand and fix bugs. BRTs are also necessary in various software engineering research areas such as fault localization, automated program repair and crash reproduction. However, it is extremely tedious and time consuming to collect them manually. Consequently, most of the research results in those areas are derived from small numbers of (8 to 119) bugs.

In this paper I propose an automatic approach of extracting BRTs from software repositories. My approach automatically finds, downloads and builds meaningful revisions. Then my algorithm automatically finds BRTs by using my definition of bugs and patches. Found BRTs are packaged with related data into readily usable forms. Using my approach, I collected 1610...[ Read more ]

Bug-Reproducing Test cases (BRTs) are essential for developers to understand and fix bugs. BRTs are also necessary in various software engineering research areas such as fault localization, automated program repair and crash reproduction. However, it is extremely tedious and time consuming to collect them manually. Consequently, most of the research results in those areas are derived from small numbers of (8 to 119) bugs.

In this paper I propose an automatic approach of extracting BRTs from software repositories. My approach automatically finds, downloads and builds meaningful revisions. Then my algorithm automatically finds BRTs by using my definition of bugs and patches. Found BRTs are packaged with related data into readily usable forms. Using my approach, I collected 1610 real-world BRTs from 891 bugs of 4 open source projects. I manually verified the collected bugs, and 72% of them were found to be real bugs. To further evaluate the usefulness of collected BRTs, I ran a program repair algorithm (GenProg) on them. A large number of BRTs allowed thorough analysis of the algorithm. I could identify a weakness of the algorithm, which had been presumed but not been able to be proven before because only a small number of BRTs had been available: many of its patches (71.4%) were nonsensical. Moreover, it was found that bugs were concentrated: top 19% of test classes revealed 50% of total bugs, 44% of runtime errors was solely due to null pointer exception, and 91% of runtime errors belonged to 8 error types. I believe my BRT collection and BRT collecting algorithm will benefit software engineering researchers in various areas, and allow them to run large-scale experiments.