Using Metamorphic Relations to Verify and Enhance Artcode Classification
  Software testing is often hindered where it is impossible or impractical to
determine the correctness of the behaviour or output of the software under test
(SUT), a situation known as the oracle problem. An example of an area facing
the oracle problem is automatic image classification, using machine learning to
classify an input image as one of a set of predefined classes. An approach to
software testing that alleviates the oracle problem is metamorphic testing
(MT). While traditional software testing examines the correctness of individual
test cases, MT instead examines the relations amongst multiple executions of
test cases and their outputs. These relations are called metamorphic relations
(MRs): if an MR is found to be violated, then a fault must exist in the SUT.
This paper examines the problem of classifying images containing visually
hidden markers called Artcodes, and applies MT to verify and enhance the
trained classifiers. This paper further examines two MRs, Separation and
Occlusion, and reports on their capability in verifying the image
classification using one-way analysis of variance (ANOVA) in conjunction with
three other statistical analysis methods: t-test (for unequal variances),
Kruskal-Wallis test, and Dunnett's test. In addition to our previously-studied
classifier, that used Random Forests, we introduce a new classifier that uses a
support vector machine, and present its MR-augmented version. Experimental
evaluations across a number of performance metrics show that the augmented
classifiers can achieve better performance than non-augmented classifiers. This
paper also analyses how the enhanced performance is obtained.
