Automated model-based performance analysis of software product lines under uncertainty

Context: A Software Product Line (SPL) can express the variability of a system through the specification of configuration options. Evaluating performance characteristics, such as the system response time and resource utilization, of a software product is challenging, even more so in the presence of uncertain values of the attributes. Objective: The goal of this paper is to automate the generation of performance models for software products derived from the feature model by selection heuristics. We aim at obtaining model-based predictive results to quantify the correlation between the features, along with their uncertainties, and the system performance. This way, software engineers can be informed on the performance characteristics before implementing the system. Method: We propose a tool-supported framework that, starting from a feature model annotated with performance-related characteristics, derives Queueing Network (QN) performance models for all the products of the SPL. Model-based performance analysis is carried out on the models obtained by selecting the products that show the maximum and minimum performance-based costs. Results: We applied our approach to almost seven thousand feature models including more than one hundred and seventy features. The generation of QN models is automatically performed in much less than one second, whereas their model-based performance analysis embeds simulation delays and requires about six minutes on average. Conclusion: The experimental results confirm that our approach can be effective on a variety of systems for which software engineers may be provided with early insights on the system performance in reasonably short times. Software engineers are supported in the task of understanding the performance bounds that may encounter when (de)selecting different configuration options, along with their uncertainties.