Evaluation Framework for Autonomous Systems: The Case of Programmable Electronic Medical Systems

This paper proposes an evaluation framework for autonomous systems, called LENS. It is an instrument to make an assessment of a system through the lens of abilities related to adaptation and smartness. The assessment can then help engineers understand in which direction it is worth investing to make their system smarter. It also helps to identify possible improvement directions and to plan for concrete activities. Finally, it helps to make a re-assessment when the improvement has been performed in order to check whether the activity plan has been accomplished. Given the high variability in the various domains in which autonomous systems are and can be used, LENS is defined in abstract terms and instantiated to a specific and important class of medical devices, i.e., Programmable Electronic Medical Systems (PEMS). The instantiation, called LENSPEMS, is validated in terms of applicability, i.e., how it is applicable to real PEMS, generalizability, i.e., to what extent LENSPEMS is generalizable to the PEMS class of systems, and usefulness, i.e., how it is useful in making an assessment and identifying possible directions of improvement towards smartness.