Index of /rozprawy2/10886

Pełen tekst

(1)AGH UNIVERSITY OF SCIENCE AND TECHNOLOGY. Faculty of Computer Science, Electronics and Telecommunication. Adaptation of Service-Oriented Systems Based on Reinforcement Learning. Ph.D. Dissertation. Author: mgr inz. ˙ Kornel Skałkowski. Supervisor: prof. dr hab. inz. ˙ Krzysztof Zielinski ´. Kraków, September 17, 2014.

(2) AKADEMIA GÓRNICZO-HUTNICZA. Wydział Informatyki, Elektroniki i Telekomunikacji. Adaptacja systemów zorientowanych na usługi realizowana za pomoca˛ uczenia ze wzmocnieniem. Rozprawa doktorska. Autor: mgr inz. ˙ Kornel Skałkowski. Promotor: prof. dr hab. inz. ˙ Krzysztof Zielinski ´. Kraków, September 17, 2014.

(3) Acknowledgments This dissertation has never been possible without help and support of many people. First of all, I would like to express my deepest appreciation to my supervisor, Prof. Krzysztof Zielinski, ´ who continually assisted and supported me during the research presented in this thesis. I would also like to cordially thank all my colleagues from the Distributed Systems Research Group at the Department of Computer Science of AGH University of Science and Technology for endless discussions and their incredible patience. My gratitude is also addressed to colleagues from the other research groups at the Department of Computer Science.. I would like to express my special thanks to Prof. Andrew Ng from the AI Lab at the Stanford University for his Machine Learning class on the Coursera platform and the course handouts published at the Stanford web pages. Participation in this course gave me solid fundamentals in the area of machine learning and strongly influenced the dissertation thesis. Without that course, this dissertation would have never achieved such a scientific level.. Quality of this dissertation would be much poorer if it was not supported by multiple projects and scholarships. I acknowledge that during the work on my thesis I have been a scholarship fellow of the "Doctus - Małopolski fundusz stypendialny dla doktorantów" project co-funded by EU funds within European Social Fund. The research presented in this thesis was also partially supported by the European Regional Development Fund program no. POIG.01.03.01-00-008/08 and the European Community’s Seventh Framework Programme (FP7/2007-2013) grant agreement no. 247950.. Last but not least, I like to deeply thank my family for their understanding and endless patience. I address special gratitude to my mother Kornelia and my father Tomasz for their strong emotional support during realization of this thesis. I also thank all my friends and relatives who have not allowed me to forget that there are other things apart from the work in the world. The deepest thanks, however, I address to the One who devised all this complexity.. i.

(4) Contents List of Tables. v. List of Figures 1. 2. 3. vii. Introduction 1.1 Adaptation in modern IT systems . . . . . . 1.2 Machine learning in autonomic computing 1.3 Thesis statement . . . . . . . . . . . . . . . . 1.4 Research contribution . . . . . . . . . . . . . 1.5 Dissertation organization . . . . . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 1 2 3 5 5 6. Background and related work 2.1 Service-Oriented Architecture (SOA) . . . . . . . . . . 2.1.1 The SOA fundamentals . . . . . . . . . . . . . 2.1.2 The S3 model . . . . . . . . . . . . . . . . . . . 2.1.3 The Adaptive SOA architecture . . . . . . . . . 2.2 Autonomic Computing (AC) . . . . . . . . . . . . . . 2.2.1 The MAPE-k pattern . . . . . . . . . . . . . . . 2.2.2 Goal-driven adaptation . . . . . . . . . . . . . 2.2.3 Relationship with the control theory . . . . . . 2.2.4 AC in enterprise systems . . . . . . . . . . . . 2.3 Online machine learning . . . . . . . . . . . . . . . . . 2.3.1 Reinforcement learning (RL) . . . . . . . . . . 2.3.2 Online clustering . . . . . . . . . . . . . . . . . 2.4 Existing approaches to the Adaptive SOA architecture 2.5 Existing approaches to AC based on machine learning 2.5.1 Reinforcement Learning in AC . . . . . . . . . 2.5.2 Online clustering in AC . . . . . . . . . . . . . 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. 7 7 8 8 10 11 12 13 15 16 17 18 19 21 24 24 26 26. The concept of the RL-based Adaptive SOA architecture 3.1 Assumptions regarding the monitoring and execution interfaces . . . . . 3.2 Issues in adaptation of the SOA systems . . . . . . . . . . . . . . . . . . . 3.3 Formulation of the adaptation process . . . . . . . . . . . . . . . . . . . .. 30 32 37 38. ii. . . . . .. . . . . .. . . . . .. . . . . .. . . . . ..

(5) CONTENTS. 3.4 3.5 3.6. Implementation of the analysis and planning components . . . . . . . . Evaluation function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Implementation of the MAPE-k pattern in the RL-based Adaptive SOA architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 45 47. Selection of online machine learning algorithms 4.1 Data stream clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Existing online clustering algorithms . . . . . . . . . . . . . . . . 4.1.2 DenStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 D-Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 DenStream and D-Stream in the RL-based Adaptive SOA architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Existing types of the RL algorithms . . . . . . . . . . . . . . . . . 4.2.3 Adaptive Gradient Descent . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Q-learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.5 AGD and Q-learning in the RL-based Adaptive SOA architecture 4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 51 52 53 54 57. Implementation of the RL-based Adaptive SOA framework 5.1 The framework architecture . . . . . . . . . . . . . . . . . 5.2 The framework services . . . . . . . . . . . . . . . . . . . 5.2.1 The normalizer service . . . . . . . . . . . . . . . . 5.2.2 The management service . . . . . . . . . . . . . . 5.2.3 The evaluation service . . . . . . . . . . . . . . . . 5.2.4 The decision service . . . . . . . . . . . . . . . . . 5.3 The framework implementation details . . . . . . . . . . 5.4 Computational and memory complexity analysis . . . . . 5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. 73 73 76 77 78 82 83 84 85 88. . . . . . . . . . .. 89 90 92 94 96 96 97 99 106 109 112. 3.7 4. 5. 6. iii. Evaluation of the RL-based Adaptive SOA architecture 6.1 Evaluation methodology . . . . . . . . . . . . . . . . . 6.2 Parametrization of algorithms . . . . . . . . . . . . . . 6.3 First evaluation scenario . . . . . . . . . . . . . . . . . 6.3.1 Testing environment . . . . . . . . . . . . . . . 6.3.2 The monitoring and execution interfaces . . . 6.3.3 Evaluation function . . . . . . . . . . . . . . . 6.3.4 Evaluation results . . . . . . . . . . . . . . . . . 6.3.5 Detailed results for AGD . . . . . . . . . . . . 6.3.6 Detailed results for Q-learning . . . . . . . . . 6.3.7 Conclusions from the first evaluation scenario. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. 48 49. 59 60 60 62 63 67 71 72.

(6) CONTENTS. 6.4. . . . . . . . .. 112 115 117 118 121 123 131 133. Conclusions 7.1 Thesis verification and confirmation . . . . . . . . . . . . . . . . . . . . . 7.2 Novel concepts and limitations . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 134 134 135 136. 6.5 7. Second evaluation scenario . . . . . . . . . . . . . . . . . 6.4.1 Adaptation scenario for TeleDICOM II . . . . . . . 6.4.2 Testing environment . . . . . . . . . . . . . . . . . 6.4.3 The monitoring and execution interfaces . . . . . 6.4.4 The evaluation function . . . . . . . . . . . . . . . 6.4.5 Evaluation results . . . . . . . . . . . . . . . . . . . 6.4.6 Conclusions from the second evaluation scenario Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . .. iv. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. Abbreviations. 138. References. 140. Appendices. 153. A The DenStream and D-Stream algorithms 154 A.1 DenStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 A.2 D-Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 B The Q-learning algorithm’s policies. 161.

(7) List of Tables 2.1 2.2 2.3. Four fundamental self-management properties of autonomic systems. . Limitations of existing approaches to the Adaptive SOA architecture (1). Limitations of existing approaches to Adaptive SOA architecture (2). . .. 11 27 28. 4.1 4.2 4.3. The DenStream and DBSCAN algorithms’ parameters. . . . . . . . . . . The D-Stream algorithm’s parameters. . . . . . . . . . . . . . . . . . . . . An example Q-learning table. . . . . . . . . . . . . . . . . . . . . . . . . .. 57 59 68. 6.1 6.2 6.3 6.4 6.5. The framework evaluation settings. . . . . . . . . . . . . . . . . . . . . . . 91 Adjusted values of the algorithms’ configuration parameters. . . . . . . 93 Management actions available in the first evaluation scenario. . . . . . . 98 Values of the validation indices obtained in the first evaluation scenario. 101 Results of the converged hypothesis functions h1 and h2 for example system states (first evaluation scenario). . . . . . . . . . . . . . . . . . . . 108 The converged q-table of the Q-learning algorithm obtained for SET2 (first evaluation scenario). . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 The converged q-table of the Q-learning algorithm obtained for SET4 (first evaluation scenario). . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 The converged q-table of the Q-learning algorithm obtained for SET8 (first evaluation scenario). . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 The converged q-table of the Q-learning algorithm obtained for SET10 in the first evaluation scenario (the brackets contain values of the counters). 110 Common issues with the example TeleDICOM II configuration. . . . . . 116 Parameters related to the TeleDICOM II adaptation scenario. . . . . . . . 120 Management actions available in the second evaluation scenario. . . . . 122 Values of the validation indices obtained in the second evaluation scenario.125 The converged q-table of the Q-learning algorithm obtained for SET7 in the second evaluation scenario (part 1). . . . . . . . . . . . . . . . . . . . 130 The converged q-table of the Q-learning algorithm obtained for SET7 in the second evaluation scenario (part 2). . . . . . . . . . . . . . . . . . . . 132. 6.6 6.7 6.8 6.9 6.10 6.11 6.12 6.13 6.14 6.15. v.

(8) List of Figures 2.1 2.2 2.3 2.4 2.5. The SOA Solution Stack model (source: [12]). The MAPE-k adaptation loop. . . . . . . . . . The concept of closed-loop control. . . . . . . The main flowchart of the RL algorithms. . . Typical cases in clusters’ evolution. . . . . . .. . . . . .. . . . . .. . . . . .. 9 12 16 18 19. 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9. Dependency of the MAPE-k components from the adaptation goal. . Types of parameters returned by the monitoring interface. . . . . . . Example system states extracted as agglomerations of state vectors. . Reflection of changes in monitored parameters in a clustering result. The idea of assigning actions to clusters. . . . . . . . . . . . . . . . . . An example mapping between clusters and actions. . . . . . . . . . . Evolution of assignment of management actions to states. . . . . . . . Looping of the adaptive manager between suboptimal states. . . . . The RL-based Adaptive SOA architecture concept. . . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. 31 33 39 40 42 43 43 44 48. 4.1 4.2 4.3 4.4. The online and offline clustering phases. . . . . . . . . . . . . . . . . . . . The density-reachability relationship. . . . . . . . . . . . . . . . . . . . . Grid-partitioning of the clustered space used in the D-Stream algorithm. The difference between the discretization-based and dimension-based approaches to the q-table creation. . . . . . . . . . . . . . . . . . . . . . .. 54 57 58. 5.1 5.2 5.3. The RL-based Adaptive SOA framework services. . . . . . . . . . . . . . The RL-based Adaptive SOA framework sequence diagram. . . . . . . . The suspend windows mechanism. . . . . . . . . . . . . . . . . . . . . . .. 74 75 79. 6.1 6.2. The simulated SOA system used in the first evaluation scenario. . . . . . Example course of values of the Service2.APT and Service3.ER parameters in the simulated SOA system. . . . . . . . . . . . . . . . . . . . . . . . . . Results from clustering of the Service2.APT and Service3.ER parameters’ values in selected time moments. . . . . . . . . . . . . . . . . . . . . . . . Example adaptation course obtained for SET1 (first evaluation scenario). Example adaptation course obtained for SET2 (first evaluation scenario). Example adaptation course obtained for SET5 (first evaluation scenario).. 95. 6.3 6.4 6.5 6.6. vi. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 70. 99 100 103 104 105.

(9) LIST OF FIGURES. 6.7 6.8 6.9 6.10 6.11 6.12 6.13 6.14 6.15 6.16 6.17 6.18 6.19 6.20 6.21 6.22 6.23 6.24 6.25. Example adaptation course obtained for SET8 (first evaluation scenario). Example adaptation course obtained for SET9 (first evaluation scenario). A typical evolution of clusters in the first evaluation scenario. . . . . . . Convergence of q-values assigned to the action a1 for different framework settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Convergence of q-values assigned to the action a3 for different framework settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The TeleDICOM II system concept. . . . . . . . . . . . . . . . . . . . . . . An example screenshot from the TeleDICOM II client application. . . . . The TeleDICOM II back-end architecture. . . . . . . . . . . . . . . . . . . The example deployment configuration of TeleDICOM II used for evaluation of the RL-based Adaptive SOA architecture. . . . . . . . . . . . . The hybrid monitoring interface of TeleDICOM II. . . . . . . . . . . . . . Example course of the VM2.CPU, VM2.IGCS_RT and VM4.VCS_RT parameters (no adaptation enabled). . . . . . . . . . . . . . . . . . . . . . . Evolution of clusters formed by the VM2.CPU and VM2.IGCS_RT parameters (no adaptation enabled). . . . . . . . . . . . . . . . . . . . . . . Example course of the VM2.Disk and VM4.I/O parameters (no adaptation enabled). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evolution of the VM2.CPU, VM2.IGCS_RT and VM4.VCS_RT parameters obtained for SET2 in the second evaluation scenario. . . . . . . . . . Evolution of the clusters formed by the VM2.CPU, VM4.VCS_RT and VM2.RS parameters obtained for SET2 . . . . . . . . . . . . . . . . . . . . Evolution of the VM2.Disk, VM4.I/O and VM3.Cost parameters obtained for SET2 in the second evaluation scenario. . . . . . . . . . . . . . . . . . Evolution of the clusters formed by the VM2.Disk, VM4.I/O and VM3.Cost parameters obtained for SET2 . . . . . . . . . . . . . . . . . . . . . . . . . . Evolution of the VM2.CPU, VM2.IGCS_RT and VM4.VCS_RT parameters obtained for SET7 in the second evaluation scenario. . . . . . . . . . Evolution of the VM2.Disk, VM4.I/O and VM3.Cost parameters obtained for SET7 in the second evaluation scenario. . . . . . . . . . . . . . . . . .. vii. 106 107 108 111 111 112 113 114 115 119 123 124 125 127 128 129 130 131 132.

(10) Chapter 1. Introduction This chapter introduces main concepts and ideas related to the thesis subject and its context. First, the background of the thesis is briefly discussed. Then, the chapter presents goals and research contribution of the thesis. The modern economy is largely dependent on Information Technology (IT) systems which play a key role in business, public services or production systems [73, 72]. Rapid development of hardware, software and networking technologies along with their versatile applications have meaningfully contributed to the economy growth in recent years. For example, the authors of [91] point that just Internet applications have generated over 21% of the GDP growth in mature economies for the last 5 years. This strategic alignment between economy and IT [99] poses new requirements for software, which is now expected to be much more flexible and agile in terms of business processes’ specification, introducing incremental system modifications or integration with external systems [102]. The software agility is vital for all stakeholders, as it allows them to quickly respond to preferences of new clients, changes on the market or new law injunctions [23, 114]. Consequently, requirements regarding contemporary IT systems concern not only vast processing and storage capabilities, but also ease of integration with external systems, flexibility in introducing changes and modifications, as well as cost-effective maintenance. The increased agility of IT systems is essential for their alignment with the modern economy expectations, however, it entails also rapid growth of the systems’ complexity. In fact, contemporary IT systems are featured by high geographical distribution and possibility of on-the-fly modifications of components or business processes but, on the other hand, their administration and governance is much more hindered [125, 110]. The traditional approach to IT systems’ administration, which relies on human supervision over the whole system and its infrastructure, is impossible to apply in case of flexible enterprise systems. It is a consequence of the fact that they constitute compositions of distributed software services, provided and managed by different vendors and dynamically bound according to momentary business’ goals or clients’ requests. The requirements posed for modern enterprise systems can be no longer satisfied by tradi1.

(11) CHAPTER 1. INTRODUCTION. 2. tional approaches to software development and maintenance. Alignment of IT systems to the modern economy needs stipulates new concepts and methodologies at all phases of the systems’ lifecycle, i.e. design, development, operation and administration [44].. 1.1. Adaptation in modern IT systems. To facilitate development and maintenance of agile IT systems several new software paradigms and models such as Service-Oriented Architecture (SOA) [41], Cloud Computing [26] or Internet of Things (IoT) [15] have been recently introduced. A common trait of systems built in accordance with these new approaches, which distinguishes them from traditional component-oriented or monolithic systems, is possibility of their dynamic adaptation, i.e. enforcement of modifications during runtime [148]. For example, systems built in accordance with the SOA paradigm constitute compositions of software services provided by different vendors [131]. The services are loosely coupled and bound on-demand, thus it is possible to exchange them during system runtime and control the information flow among them. Such flexibility in choosing the services and their binding allows on seamless modifications of the SOA systems, according to changing business goals or clients’ preferences. In turn, the Cloud Computing model enables elastic and effective assignment of computing and storage resources to different applications, according to their momentary requirements [11]. In the IoT systems, eventually, it is possible to adapt energy consumption, ad-hoc routing protocols or security algorithms [87]. Adaptation of IT systems can be considered from two perspectives: functional and non-functional [109]. The functional adaptation relies on introducing modifications into a system in order to support new use-cases or usage scenarios. This kind of adaptation is strictly associated with the business perspective of a system and, as it was mentioned in the previous paragraph, it is often crucial to accomplish the evolving users’ requirements and preferences. For example, such adaptation can rely on defining of a new business process, which accomplishes a new requirement using existing system services. The non-functional adaptation, in turn, relies on adjusting system behavior in a way which modifies its operation accordingly to the current working environment and the system itself [45, 84], in order to provide a satisfactory Quality of Service (QoS)/Quality of Experience (QoE) [89, 85] level to its end-users. This kind of adaptation is strictly associated with the first one, since the necessity of changes in a system structure or working environment often constitutes a result of modifications introduced into a system by its users. An example of such adaptation can be replication of an overloaded system service or increase in the number of processors assigned to a system server in order to boost the system performance. In contrary to the functional adaptation, the non-functional adaptation is usually supposed to be provided by a system itself, without or with minimal human supervision [90]. It is a consequence of the fact that complex, distributed IT systems such as the service-oriented ones, involve external services and infrastructure providers, what.

(12) CHAPTER 1. INTRODUCTION. 3. makes their coherent administration difficult or even impossible to be made manually. Nonetheless, in the case of such systems, provisioning of such self-adaptation is also particularly challenging from both: scientific and implementation viewpoints [51, 74, 149], as it requires application of specific Artificial Intelligence (AI) methods to selection of proper adaptation actions, based on analysis of system behaviour. These issues have led to emergence of the Autonomic Computing (AC) paradigm [75] and the concept of autonomous, self-adaptive systems. The AC paradigm assumes that a system can modify itself (i.e. is self-adaptive) by executing management actions, which are selected based on analysis of the system condition and fulfillment of a prescribed goal [92]. It means that the adaptation process is goal-driven, i.e. the management actions are adjusted not only to a specific system state, but also to business objectives posed for the system. The AC paradigm defines a general model of the self-adaptive system and the adaptation process, however, it does not specify any particular method of introducing modifications into a system and making decisions about them, since these issues are highly dependent on a specific system architecture. An open problem, therefore, constitutes selection of decision making methods and their adjustment to a specific system type. This problem can be easily resolved for tiny systems, i.e. those which can be modeled as finite automata, nonetheless, it is problematic in complex IT systems such as the service-oriented ones, which have infinite number of possible states and are subjected to continuous evolution. In such a case, the approaches based on direct system modeling [149] cannot be applied, and some AI methods capable to cope with the infinite number of system states and their evolution have to be proposed. A possible group of methods which can address such issues constitute machine learning algorithms.. 1.2. Machine learning in autonomic computing. The problem of autonomic adaptation of IT systems, described in the previous section, can be roughly described as a process of executing appropriate management actions in specific system states, i.e. finding a mapping state → action. The mapping has to be discovered through observation of system evolution, i.e. transition between the states resulting from execution of the actions. Such a problem can be treated as the wellknown classification task, which belongs to the area of machine learning methods. The classification task (also referred to as supervised learning), in general, relies on devising algorithms which given a set of example pairs (point, label), e.g. {(x(1) , y(1) ), (x(2) , y(2) ), ..., (x(n) , y(n) )}, are able to predict correct labels y(i) for new points x(i) , which were not present in the original set [53]. Example classification problems are Optical Character Recognition (OCR), spam classification or face recognition. The problem of making decisions regarding adaptation of a system, posed at the end of the previous section, also belongs to the category of classification problems. Indeed, it relies on finding a correct assignment of available management actions to different states of a system, based on former observations. Obviously, particular system states have to be recognized and properly extracted.

(13) CHAPTER 1. INTRODUCTION. 4. before assignment of any actions to them becomes possible. This is especially important in case of complex systems with unpredictable or infinite number of possible states, such as the SOA ones. Fortunately, this problem can be also categorized as a machine learning task, and solved by clustering or pattern matching algorithms. Clustering (also referred to as unsupervised learning) relies on construction of methods which are able to find a structure in a set of points {x(1) , x(2) , ..., x(n) }, i.e. groups of points which are similar to each other with regard to a specific metric function (e.g. Euclidean or Cosine) [5]. A clustering algorithm’s result constitutes a set of clusters, which are formed by agglomerations of points from the input set. Example clustering problems are market segmentation or search results’ grouping. The problem of recognition of stable system states and distinguishing them from volatile or temporary states can be also solved by a clustering algorithm, since it relies on discovering similar system conditions and treating them as they would represent the same system state. In this way, the clusters resulting from clustering a system state space can be identified with stable states of the system. Both kinds of machine learning algorithms (i.e. clustering and classification) can be divided on two groups: offline algorithms which work on a fixed dataset, and online algorithms working on data streams. The first group of algorithms takes as the input the whole dataset, i.e. a sequence of labeled points in case of classification algorithms or a set of points in case of clustering algorithms. These algorithms are appropriate for offline applications such as images processing or OCR, however, they cannot be applied in applications dealing with online data processing, e.g. recommendation or monitoring systems. For such applications, a separate group of machine learning algorithms operating on data streams has been contrived: Reinforcement Learning (RL) algorithms [71] which constitute classification algorithms working on data streams, and online clustering algorithms also referred to as data stream clustering algorithms [55]. These algorithms do not follow the single-train, multiple-use approach as in the case of offline algorithms, but they are trained in a continual way, through observation of succeeding elements appearing in a data stream. The RL algorithms are pointed in some papers [133, 65] as a possible approach to implementation of decision making components in autonomic IT systems. In fact, numerous successful applications of these algorithms to autonomic management of different kinds of systems have been published [77, 57, 134]. Nonetheless, the RL algorithms are not seamlessly applicable to adaptation of complex IT systems, since state spaces of such systems are usually infinite, high dimensional, and are subjected to continuous evolution. Application of the RL algorithms to making decisions regarding adaptation of complex IT systems requires introduction of additional mechanisms supporting these algorithms by identification of stable system states and tracking their evolution. This issue can be addressed by the online clustering algorithms which, as it was mentioned above, are intended to discover groups of similar points (referred to as clusters) and following their evolution. Clustering of a system state space and passing clusters, instead of raw states, to an RL algorithm reduces the number of states which the RL algorithm has to classify, allowing it to focus only on a small number of states representing stable system.

(14) CHAPTER 1. INTRODUCTION. 5. conditions.. 1.3. Thesis statement. The goal of this dissertation is to show that autonomic, goal-driven adaptation in the service-oriented systems can be achieved by application of the RL algorithms with support of the online clustering algorithms. In other words, the dissertation proposes a new approach to realization of the Adaptive SOA architecture [149], which is based on the RL algorithms and is referred in the further part of the dissertation to as RL-based Adaptive SOA architecture. The dissertation thesis is formulated as follows: Reinforcement learning algorithms supported by online clustering of a system state space can be effectively used to achieve goal-driven adaptation in dynamic service-oriented systems. The dissertation elucidates, how a combination of the online clustering and RL algorithms can be applied to implementation of the self-adaptive SOA systems. The proposed approach focuses on the elements responsible for making decisions and analyzing system behaviour, which makes it independent from monitoring and management technologies used for instrumentation of a specific SOA system. The monitoring and management technologies have only to accomplish certain assumptions, which constitute an integral part of the proposed RL-based Adaptive SOA architecture. The thesis is supported by a prototype implementation of the architecture and its evaluation on example use-cases from the SOA systems realm. The proposed thesis has general applicability in the area of the SOA systems, and presumably can be used in other kinds of adaptive systems (e.g. IoT), as long as the assumptions regarding the system instrumentation are satisfied.. 1.4. Research contribution. The main contributions of this dissertation are as follows: • an exhaustive review of existing approaches to implementation of the Adaptive SOA architecture and implementations of adaptive IT systems based on machine learning algorithms; • a specification of necessary assumptions which have to be satisfied by monitoring and management frameworks used for SOA system instrumentation, in order to make machine learning algorithms applicable to its self-adaptation; • a concept of the Adaptive SOA architecture implemented using the RL algorithms supported by the online clustering algorithms;.

(15) CHAPTER 1. INTRODUCTION. 6. • a mathematical formulation of the adaptation process and the adaptation goal in the RL-based Adaptive SOA architecture; • a thorough survey on applicability of existing online clustering and RL algorithms for the purpose of implementation of the proposed architecture; • design and a prototype implementation of a proof of concept framework implementing the RL-based Adaptive SOA architecture • verification of correctness and usefulness of the RL-based Adaptive SOA architecture on example scenarios from the SOA systems domain.. 1.5. Dissertation organization. The dissertation is organized as follows. Chap. 2 presents background of the thesis including the SOA architecture, AC and online machine learning methods. In the second part of the chapter and exhaustive review of existing implementations of the Adaptive SOA architecture and machine learning-based approaches to IT systems’ adaptation is presented. In Chap. 3 the concept of goal-driven adaptation of the SOA systems based on application of the RL and online clustering algorithms is explained. Besides the concept, the chapter discusses also assumptions which have to be satisfied by monitoring and management technologies used for instrumentation of an SOA system, and formulates precisely the adaptation process and the adaptation goal. In Chap. 4 requirements for RL and online clustering algorithms are defined and, based on these requirements, specific algorithms are selected and evaluated. The algorithms which are pointed as the most promising for realization of the RL-based Adaptive SOA architecture are thoroughly presented, along with introduced modifications. Chap. 5 describes a framework which constitutes a prototype implementation of the proposed RL-based Adaptive SOA architecture. Additionally, the chapter contains a section devoted to theoretical evaluation of computational/memory complexity of the proposed solution. A comprehensive evaluation of the thesis along with a description of the evaluation methodology, evaluation scenarios and discussion of the evaluation results is presented in Chap. 6. Finally, Chap. 7 concludes the dissertation and discusses further directions of research..

(16) Chapter 2. Background and related work This chapter presents the thesis background and its placement in the context of related research areas. The second part of the chapter compares the posed thesis statement with to-date published approaches to the Adaptive SOA architecture and implementations of self-adaptive systems based on machine learning methods. The RL-based Adaptive SOA architecture coined in Section 1.3 as the main thesis idea constitutes a multidomain problem, which involves research in several areas: the SOA paradigm, the AC paradigm and online machine learning. This chapter deeply investigates each of these areas (in Sec. 2.1, 2.2 and 2.3 respectively) and compares the RL-based Adaptive SOA architecture with similar approaches (in Sec. 2.4 and 2.5), in order to clarify the thesis contribution to science. Finally, Sec. 2.6 summarizes the chapter.. 2.1. Service-Oriented Architecture (SOA). Since the early beginnings of computer science much effort has been devoted to address issues concerning manageability of software architectures and supervision of the IT systems’ development process [25]. Before 90’s, software engineering was mainly focused on maintenance and reduction of the amount of source code. This has yielded invention of the Object Oriented (OO) paradigm and, afterwards, the component-based approach to IT systems designing and development. Since 95’s, leading IT companies and software designers have observed that the amount of newly produced source code has been decreasing, while most of the software is developed through integration of existing components and libraries. Around 2000, rapid development of the Internet has contributed to emergence of a new software development model, which relies on building software through appropriate selection and integration of off-the-shelf services, exposed on the Internet by various providers. This approach to software development is referred to as service-oriented and is considered as a foundation of the SOA paradigm [41]. 7.

(17) CHAPTER 2. BACKGROUND AND RELATED WORK. 2.1.1. 8. The SOA fundamentals. A basic element of each system built in accordance with the SOA paradigm is a service. The service is an entity which encapsulates a set of functionalities behind a welldefined, standardized external interface [41, 66]. Examples of services are a hotel booking service, a metric calculation service, a dictionary service or a hosting service. Nowadays, almost all economies of the developed countries are based on services, hence it becomes a natural tendency that the services are also exposed on the Internet. Suppliers of the SOA services, i.e. companies or persons which expose them on the Internet, are called service providers, whereas users of the services, i.e. other services involved in the SOA systems or applications, are called service consumers. According to the SOA paradigm principles, each service should possess eight characteristic features [66, 42]. These features are: service discoverability, standardized service contract, loose coupling, service abstraction, service reusability, service autonomy, service statelessness and service composability. Two of these features – service abstraction and service autonomy – are particularly relevant in the context of the thesis, as they allow to treat services as autonomic components and enable their self-adaptation. The service abstraction feature hides service logic from its consumers, preventing from its undesirable modifications. On the other hand, the service autonomy feature means that each service has complete control over its operation and is independent on other services. Obviously, an SOA application usually involves many services to perform its business goals, thus full independence of services is rarely meet in practice. However, the service autonomy principle is still preserved because each service can freely choose among other services offering a specific functionality.. 2.1.2. The S3 model. In the early beginnings of the SOA paradigm, the process of services’ composition, their hosting and integration was not strictly standardized. In 2007, IBM T. J. Watson Research Center published the SOA S3 model [12] which describes all elements of the SOA systems from hosting infrastructure to end-users. The S3 model had been devised based on observation of multiple SOA projects for six years, and now is commonly accepted as a reference model for all SOA systems. The model consists of five horizontal layers and four vertical layers. The horizontal layers are associated with an SOA application structure, whereas the vertical layers are responsible for cross-application aspects such as integration, security or Business Intelligence (BI). The original SOA Solution Stack (S3) model published in [12] is shown in Fig. 2.1. Starting from the horizontal layers and going from the bottom, the fist layer is the operational layer. This layer comprises all applications and libraries which participate in a service creation process, thus it contains legacy systems, external libraries or pieces of programs. Currently, this layer often includes also system operational infrastructure, because virtualization technologies allow on dynamic management of computing and storage resources. The elements shared within the operational layer are combined.

(18) CHAPTER 2. BACKGROUND AND RELATED WORK. 9. Figure 2.1: The SOA Solution Stack model (source: [12]). into software components, which constitute elements of the next layer – the service components layer. Looking from the perspective of evolution of software architectures, the operational layer and the service component layer can be identified with Object Oriented and component-based approaches to software engineering (see the beginning of Sec. 2.1). This coincidence is deliberate because, as it was mentioned, the SOA paradigm focuses on creation of new systems through integration of existing ones, not on development of new systems from scratch. The first completely new layer defined in the S3 model is the services layer which is responsible for exposition of functionalities offered by the former layers outside. There is no rule regarding association of components and operational layer elements into services, so a service can involve a legacy application, a component, a component and an application, several components, etc., as long as it meets the features mentioned in Sec. 2.1.1. The all three described layers (i.e. operational, component and services) concern implementation and exposition of services. The next layer – business process layer – is yet fully on the consumer side, as its goal is to compose different services into a business process. The business process is a workflow which involves several services in order to provide a specific business objective. For example, a holiday trip reservation process involves reservation of hotels, reservation of museum tickets, flights booking, etc. Transitions between invocations of subsequent services within a single business process are considered as highly asynchronous, since execution of a single service can take hours or even days. The business process layer exposes ultimate functionalities offered by an SOA application to its clients. The clients of the SOA applications (i.e. users or other applications) are reflected in the S3 model in the consumers layer. Regarding vertical layers of the S3 model, the most internal layer is the integration layer. The role of this layer is provisioning of methods for integration of different elements of an SOA application, i.e. applications, components, services and business’ processes. The integration layer is based upon the concept of Enterprise Service Bus (ESB).

(19) CHAPTER 2. BACKGROUND AND RELATED WORK. 10. [31]. The ESB provides normalized, inter-elements communication based on normalized messages, which can be integrated with most of technologies used for implementation of web services. The next vertical layer – QoS layer – is responsible for non-functional aspects of the SOA systems such as performance, security, monitoring or management. In contrary to the integration layer, a common approach to implementation of the QoS layer relies on supporting only those aspects which are essential from the business viewpoint, e.g. monitoring [13] or security [115]. Information provided by the QoS layer, especially from monitoring of an SOA system, constitutes a staple of the information architecture layer. This layer performs tasks from the area of BI [98], i.e. intelligent analysis of the information about an SOA system provided by the QoS layer. The data analysis performed at the information architecture layer is crucial for the governance layer, which goal is to make decisions regarding management of an SOA system. Accuracy and fitness of the decisions made at the governance layer is highly dependent on results obtained from the information architecture layer. The S3 model constitutes a reference for all SOA-compliant applications and all leading technologies applied to development of the SOA systems. However, the SOA systems are constantly evolving and several new extensions to the original S3 model have been recently proposed [86]. One of such extensions is the Adaptive SOA architecture, which constitutes a starting point for the main subject of this dissertation.. 2.1.3. The Adaptive SOA architecture. One of the most frequently raised problems with the SOA systems is their hindered maintenance and management, which results from high complexity and dynamism of service-oriented systems [125, 110]. To address these issues, the Adaptive SOA architecture has been recently proposed [147, 149], as a conjunction of the AC and SOA paradigms. The concept is based on the observation that service-oriented systems are, on the one hand, complex and difficult to manage but, on the other hand, they possess extensive adaptation opportunities, which were not present in traditional systems [148]. These extensive adaptation opportunities result from agility of the SOA systems which manifests, by instance, in dynamic binding of services or possibility of their exchanging during system runtime. Moreover, many SOA-compliant services exposed on the Web are now built as adaptive components [18], what means that apart from a standard business interface they expose also an additional management interface supporting their configuration and tuning. These features of the contemporary SOA systems enable their extensive non-functional adaptation, on condition that a proper adaptation method is applied. Since self-adaptation of an SOA system is related to its non-functional aspects, components responsible for the adaptation process are located at the QoS layer of the S3 model and they refer to all horizontal layers of the model. Although the aforementioned papers [149, 147] which have introduced the Adaptive SOA architecture propose also its prototype implementations, they are limited to simplified cases and do not resolve all vital issues emerging in adaptation of the SOA systems. A.

(20) CHAPTER 2. BACKGROUND AND RELATED WORK. 11. Property. Description. Self-configuration. A system is able to manage its configuration itself. For example, if the system’s thread pool can be increased by a change in a configuration file, the system should be able to introduce this change when it is necessary (e.g. when the current thread pool size is insufficient). A system should continually seek for ways of its improvement. For example, an SOA system should continuously search for counterpart services which would improve a business process execution. In case of a failure, a system should be able to repair itself. For example, in case of a virtual machine elimination the system should migrate itself to another machine. System should automatically undertake preventive actions in case of malicious attacks or repeating failures. E.g. detection of a Denial of Service (DoS) attack should entail disabling of external network interfaces.. Self-optimization. Self-healing. Self-protection. Table 2.1: Four fundamental self-management properties of autonomic systems. detailed description of lacks of the already published approaches to the Adaptive SOA architecture is contained in Sec. 2.4. Nonetheless, it should be stressed here that none of the existing Adaptive SOA architectures is enough flexible and comprehensive to cover all their specific features of the SOA systems, thus implementation of the self-adaptive SOA systems still constitutes an open research issue.. 2.2. Autonomic Computing (AC). The concept of Autonomic Computing (AC) was originally proposed by IBM in 2001. In [64], Paul Horn from IBM compared a complex IT system to a human body and suggested that management of such complex systems should be done automatically as in the case of the human’s body, where most of operations is done unconsciously by the nervous system. Based on the comparison with the human’s nervous system, [64] identified four central self-management properties which are described in Tab. 2.1 [75]. Definition of the four fundamental characteristics of autonomous systems was a first step towards the AC paradigm. The next step was devising an AC reference model which would support achievement of the self-management properties defined in Tab. 2.1 in different types of systems. The basic reference model for AC was published by IBM in 2003 [67] along with the whole vision of autonomic computing [75]. The model has introduced a basic design pattern for implementation of all self-adaptive systems – MAPE-k (Monitor, Analyze, Plan and Execute using a gathered knowledge), which.

(21) CHAPTER 2. BACKGROUND AND RELATED WORK. 12. Figure 2.2: The MAPE-k adaptation loop. has quickly become a reference for building mature self-adaptive systems.. 2.2.1. The MAPE-k pattern. The MAPE-knowledge (MAPE-k) pattern constitutes a basic reference model which is somehow reflected in almost all autonomic systems. The pattern is depicted in Fig. 2.2 and, as it is shown, it consists of a loop formed by four components: monitoring, analysis, planning and execution. A common element utilized by all four components is knowledge, which together with the components creates the whole autonomic manager. The pattern is based on the concept of a managed element which refers to an entity (e.g. service, component or system) exposing two kinds of external interfaces: sensors and effectors. Sensors are used by the monitoring component of the MAPE-k pattern to obtain information regarding the managed element state. In turn, effectors are used by the execution element to introduce changes into the managed element, i.e. invoking management actions or introducing configuration changes. Presence of sensors and effectors in a system is a necessary condition to make it adaptable. Each of the MAPE-k components is responsible for realization of a particular task, as follows. 1. Monitoring: The monitoring component collects data about a managed element and transforms the data to a format expected by the analysis component. The component can be identified with a monitoring system providing information about a current state of the managed element. 2. Analysis: The analysis component is responsible for extraction of essential information about the managed element states and their evolution, and passing the information to the planning component..

(22) CHAPTER 2. BACKGROUND AND RELATED WORK. 13. 3. Planning: The planning component makes decisions regarding execution of management actions based on information received from the analysis component and knowledge about the managed element gathered in the knowledge component. The component makes two kinds of decisions. Firstly, it decides whether execution of any management action is necessary (for example, there is no reason to execute a management action when a system is in healthy conditions). Secondly, when a management action is necessary the component is responsible for selection of a pertinent action, which is the most proper for a current state of the managed element. 4. Execution: The execution component provides information about management actions exposed by the managed element and executes them, when requested by the planning component. The component often constitutes a part of an internal implementation of the managed element, e.g. when the managed element is implemented itself as an adaptive component. 5. Knowledge: The knowledge component gathers information about the managed element, which is necessary to make proper decisions regarding its adaptation. The knowledge can be complemented and utilized by all four components working in the adaptation loop. The above components combined into a process accordingly to Fig. 2.2, implement the whole adaptation loop of the autonomic manager. In almost all implementations of the autonomic manager, the adaptation loop works continuously, what means that after execution of a management action the whole loop starts from the beginning, i.e. the sensors interface provides information about a new system state, the analysis component analyzes it, the planning component decides whether another action is necessary and so forth. All properties mentioned in Tab. 2.1 can be changed by the autonomic manager on two ways: reactively and proactively. A reactive change takes place only when the manager observes that a system is in an ill condition and requires execution of a management action, i.e. the action is executed after deterioration of a system state. On the other hand, a proactive change can be introduced at any moment of system operation in order to prevent deterioration of a system state in the future. Therefore, the proactive adaptation is more reliable, but it is also more difficult to achieve.. 2.2.2. Goal-driven adaptation. The previous section described the adaptation loop defined by the MAPE-k pattern as a continuous process which, roughly-speaking, relies on execution of management actions and evaluation of their influence on a system condition. A main supposition behind this concept is that execution of management actions affects system operation, thus it is reflected somehow in the information obtained from the sensors interface. In other words, execution of management actions influences a system condition and the.

(23) CHAPTER 2. BACKGROUND AND RELATED WORK. 14. influence can be observed in the data provided by the monitoring component. Because the overall goal of the adaptation process is improvement of a system condition, thus there must exist a way of evaluation of action’s influence on a system. Lack of an appropriate evaluation method would result in a rigmarole adaptation process, in which actions would be executed chaotically as their influence on the system would remain unknown. This reason explains the necessity of introduction of an adaptation goal, i.e. a definition of the whole adaptation process’ direction. As it was mentioned in Sec. 1.1, the autonomic adaptation almost always refers to the non-functional aspects of a system, i.e. those which refer to a system condition, not to its functionality. A general problem with assessing a system condition is that an assessment manner is highly correlated with business objectives of a particular system’s provider, specified at the governance layer of the S3 model (see Sec. 2.1.2). By instance, one provider can focus on provisioning of high system performance regardless of the infrastructure cost, in contrast to another provider which can be interested in minimization of the incurred costs. Moreover, a proper definition of the adaptation goal must be carried carefully because some requirements may be in contrary with other, e.g. it is rather impossible to provide vast computing power and storage resources for a very low price. Consequently, implementation of each adaptive manager stipulates an unequivocal definition of the adaptation goal, which will enable unambiguous evaluation of management actions’ influence on a system, and goal-driven planning of their execution. For example, let consider an example system which can be adapted through assignment of different number of cores, but each additional core is associated with an additional cost. Even for such a simple system it is possible to define several completely different adaptation goals. 1. Maximization of the system performance – an adaptive manager given this goal will assign more cores to the system whenever it observes a performance degradation. 2. Minimization of the system cost – an adaptive manager will reduce the number of assigned cores to the minimum possible number and will be maintaining it, regardless of performance degradations. 3. Minimization of the system cost provided that the average system processing time is less than 2 seconds – an adaptive manager will try to keep the smallest possible number of cores and will increase it only when the measured mean system processing time is greater than 2 seconds. The first two examples of adaptation goals are trivial in implementation, however, they rather do not have any practical meaning because they represent rigid system behavior which can be achieved without introduction of any adaptation. The third example is an interesting and the most practical one, as it requires the adaptive manager to invoke different actions depending on a system condition. To properly accomplish this goal, the monitoring interface of an adaptive manager has to provide information about the.

(24) CHAPTER 2. BACKGROUND AND RELATED WORK. 15. mean system processing time and the analysis component has to distinguish between the cases when it is below and above the given threshold. Based on this information, the planning component should either increase the number of cores (when the mean processing time is greater then 2 seconds) or decrease it (when the mean processing time is less than 2 seconds). Apart from practical value of the last example, it also best illustrates the most commonly met manner of specification of adaptation goals – through a list of QoS parameters and their thresholds. Such lists of QoS requirements are commonly met in self-adaptive systems, and they are referred to as Service Level Agreement (SLA)/Traffic Condition Agreement (TCA) agreements [117]. The term Quality of Service (QoS), which has already appeared several times in this dissertation, refers in general to all non-functional features of systems. Depending on a specific kind of a system, different features are considered to be QoS parameters. For example, in the case of storage systems the QoS parameters are read/write transfer rate or storage capacity [122], whereas in the case of the SOA systems the QoS parameters are average request processing time of a service or its invocation rate [62]. From the adaptation process viewpoint, a crucial issue is monitoring of QoS parameters values and keeping them below or above a specified level, as it was discussed in the presented example. The problem of achieving and maintaining a specific level of QoS parameters is an open research issue, which has been solved only in certain cases, e.g. provisioning of storage resources for grid applications [118]. Most implementations of the autonomic manager do not guarantee that the adaptation goal is always accomplished, nonetheless, they try to achieve it reactively or proactively by influencing a system through execution of management actions. In other words, it can be said that the adaptive manager supports the best-effort QoS maintaining strategy, as it tries to keep a specific QoS level but it does not guarantee that it is always ensured (i.e. the adaptation process is driven by a goal, not constrained by it).. 2.2.3. Relationship with the control theory. Implementation of the adaptive manager can be treated as a resolution of the optimal closed-loop control problem, well-known from the control theory, which is also sometimes referred to as feedback control [104]. The concept of the closed-loop control is depicted in Fig. 2.3. Main elements of the feedback control, as in all types of controls, are a system and a controller. In the control theory, the system is considered a physical object, which transforms an input signal into an output [76]. The output is a function of the input signal and the perturbation signals which influence the system operation. The goal of the system control is adjustment of the input signal so that the system returns the expected output signal regardless of any perturbations. The controller makes decisions based on the error value, which results from a comparison of the system state measurements and their reference values [47]. The feedback control problem manifests a close similarity with the autonomic management problem, as both of them deal with influencing system operation so that to.

(25) CHAPTER 2. BACKGROUND AND RELATED WORK. 16. Figure 2.3: The concept of closed-loop control. achieve a prescribed goal, i.e. a certain QoS level in the case of the adaptive manager and a reference signal value given on the controller input in the case of the feedback control. The controller logic is equivalent to the analysis and planning components of the MAPE-k pattern, whereas the monitoring and execution components can be identified with the measurement and system input signals. The difference between the AC paradigm and the control theory lies in the way of treating the system. Most of the control theory approaches strive at modeling the system logic in some way, e.g. using physics laws or state machines, whereas the MAPE-k pattern assumes that the system is a ’black box’ and tries to gather knowledge regarding its operation based on observation of its evolution through the monitoring component and influence of undertaken management actions. This difference in treating the system entity appoints different areas of application for both approaches. The control theory approach is more appropriate for devices or tiny systems which have a finite state space and easily modeled behavior. For complex IT systems, whose state space is usually infinite and there is an enormous number of possible transitions between the states, the AC paradigm is usually more applicable as it does not require contriving any system models.. 2.2.4. AC in enterprise systems. The main problem with the AC paradigm is that it specifies only a model for autonomic systems but it does not propose any way of its implementation, leaving all implementation issues to the systems’ developers. In fact, MAPE-k is only a pattern of the adaptation process, but it neither defines how instrumentation of a system with sensors and effectors elements can be done, nor proposes any decision algorithms for the analysis and planning components. According to a recently published survey paper, existing implementations of the MAPE-k pattern for complex IT systems are limited [90]. The authors of the paper claim that in order to go beyond the current state-of-the-art approaches to implementation of autonomic enterprise systems it is necessary to re-formulate or evolve current methods and tools related to feedback loops through hybridization of concepts from control engineering, AI and computer science. Particularly, they mention that real-time decision making requires additional research since existing decision making models are irrelevant to real-time environments such as enterprise systems. As a research challenge [90] points also high dimensionality of data produced by future enterprise systems, which self-adaptive.

(26) CHAPTER 2. BACKGROUND AND RELATED WORK. 17. systems will have to cope with. The approach proposed in the dissertation aims at solving the issues posed in [90], as it proposes an approach based on hybridization of supervised and unsupervised machine learning methods. The issue of real-time decision making is addressed in this approach through application of the RL algorithms, which are commonly used technique for making decisions in real-time systems such as robots [77, 82]. On the other hand, the problem of high dimensionality of data is undertaken by online clustering algorithms which should be able to refine dimensions substantially contributing to changes in a system. A problem constitutes appropriate composition of these algorithms into the MAPE-k pattern, so they properly implement the adaptation loop.. 2.3. Online machine learning. Online machine learning algorithms is a group of methods for classification and clustering of data streams. In contrary to the offline machine learning algorithms, their online counterparts’ input constitutes a labeled stream of points (x(0) , y(0) ), (x(1) , y(1) ), ..., (x(i) , y(i) ), ... in case of online classification algorithms and an unlabeled stream of points x(0) , x(1) , ..., x(i) , ... in case of online clustering algorithms. The goal of online classification algorithms is to predict labels y(m) for points x(m) which have not been observed previously in the stream, based on knowledge gathered though observation of the stream [96]. An example data stream can contain subsequent days of the week as data points and labels denoting the weather in those days. An online classification algorithm can observe, for example, that most of Wednesdays are rainy, and based on this observation predict rainy weather on next Wednesdays. The difference between an online classification algorithm and its offline counterpart lies in ability of dynamic adaptation of its predictions to data observed in a stream. The algorithm can change returning predictions even for the same data points, if it observes that similar points in the stream take different labels than they took previously. This technique of learning is called reinforcement learning, as predictions of an algorithm are reinforced when they are correct, i.e. they coincide with values observed in a data stream afterwards. Returning to the example of weather prediction, the algorithm can observe after a period of time that Wednesdays are no longer rainy but they have become sunny (e.g. because of a season’s change). Such an observation weakens its previous predictions (Wednesday → rainy) and reinforces new predictions (Wednesday → sunny) resulting in a change of the returning predictions. The online clustering algorithms are aimed at discovering clusters, i.e. agglomerations of points, in points incoming from a data stream [5]. A data stream clustering algorithm observes a sequence of points and if it notices that some points from the sequence are closely located, it recognizes such a group of points as a cluster. An example application of data stream clustering algorithms can be segmentation of retail shops’ customers described by feature vectors containing information about made shoppings (e.g. selected products, value of purchases, etc.). Discovering groups of similar vectors.

(27) CHAPTER 2. BACKGROUND AND RELATED WORK. 18. Figure 2.4: The main flowchart of the RL algorithms. in such a stream allows us to know what types of clients are currently buying our products, how much money they usually spend or which products are selected most frequently. Clustering of such a stream allows also to detect strange clients, i.e. those which shoppings are unusual, thus are characterized by feature vectors which are not assigned to any cluster. Such isolated points which do not match to any cluster are referred in the literature to as outliers or noise points. The difference between online and offline clustering algorithms is a possibility of tracking evolution of clusters, i.e. appearance of new clusters, divisions or joins of existing ones as well as their vanishing. If, for example, a particular group of clients moved out from a specific shop and feature vectors characteristic for them no longer appear in the stream, an online clustering algorithm is able to recognize such a case and remove the cluster previously formed by those clients from the list of returned clusters. This advantage of data stream clustering algorithms make them applicable to analysis of dynamic systems, in contrary to the offline clustering algorithms which cannot change once discovered clusters [69].. 2.3.1. Reinforcement learning (RL). The idea of reinforcement learning as an online classification method was first proposed by Richard Sutton in his PhD Thesis in 1984 [126] where he described theoretical fundamentals for this group of methods. In his latter papers he proposed the first RL algorithm – Temporal Difference (TD) learning [127, 128]. Since that time many researchers have worked on these group of methods, and several new RL algorithms have been proposed [71]. The original TD-learning approach has been also developed [139]. Nevertheless, all RL algorithms share the same working flowchart, shown in Fig. 2.4. The flowchart relies on making some predictions (which can be treated as classifications) and comparing them with actual results observed afterwards. The more accurate the prediction was, the higher reward is returned to the algorithm. The driving goal of each RL algorithm is to maximize the cumulative reward resulting from predictions made over the whole system runtime. The RL algorithms belong to the machine learning field, as they constitute dynamic classification algorithms. The term ’dynamic’ means that the RL algorithms do not follow the standard single-train, multiple-use approach to development of classifiers, but they.

(28) CHAPTER 2. BACKGROUND AND RELATED WORK. 19. Figure 2.5: Typical cases in clusters’ evolution. learn continuously, through observation of a system state and results of made decisions. This is the key idea behind the RL algorithms, which makes them a proper method to solve real-time classification problems such as spam classification, recommendation of products or robots controlling. At it was concluded in Sec. 1.2, implementation of the autonomic manager for the SOA systems constitutes a dynamic classification problem. The classification task relies in this case on assignment of management actions to different system states, so that execution of these actions yields better accomplishment of an adaptation goal. The real-time aspect appears in self-adaptation of the SOA systems because of their continuous evolution which can result in the necessity of changing the assignments of actions to states over the system runtime. Fortunately, this issue is well addressed by the RL algorithms, since they can adapt in real time to changes in observations passed on their input (see the weather prediction example at the beginning of this section). An open issue, however, remains an appropriate choice of a specific RL algorithm for the purpose of implementation of the autonomic manager and its coupling with a data stream clustering algorithm responsible for recognition of stable system states. These issues require a study of existing algorithms and their careful selection or adaptation to the posed requirements as well as an in-depth evaluation.. 2.3.2. Online clustering. The issue of online clustering of data streams was posed first time in late 90’s, when the problem of processing large data streams produced, e.g. by monitoring systems, emerged. First data stream clustering algorithms were implemented through application of offline clustering methods, such as k-means, for moving time windows [55]. Since that time many new kinds of online clustering algorithms have been developed, but virtually all of them constitute modifications of offline clustering algorithms or are based on concepts borrowed from their offline counterparts. The difference between offline and online clustering algorithms is that apart from accomplishing standard tasks of clustering algorithms such as detection of clusters or refining isolated points, all data stream clustering algorithms are capable of following evolution of clusters in a stream. Fig. 2.5 presents an example process of clusters’ evolution, which comprise all basic cases supported by most of online clustering algorithms, i.e. appearance of a new cluster, cluster’s drifting, merging of clusters, division of a cluster and vanishing of a cluster. The ability of following evolution of clusters make data stream clustering algorithms an appropriate tool for accomplishing data.

(29) CHAPTER 2. BACKGROUND AND RELATED WORK. 20. mining tasks in dynamic systems such as hardware monitoring infrastructures, market information systems or enterprise IT systems. The problem of tracking evolution of enterprise systems through clustering of their state space posed in Sec. 1.2 belongs also to this kind of problems, thus it can be solved by the online clustering algorithms. The role of a data stream clustering algorithm in the RL-based Adaptive SOA architecture is recognition of stable states through analysis of the system state space and their passing to an RL algorithm in order to reduce the number of states which the RL algorithm will have to classify. Moreover, the ability of following evolution of clusters can be utilized in order to properly react to changes in the system. The changes can occur due to miscellaneous factors such as alterations in a system configuration, infrastructure failures, changes in the number of clients, etc. New states emerging as a result of these factors will be reflected in the data points as appearances of new clusters (in case of abrupt changes) or drifts of existing clusters (in case of slow changes spread over time). Sometimes, a single system state can split into two states (e.g. due to narrowing of some parameters’ values) or two states can merge into one state (e.g. if they became close to each other due to drifting) – these situations correspond to divisions/mergers of clusters shown in Fig. 2.5. Eventually, a system state can disappear as irrelevant to a current configuration or working environment, what can be treated as vanishing of a cluster. Most of data stream clustering algorithms are able to recognize the aforementioned situations and, therefore, pass adequate clustering results to an RL algorithm. A problem constitutes a proper choice of an algorithm for a specific application, since there are no online clustering algorithms which are applicable to all kinds of tasks. Many different kinds of data stream clustering algorithms have been already proposed in the literature [68] but most of them can be assigned to one of the following three categories [105]. • Partitional: these algorithms stem from the offline partitioning algorithms such as k-means and rely on division of a data space onto subspaces characterized by the lowest possible variance of points within a subspace and the greatest possible variance between individual subspaces (with regard to some metric). The subspaces constitute clusters returned by an algorithm. • Hierarchical: this group of algorithms works by recursive joining of points from a data space as long as the merged clusters do not have the expected granularity. Another variant of these algorithms rely on recursive dividing the data space to the required granulation. • Density-based: these algorithms work by dividing the whole clustered space onto a grid of regular subspaces (e.g. hypercubes or spheres), calculating density of points within each subspace and on-demand merging of adjacent subspaces which have sufficiently high density into clusters. Other kinds of online clustering algorithms comprise probabilistic streaming algo-.

(30) CHAPTER 2. BACKGROUND AND RELATED WORK. 21. rithms, fuzzy clustering or neural-network based clustering, nevertheless, they have limited applications and are not relevant to the scope of the dissertation, thus their description has been omitted. The choice of a specific data stream clustering algorithm is appointed by structure of data subjected to clustering, expected functionality, complexity as well as responsiveness to clusters’ evolution (certain algorithms react quicker to changes in the clusters’ structure than other). Hence, similarly as in the case of the RL algorithms, selection of a data stream clustering algorithm has to preceded by its theoretical analysis and practical evaluation.. 2.4. Existing approaches to the Adaptive SOA architecture. This section presents existing approaches to implementation of the Adaptive SOA architecture which have been already published, and points main advantages and drawbacks of each of them. Adaptive SOA Solution Stack The most mature approach to the Adaptive SOA architecture has been proposed in [149] as a reference model for adaptive SOA systems named Adaptive SOA Solution Stack. The approach constitutes a direct extension of the S3 model [42] and relies on introduction of an independent adaptation loop based on the MAPE-k pattern [67] at each layer of the S3 model. As a result, each layer of the S3 model becomes an autonomic component which adapts itself accordingly to a prescribed goal. Apart from the concept of Adaptive SOA Solution Stack, [149] presents also a reference implementation of this model which has been described more thoroughly in later papers and PhD Thesis published by the authors [129, 130]. One of the most important issues regarding implementation of the adaptation loop for different layers of the S3 model pointed in [149] is inability of formulation of a precise mathematical model of the SOA systems. This issue has led the authors to the conclusion that implementation of the adaptation loop for the SOA systems cannot be achieved through standard approaches such as control theory or fuzzy logic. As a solution to this issue the authors have proposed utilization of Complex Event Processing (CEP) frameworks such as Esper [88] to implementation of the analysis component and policy engines, e.g. Jess [48] or Drools [106], to implementation of the planning component. Although the obtained reference implementation has been proved to be valid [129, 130, 9], it possesses certain drawbacks which make them difficult to deploy in real enterprise systems. The first drawback, pointed somehow also by the authors, is lack of cooperation across autonomic managers at different layers, what can result in incoherent adaptation of the whole SOA application. The second flaw of the proposed reference implementation is necessity of manual specification of the adaptation logic, i.e. the Event Processing Language (EPL) statements for the CEP engine and policies/rules for the policy engines. This flaw significantly limits practical applications of the proposed.