Scholarly article on topic 'A Review of the Applications of Bio-inspired Flower Pollination Algorithm'

A Review of the Applications of Bio-inspired Flower Pollination Algorithm Academic research paper on "Computer and information sciences"

CC BY-NC-ND
0
0
Share paper
Academic journal
Procedia Computer Science
OECD Field of science
Keywords
{"Flower Pollination Algorithm" / "Single Objective Flower Pollination Algorithm" / "Multi-objective Flower Pollination Algorithm" / "Hybrid Flower Pollination Algorithm"}

Abstract of research paper on Computer and information sciences, author of scientific article — Haruna Chiroma, Nor Liyana Mohd Shuib, Sanah Abdullahi Muaz, Adamu I. Abubakar, Lubabatu Baballe Ila, et al.

Abstract The Flower Pollination Algorithm (FPA) is a novel bio-inspired optimization algorithm that mimics the real life processes of the flower pollination. In this paper, we review the applications of the Single Flower Pollination Algorithm (SFPA), Multi-objective Flower Pollination Algorithm an extension of the SFPA and the Hybrid of FPA with other bio-inspired algorithms. The review has shown that there is still a room for the extension of the FPA to Binary FPA. The review presented in this paper can inspire researchers in the bio-inspired algorithms research community to further improve the effectiveness of the PFA as well as to apply the algorithm in other domains for solving real life, complex and nonlinear optimization problems in engineering and industry. Further research and open questions were highlighted in the paper.

Academic research paper on topic "A Review of the Applications of Bio-inspired Flower Pollination Algorithm"

CrossMark

Available online at www.sciencedirect.com

ScienceDirect

Procedía Computer Science 62 (2015) 435 - 441

The 2015 International Conference on Soft Computing and Software Engineering (SCSE 2015)

A Review of the Applications of Bio-Inspired Flower Pollination

Algorithm

Haruna Chiromaa,*, Nor Liyana Mohd Shuibb, Sanah Abdullahi Muazc, Adamu I. Abubakard, Lubabatu Baballe Ilae, Jaafar Zubairu Maitamaf

aFederal College of Education (Technical), Department of Computer Science, Gombe, Nigeria aUniversity of Malaya, Department of Artificial Intelligence, Kuala Lumpur, Malaysia bUniversity of Malaya, Department of Information System, Kuala Lumpur, Malaysia cBayero University Kano, Faculty of Computer Science and Information Technology, Department of Software Engineering, Kano, Nigeria international Islamic University Malaysia, Faculty of Information and Communication Technology, Kuala Lumpur, Malaysia dBayero University Kano, Faculty of Engineering, Department of Machatronics Engineering, Kano, Nigeria fBay ero University Kano, Faculty of Computer Science and Information Technology, Department of Information Technology, Kano, Nigeria

Abstract

The Flower Pollination Algorithm (FPA) is a novel bio-inspired optimization algorithm that mimics the real life processes of the flower pollination. In this paper, we review the applications of the Single Flower Pollination Algorithm (SFPA), Multi-objective Flower Pollination Algorithm an extension of the SFPA and the Hybrid of FPA with other bio-inspired algorithms. The review has shown that there is still a room for the extension of the FPA to Binary FPA. The review presented in this paper can inspire researchers in the bio-inspired algorithms research community to further improve the effectiveness of the PFA as well as to apply the algorithm in other domains for solving real life, complex and nonlinear optimization problems in engineering and industry. Further research and open questions were highlighted in the paper.

© 2015 TheAuthors.PublishedbyElsevierB.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.Org/licenses/by-nc-nd/4.0/).

Peer-reviewunderresponsibility of organizing committee of The 2015 International Conference on Soft Computing and Software Engineering(SCSE2015)

Keywords: Flower Pollination Algorithm; Single Objective Flower Pollination Algorithm; Multi-objective Flower Pollination Algorithm; Hybrid Flower Pollination Algorithm

* Corresponding author. Tel.: +60143873685. E-mail address: hchiroma@acm.org

1877-0509 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.Org/licenses/by-nc-nd/4.0/).

Peer-review under responsibility of organizing committee of The 2015 International Conference on Soft Computing and Software Engineering (SCSE 2015)

doi: 10.1016/j.procs.2015.08.438

1. Introduction

Optimization can be explained as the process of searching to locate the best solution to a problem. The searching process can be performed using agents which typically evolve iteratively based on specified rules of mathematical models. Emergent characteristics of the system behavior lead to its adaptation that correspond to the best solution in the search space. The system converges when the criteria's fixed for the stoppage is reach [1]. Optimization problems are solved using optimization algorithms often refers to as tools and techniques for the solving of problems related to optimization to find the best solution. Though, the best solution has no guarantee that it can always be reachable. The search for optimal solutions is complex in view of the fact that real world problem are mostly accompanied by uncertainties. Thus, concentration is not only focused on the optimal solution, but also robustness of the solution are needed in engineering design and industry. Real life application of solutions requires a solution that is both optimal and robust [2]. In the 1950's and 1960's computer scientist model the concept of evolution that give birth to Genetic algorithm (GA) [3]. This development inspired the springs of bio- inspired algorithms to circumvent the limitations of the gradient descent algorithms in solving optimization problems. As such, several biologically inspired global optimization algorithms were proposed by researchers, for example, particle swarm optimization (PSO) [4], artificial bee colony (ABC) [5], Bat algorithm [6], Bees algorithms [7], Monkey Search [8], Glow warm swarm [9], bacterial foraging [10], Fish-swarm algorithm [11], Cat swarm [12] etc. a brief review of the biologically inspired algorithms can be found in [13]. More recently, Flower Pollination Algorithm (FPA) [14], Chicken Swarm Optimization (CSO) [15], Approximate Muscle Guided Beam Search (AMGBS) [16], Magnetotactic Bacteria Optimization based on Moment Migration (MBOMM) [17], etc.

However, among the more recently biologically inspired algorithms, FPA has witnessed significant applications in several domains compared to CSO, AMGBS, and MBOMM. Therefore, we review the applications of the FPA in solving complex and nonlinear problems in numerous domains and the extension of the PFA. This review can offer researchers a quick glance of the applications of the FPA. Also, the review can allow researchers to quickly identify areas that require further development and propose novel applications of the FPA.

Other sections of the paper are organized as follows: Section 2 gives the description of the PFA including both single objective and multiple objective FPA. Section 3 presents the review of the applications of the FPA in several domains, including the results of the studies. In Section 4, concluding remarks, further extension of the FPA and open questions that if answered can probably improve the performance and robustness of the FPA are presented.

2 Flower Pollination Algorithm

This section introduces the basic theory of the FPA including the single objective FPA (SFPA), Hybrid FPA (HFPA) and multi-objective FPA (MFPA), an extension of the SFPA.

2.1 Single Objective Flower Pollination Algorithm

Yang [14] emulated the characteristic of the biological flower pollination in flowering plant to develop SFPA based on the rules listed as follows:

1. The global pollination processes are biotic and cross pollination through which the pollen transports pollinators perform the levy flight.

2. Local pollination is viewed as abiotic and self pollination.

3. Reproduction probability is considered as flower constancy which is proportional to the resemblance of the two flowers in concerned.

4. The switching probability controlled both the local and global pollinationp e [0, 1]. Local pollination can have fraction p that is significant in the entire processes of the pollination because of physical proximity and wind.

The plant can possess multiple flowers and every flower patch typically emits millions or even billions of pollen gametes in real life pollination practice. To simplify the proposed algorithm development, it was assumed that each plant has a single flower and each flower emit only a single pollen gamete. This result to the elimination of the need to differentiate pollen gamete, plant or solution to a problem. This means that a solution xi to a problem is equivalent to a flower and pollen gamete. The major stages in the design of SFPA are global and local pollination. In the

global pollination, the pollens of the flowers are moved by pollinators e.g. insects and pollens can move for a long distance since the insects typically fly for a long range of distances. This process guarantees pollination and reproduction of the fittest solution represented as g*. The flower constancy can be represented as:

x+1 = xt + L(xt - g*) (1)

From Eq.(1), x't , t, g* and L are pollen i or solution vector xi, iteration, the best solution found in the current

generation or iteration and strength of the pollination(step size) respectively. The levy flight is used to represent movement of the insects, thus, L > 0 from a Levy distribution

l ~lrq)sin(n/2)_L (, >>> 0). (2)

From Eq. (2), r(X) represent the standard gamma function, and the distribution is valid for large steps s > 0. From rule 2, the local pollination and flower constancy is expressed as:

x+1 = x+e (xj - xk). (3)

where xtj and xkt represent pollen from different flowers of the same species of plant. Thus, mimic the flower

constancy in a limited neighborhood. The switch probability or proximity probability is used to switch between common global pollination to intensive local pollination. The effectiveness of the PFA can be attributed to the following two reasons: (1) Insect pollinators can travel in long distances which enable the FPA to avoid local landscape to search in a very large space (explorations). (2) The FPA ensures that similar species of the flowers are consistently chosen which guarantee fast convergence to the optimal solution (exploitation).

2.2 Multi-objective Flower Pollination Algorithm

The MFPA is an extension of the SFPA described in section 2.1 proposed by Yang et al. [18] for solving multi-objective optimization problems. The difference between the SFPA and the MFPA is that, the former can only solve single objective optimization problem which lacks the capability to search for solutions for multi-objective problems. The later can solve multi-objective optimization problems. A number of approaches exist for solving multi objective problems using a single objective algorithm. However, the simple approach is to use a weighted sum to integrate the whole multiple objectives into a composite single objective:

f = Z wifi' Z w = 1, wi > 0, (4)

i=1 i=1

where m, and wi (i = 1, ...,m) are number of objectives and non-negative weights respectively. The random weights wi drawn from uniform distribution or low-discrepancy random numbers is used to accurately obtain Pareto front with the optimal solutions distributed uniformly on the front.

3 Applications of the Flower Pollination Algorithm

3.1 The Single Objective Flower Pollination Algorithm

Yang [14] proposed SFPA in view of the fact that researchers have used only limited characteristic of nature. This allow a vacuum for the development of nature inspired algorithm. The SFPA was implemented on benchmark functions. Results showed that the performance of SFPA outperforms that of the GA and PSO. Sharawi et al. [19]

used SFPA in wireless sensor network (WSN) to effectively select cluster heads and to distribute cluster nodes so as to overcome the limitations of the Low-Energy Adaptive Clustering Hierarchy (LEACH). Experimental simulation results suggested that the SFPA was found to perform better than the LEACH in terms of lifetime, the packet sent to base station Vs. round number, and the number of dead nodes. Emary et al. [20] proposed SFPA for the optimization of retinal vessel segmentation. The optimization was conducted in two stages: (1) the SFPA searches for the optimal vessels of the retina. (2) The cluster center obtained was enhanced using local searches. The results indicated that the SFPA convergence to the optimal solution very fast. It was found that the SFPA was robust even with abnormal images. Sakib et al.[21] conducted a comparative study between SFPA and Bat algorithm on both unimodal and multimodal, low and high dimensional continuous functions. Comparative simulation results indicate that the SFPA performance was superior to that of the Bat algorithm for the continuous optimization problems. Platt [22] applied SFPA in solving dew point pressure of a system exhibiting double retrograde vaporization. It was reveal from the experiments that the SFPA was able to obtain the pressures and molar fractions of the liquid phase. Lukasik and Kowalski [23] compared the performance of the SFPA with that of PSO on benchmark continuous optimization problems. The comparative studies showed that the SFPA and PSO demonstrated similar performance on some of the benchmark functions. However, the SFPA outperforms the PSO on some of the functions. The performance of the two algorithms (SFPA and PSO) was not clear due to inconsistent performance that cannot obviously differentiate between the SFPA and the PSO.

3.2 The Multi-objective Flower Pollination Algorithm

Yang et al. [18] extended SFPA to MFPA in order to solve multi-objective problems. The performance of the MFPA was evaluated on benchmark functions and structural design (design of a disc brake). Comparative analysis of the MFPA and other algorithms indicated that the MFPA performs better than the vector evaluated genetic algorithm (VEGA), NSGA-II, multi-objective differential evolution (MODE), differential evolution for multi-objective optimization (DEMO), multi-objective bees algorithms (Bees), strength Pareto evolutionary algorithm (SPEA), GA, and PSO. Additional analysis of the MFPA on welded beam design further proved the superiority of the MFPA over GA, PSO, VEGA, NSGA-II, MODE, DEMO, Bees, and SPEA [24].

Table 1. Brief comparisons of the FPA variants with other algorithms and contributions of the studies

Reference Proposed algorithm Algorithm/s compared with Contribution

[14] SFPA GA and PSO SFPA outperform GA and PSO

[18] MFPA VEGA, NSGA-II, MODE, DEMO, Bees, MFPA outperform the comparative algorithms

SPEA, GA and PSO

[19] SFPA LEACH SFPA outperform LEACH

[20] SFPA Not compared Advanced convergence speed

[21] SFPA Bat algorithm SFPA outperform Bat algorithm

[22] SFPA Not compared SFPA obtained the pressures and molar

fractions of the liquid

[23] SFPA PSO SFPA outperform PSO in some cases

[26] SFPA + FPCHS FPCHS SFPA + FPCHS outperform FPCHS

[27] SFPA + PSO SFPA SFPA + PSO outperform SFPA

[28] ESSFPA PSO ESSFPA outperform PSO

[29] DDIFPA ABC, PSO, Cuckoo Search, Gravitational DDIFPA was better than the compared

Search Algorithm, differential evolution, algorithms

standard SFPA, and firefly algorithm

[30] SFPAPSO Not compared Reduce power loss

3.3 The Hybrid Flower Pollination Algorithm

Hybridization in FPS is the hybrid of FPA with other biologically inspired meta-heuristic algorithms, e.g. PSO, GA, etc. Hybridization eliminates the weakness of the FPA to improve the performance of the FPA to converge to the optimal solution in a short period of time. The hybrid of two or more algorithms has proven to have the ability to eliminate their limitations and capitalized on their strengths to converge faster to the best solution than the individual algorithm [25]. Abdel-Raouf et al. [26] proposed a hybrid of SFPA and Chaotic Harmony Search (FPCHS) to enhance

the search accuracy of the SFPA. The performance of the proposed hybrid method was evaluated on Sudoku puzzles. Results have shown that the hybrid of SFPA and FPCHS converges faster to the optimal solution than the FPCHS. Similarly, the SFPA was hybridized with PSO to improve performance of the SFPA. The proposed hybrid SFPA was used to solve constrained global optimization problem. It was found that the proposed hybrid approach improves the performance of the state of the art algorithms [27]. Yang et al. [28] combined eagle strategy with SFPA (ESSFPA) to strike a balance between exploitation and exploration to improve efficiency of the algorithm. Unimodal and multimodal functions were used to test the algorithm performance. Results showed that the ESSFPA utilized only 10% of the computational effort compared to PSO. Wang and Zhou [29] introduced neighborhood searching strategy, dynamic switching probability strategy, dimension by dimension update and improvement strategy in the SFPA to enhanced convergence speed and solution quality. These strategies were integrated to design a dimension by dimension improvement of SFPA (DDIFPA). The simulation results on the benchmark functions suggested that the DDIFPA outperforms the ABC, PSO, Cuckoo Search, Gravitational Search Algorithm, differential evolution, standard SFPA, and firefly algorithm. Kanagasabai and RavindhranathReddy [30] integrated SFPA and PSO (SFPAPSO) to solve a problem of reactive power dispatch. The SFPAPSO was evaluated on standard IEEE 30, IEEE 57 bus test systems. The simulation comparative results showed that the SFPAPSO can reduce real power loss. Table 1 presents a summary for comparisons of the variants of FPA with other algorithms and contributions.

4 The Trend of Publications in the Applications of Flower Pollination Algorithm

Table 2 present trend of the publications on the applications of FPA from 2012 to 2015, i.e. since inception of the algorithm. The publications trend showed that the FPA is attracting attention from researchers and the number of publications is growing very fast.

Table 2: The number of publications of the FPA applications by year

Number of publications

The lowest number of publications was recorded in 2012 and 2015 which is expected since the algorithm was proposed in 2012 and the year 2015 is at early stage. The highest number of publications was recorded in 2014 with a total of nine publications. This implies that the publication in this area of research is growing and could probably continue to grow very fast into the future literature.

Figure 1: Classification of the variants of FPA and the number of applications Figure 1 showed the classification of the FPA and the numbers associated with each of the FPA variants represent

the number of applications found in the literature. From Figure 1, it can be stated that the most used variants of the FPA is the SFPA which has the highest number of applications. Probably is because of its efficiency and earlier appearance in the literature than the other variants. Other variants of the FPA result from the modifications of the SFPA.

5. Limitations of the studies

The binary variant of the FPA is scarce in the literature. Presently, the FPA cannot be able to solve binary problems that can produce a binary solution. In a binary meta-heuristic algorithms, e.g. binary CS, binary PSO, binary GA, etc. the solution typically comprised a set of binary numbers or bits. However, there is no Binary FPA (BFPA) as shown in Figure 1, indicated by 0. This is an interesting topic for further research to extend the FPA to BFPA. In the researches presented in Table 1, the researchers have not used statistical analysis to validate their experimental results as recommended by Dem'sar [31]. Comparing performance of a proposed algorithm is critical in order to assess the effectiveness of the algorithm but the work of [20, 22, 30] ignored the comparison. This make it very difficult to measure the impact of the algorithms proposed by the researchers. The researchers that compared their method for evaluation purposes overlooked to move further and compute the percentage improvement made by their proposed algorithms [14, 18-19, 21, 23, 26-29]. Yang et al. [28] improve the performance of the SFPA based on hybridization. However, the study has not directly compared the effectiveness of the hybrid SFPA with the standard SFPA as done in [29].

6 Conclusion and Futur Research

The applications of the FPA in numerous domains are review to offer researchers a quick glance of the applications of FPA. This can allow researchers to propose novel applications and quickly identify areas that require improvement. The FPA is in two major classes which are SFPA and the MFPA, the former is dedicated to solving single objective problems, whereas the latter is for solving multiple objective problems. The review revealed that the domains that are attracted by the applications of the FPA include: energy, structural design, function optimization, game, images, and WSN. Results suggestions from the applications of the FPA indicated that the FPA advances the performance of established biologically inspired algorithms such as GA, ABC, PSO, etc. including the CS that is presently attracting extraordinary attention from the research community due to its effectiveness and few parameters settings. In a single case, the performance of the PFA was found to be similar to that of PSO. From the review presented in this paper, it can clearly be seen that the applications of FPA are not quite much in the literature because the algorithm is relatively new and presently under explorations by the research community. We suggest the extension of FPA to binary FPA. The FPA can further be explored in other domains such as carbon dioxide emissions, crude oil production, clustering, big data, software engineering, education data mining, web mining, cloud computing, quantum computation, etc. The following open questions, if answered, can improve the performance of the FPA: What are the optimum parameter values required for the execution of FPA in any domain? Which biologically inspired algorithm can best be hybridized with FPA?

References

1. Keller EF. Organisms, machines, and thunderstorms: a history of self-organization, part two. Complexity, emergence, and stable attractors. Hist Stud Nat Sci 2009;39:1-31.

2. Yang XS, Deb S. Cuckoo search: recent advances and applications. Neural Comput Appl 2014; 24:169-174.

3. Mitchell, M.: An introduction to genetic algorithms. Massachusetts: MIT press; 1998.

4. Kenneth J, Eberhart RC. Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, NJ, 1995. p. 1942-1948.

5. Karaboga D, Basturk B. A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. J Global Optim 2007; 39:459-471.

6. Yang XS. A new metaheuristic bat-inspired algorithm. In: Gonzalez et al. Nature Inspired Cooperative Strategies for Optimization. 2010. 284, p.65-74.

7. Pham DT, Ghanbarzadeh A, Koc E, Otri S, Rahim S, Zaidi M. The bees algorithm-a novel tool for complex optimization problems. In:

Proceedings of the 2nd Virtual International Conference on Intelligent Production Machines and Systems (IPROMS 2006), 2006, p.454-45.

8. Mucherino A, Seref O. Monkey search: a novel metaheuristic search for global optimization. In: proceedings Data Mining, Systems Analysis and Optimization in Biomedicine, 2007, 953, p.162-173.

9. Krishnanand KN., Ghose D. Detection of multiple source locations using a glowworm metaphor with applications to collective robotics. In: Proceedings of the Swarm Intelligence Symposium. 2005. p.84-91.

10. Passino KM. Biomimicry of bacterial foraging for distributed optimization and control. Control Systems, IEEE 2002, 3; 52-67.

11. Li XL, Shao ZJ, Qian JX. Optimizing method based on autonomous animats: Fish-swarm algorithm. Xitong Gongcheng Lilun yu Shijian/System Engineering Theory and Practice, 2002; 22: 11-32.

12. Chu SA, Tsai PW, Pan JS. Cat swarm optimization. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2006; 4099:LNAI:854-858.

13. Fister Jr I, Yang XS, Fister I, Brest J, Fister D. A Brief Review of Nature-Inspired Algorithms for Optimization. Elektrotehniski Vestnik, 2013; 80:116-122.

14. Yang XS. Flower pollination algorithm for global optimization. In Unconventional Computation and Natural Computation 2012; 240249.

15. Meng X, Liu Y, Gao X, Zhang H. A New Bio-inspired Algorithm: Chicken Swarm Optimization. In Advances in Swarm Intelligence, 2014; 86-94.

16. Jiang H, Zhang S, Ren Z, Lai X, Piao Y. Approximate Muscle Guided Beam Search for Three-Index Assignment Problem. In Advances in Swarm Intelligence 2014; 44-52.

17. Mo H, Liu L, Geng M. A Magnetotactic Bacteria Algorithm Based on Power Spectrum for Optimization. In Advances in Swarm Intelligence 2014; 115-125.

18. Yang XS, Deb S, He X. Eagle strategy with flower algorithm. In Advances in Computing, Communications and Informatics (ICACCI), 2013 International Conference on 2013; 1213-1217.

19. Sharawi M, Emary E, Saroit IA, El-Mahdy H. Flower pollination optimization algorithm for wireless sensor network lifetime global optimization. International Journal of Soft Computing and Engineering, 2014; 4:54-59.

20. Emary E, Zawbaa HM, Hassanien AE, Tolba MF, Snasel V. Retinal vessel segmentation based on flower pollination search algorithm. http://scholar.cu.edu.eg/sites/default/files/abo/files/ibica2014_p11.pdf Retrieved 23 October, 2014.

21. Sakib N, Kabir MWU, Subbir M, Alam S. A Comparative Study of Flower Pollination Algorithm and Bat Algorithm on Continuous Optimization Problems. International Journal of Soft Computing and Engineering, 2014; 4, 13-19 (2014)

22. Platt, G. M.: Computational Experiments with Flower Pollination Algorithm in the Calculation of Double Retrograde Dew Points. International Review of Chemical Engineering (IRECHE), 2014; 6: 95-99.

23. Lukasik S, Kowalski PA. Study of Flower Pollination Algorithm for Continuous Optimization. In Intelligent Systems 2015; 451-459.

24. Yang XS, Karamanoglu M, He X. Flower pollination algorithm: A novel approach for multiobjective optimization, Engineering Optimization, 2014; 46:1222-1237.

25. Editorial.: Hybrid learning machine. Neurocomputing, 72, 2729-2730 (2009)

26. Abdel-Raouf O, Abdel-Baset M, El-henawy I. A Novel Hybrid Flower Pollination Algorithm with Chaotic Harmony Search for Solving Sudoku Puzzles. International Journal of Modern Education and Computer Science 2014; 3:38-44.

27. Abdel-Raouf, O., Abdel-Baset, M., El-henawy, I. A New Hybrid Flower Pollination Algorithm for Solving Constrained Global Optimization Problems. International Journal of Applied, 2014; 4:1-13.

28. Yang XS, Deb S, He X. Eagle strategy with flower algorithm. In Advances in Computing, Communications and Informatics (ICACCI), 2013 International Conference on 2003; 1213-1217.

29. Wang, R., Zhou, Y.: Flower Pollination Algorithm with Dimension by Dimension Improvement. Mathematical Problems in Engineering 2014; Article ID 481791, 9 pages, http://dx.doi.org/10.1155/2014/481791.

30. Kanagasabai L, RavindhranathReddy B. Reduction of real power loss by using Fusion of Flower Pollination Algorithm with Particle Swarm Optimization. Journal of the Institute of Industrial Applications Engineers, 2014; 2:97-103.

31. Dem"sar J. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 2006; 7, 1-30.