Scholarly article on topic 'Usage-pattern based Statistical Web Testing and Reliability Measurement'

Usage-pattern based Statistical Web Testing and Reliability Measurement Academic research paper on "Economics and business"

CC BY-NC-ND
0
0
Share paper
Academic journal
Procedia Computer Science
OECD Field of science
Keywords
{"statistical Web testing" / "Markov usage model" / "reliability measurement"}

Abstract of research paper on Economics and business, author of scientific article — Chao Chen, Yihai Chen, Huaikou Miao, Hao Wang

Abstract With the increasing complexity of Web applications, ensuring the reliability of Web applications becomes an enormous challenge. Statistical Web testing, proposed by researchers in recent years, is an appropriate approach to testing Web applications and to estimating their reliability. However, most of the researchers only focus on estimating the overall reliability of Web applications. They only discuss the failures which are recorded in error log files. In fact, the overall reliability cannot reflect the quality of Web applications under different usage patterns. In this paper, we propose an approach to estimating the particular reliability of Web applications under a particular usage pattern. Moreover, our approach can automatically identify the difference between output and expected pages.

Academic research paper on topic "Usage-pattern based Statistical Web Testing and Reliability Measurement"

Available online at www.sciencedirect.com

ScienceDirect

Procedia Computer Science 21 (2013) 140 - 147

The 4th International Conference on Emerging Ubiquitous Systems and Pervasive

Networks (EUSPN-2013)

Usage-pattern Based Statistical Web Testing and Reliability

Measurement

Chao Chena b, Yihai Chena b *, Huaikou Miaoa b, Hao Wangc

a School of Computer Engineering and Science, Shanghai University, Shanghai, China b Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China cMcMaster Centre for Software Certification, McMaster University, Hamilton, Ontario, Canada

Abstract

With the increasing complexity of Web applications, ensuring the reliability of Web applications becomes an enormous challenge. Statistical Web testing, proposed by researchers in recent years, is an appropriate approach to testing Web applications and to estimating their reliability. However, most of the researchers only focus on estimating the overall reliability of Web applications. They only discuss the failures which are recorded in error log files. In fact, the overall reliability cannot reflect the quality of Web applications under different usage patterns. In this paper, we propose an approach to estimating the particular reliability of Web applications under a particular usage pattern. Moreover, our approach can automatically identify the difference between output and expected pages.

© 2013 TheAuthors. Published by Elsevier B.V.

Selection and peer-review under responsibility of Elhadi M. Shakshuki

Keywords: statistical Web testing; Markov usage model; reliability measurement

1. Introduction

A Web application is a software application that is accessible via a thin client (i.e., Web browser) over a network such as the Internet or an Intranet [1]. Nowadays, Web applications have been widely used to support various activities in our daily life. Although Web applications bring us convenience and improve productivity, the failures of Web applications can be chaotic. Therefore, techniques to ensure the reliability of Web applications have been the research focus among academia and industry.

Web applications have characteristics such as: massive user population, diverse usage environments, document and information focus [2]. Statistical Web testing, proposed by Kallepalli et al. [2], allows testers to focus on the parts of Web applications under test that are more frequently accessed in order to ensure their reliability. Kallepalli et al. only consider the errors stored in the error log file. However, as Tonella et al. argued in [12], the difference between output and expected pages should be regarded as one type of failure, which is not stored in the error log file.

* Corresponding author: Yihai Chen

Email address: yhchen@staff.shu.edu.cn (Yihai Chen)

1877-0509 © 2013 The Authors. Published by Elsevier B.V. Selection and peer-review under responsibility of Elhadi M. Shakshuki doi: 10.1016/j.procs.2013.09.020

Up to now, existing statistical testing approaches can only measure the overall reliability of a Web application. In fact, under specific usage patterns, the particular reliability of a Web application may be lower than the overall reliability. In this way, Web users with certain usage patterns may have lower satisfaction level than other users. Therefore, it is necessary and meaningful for testers to estimate the reliability w.r.t. different usage patterns. In this paper, we propose to apply k-means algorithm to partition the Web users into groups, and extract a Markov usage model for each group; with a test suite generated for each model, we can measure the particular reliability for each user group. Our approach can automatically detect the difference between output and expected pages of a Web application. We use a case study of online bookstore to demonstrate the viability and effectiveness of our approach.

The remainder of the paper is organized as follows: Section 2 briefly discusses the related work on statistical Web testing. Section 3 and 4 present our approach and case study respectively. Section 5 concludes and proposes further work.

2. Related Work

Various kinds of Web testing approaches have been used to improve the reliability of Web applications, such as: data flow testing [4], user session based testing [5, 6], and URL-based testing [7]. In the recent years, there has been increasing interets in statistical Web testing. Kallepalli et al. [2, 8] constructed Unified Markov Models (UMMs) to reflect possible usage scenarios of Web applications by analyzing the access log. Test cases were generated randomly from UMMs. By analyzing both the access log and the error log, they estimated the overall reliability of Web applications with the Nelson model [9] and the G-O model [10]. Tonella and Ricca [12,13] proposed a dynamic analysis approach to extracting a Web application model as a Markov usage model from an existing Web application with a probability distribution. As mentioned earlier, they regard as difference between output and expected pages as one type of failure. However, the testers have to identified the differences manually. Sant et al. [14] proposed an approach to building statistical models of user sessions and generate test cases from these models automatically. They designed a control model and a data model to represent the users' dynamic behaviors. The control model was used to represent the possible sequences of URLs, and the data model to represent the possible sets of parameter values, which were sent as the name-value pairs in users' requests. Sprenkle et al. [15] directly construct a navigation model from user sessions. Both the construction of the navigation model and the generation of abstract test cases were automatic.

3. Usage-pattern Based Statistical Web Testing

The complete process of our approach to statistical Web testing is different from existing approaches and it is shown in Fig. 1. In order to estimate the particular reliability of a Web application under a particular usage pattern, we need to build more than one statistical testing model. Each model reflects one usage pattern of the Web application. Step 1 is to preprocess the log entries and partition the users into disparate groups. Step 2 is to build a Markov usage model as a statistical testing model from each group's user sessions. Step 3 is to generate the test cases based on the usage model built in step 2. Test data can be obtained from the data model which is built from a group's user sessions and used to support the execution of abstract test cases. Step 4 is to decide whether an execution of the test case is correct. We count the failures and hits to estimate the reliability of a Web application.

3.1. Log Preprocessing and Web User Partition

A Web access log file has huge numbers of log entries. Each log entry includes IP address, Time, Resource, Referrer, etc. Two columns, IP address and User-Agent, are used to identify the users of log entries. The log entries of one user within a given time period are grouped into a user session. We define the access log entry as a 6-tuple {ip, time, preSt, nextSt, agent, names). In this tuple, ip denotes user's IP address; time denotes the date and time when the user request the Web application; preSt denotes the state before the Web user request the web application; nextSt denotes the next state; agent denotes the browser information of users; names represents the set of parameter variables, which are sent when the Web users request a Web application.

Step 2: Building usage model

Markov usage model for group i

Step 3: Generating test cases

A Test suite

Step 4: Measuring Reliability

A particular reliability of a Web application

Fig. 1. Complete Process of Our Approach

Before we partition the Web users, the characteristics of Web users need to be described. We use the transitions (i.e., preSt ^ nextSt) and their occurrence probabilities to reflect the characteristics of one user's usage pattern. Table 1 shows a sample of a user-transition matrix, which reflects the relationship between users and transitions. The value of each element is used to represent the occurrence probability of one transition. Euclidean distance [16] is used to calculate the similarity between two users. If the distance is large, then the similarity of these two users is small. The Euclidean distance can be calculated as follows:

distance(p, q) =

J] (ptri - qtri)2 (1)

In Equation (1), ptri denotes the occurrence probability of the ith transition for user p. Similarly, qtri denotes the occurrence probability of the ith transition for user q. Based on the similarities of the Web users, we adapt the classic k-means clustering algorithm [17], to partition the Web users into k groups. The members of one group have the similar usage pattern.

Table 1. A Sample of a User-transition Matrix

Web Users Transition Transition2 Transition3

User-1 0.32 0 0.13

User-2 0.18 0.15 0.2

3.2. Building Usage Model

Two formalisms of statistical testing model, flat operation profile [3] and Markov usage model [2], can be used to describe the usage pattern of a Web application. A flat operational profile presents commonly used operations with the associated occurrence probabilities [2]. A Markov usage model presents commonly used operational units in a Markov chain [2], which consists of several states and state transitions. If we regard the Web pages as the states of a Markov usage model and regard the links as the state transitions, then the structure of a Web application and its usage pattern can be described easily by a Markov usage model. As we find in [21], a Markov usage model can be defined as: MC = {S, r, 6, s0, f), where S is the set of states; r is the set of transition labels; 6: S x r ^ S is next state function; so is the initial state; f is the final state. The transition label t e r is defined as a tuple {preSt, nextSt, count, prob, vars), where preSt is the present states of a state transition; nextSt is the next state of a state transition; prob is the probability of occurrence of the transition label t; count denotes the execution counts; and vars is the set of the required parameters when the transition occurs.

A Markov usage model can be represented by a directed graph, which includes some nodes and arcs. The nodes represent the states of a Web application. The arcs represent the transitions. The construction of a Markov usage model can be detailed as algorithm 1.

Algorithm 1 Constructing a Markov usage model from user sessions Input: the log entries of one user group. Output: a set of states and a set of transitions. Step 1. Get the set of states S. 1: for each log entry k do

2: if the previous (or next) state of li is not in S then 3: add the previous (or next) state into S. 4: end if 5: end for Step 2. Get the set of transition labels. 1: for each log entry k do

2: if Vt € r (the previous state, the next state and form parameters of k are not same as the ones of t) then

3: create a new transition label t'; make its previous state, next state and form parameters same as the

ones of li\ set the count of t' to 1. r = r u |t'\. 4: else

5: the count of t increases by one. 6: end if 7: end for

Step 3. Calculate the probability of the transition labels. 1: for each transition label ti do

2: find all the transition labels whose previous state is same as the one of ti, and put them into a set T;;

calculate the summation (denoted by counttotal) of count of transition labels in Ti. 3: The probability of ti can be calculated as: ti.count/counttotal. 4: end for

3.3. Test Cases Generation, Selection and Execution

The test cases generation phase of statistical Web testing is different from traditional Web coverage-based testing. In coverage-based testing, the test cases are generated according to the coverage criteria. However, in statistical Web testing, the abstract test cases are generated from a usage model randomly. Each abstract test case has its probability, that can be calculated by Equation (2), as is proposed by Gutjahr [20]:

Ptci = f~[ ptj (2)

In Equation (2), the abstract test case tci has k transitions; ptj denotes the probability of the jth transition. A threshold probability is used to control the size of the test suite. The abstract test cases generated above can not be executed directly. Testers need to design test data in order to execute these abstract test cases [19]. Sant et al. [14] proposed the data model which can provide the test data for abstract test cases. We agree with their approach to providing test data by building a data model. Our data model is similar to Sant's advanced data model, and it can be represented by the conditional probability of the given lastPage and current page.

P(name-values | lastPage, currentPage)

The executable test cases can be executed automatically by the HttpClient. During the process of test case execution, the output pages are downloaded to the specific folders.

3.4. Reliability Measurement

The reliability of Web applications can be defined as the probability of failure-free Web operation completions [18]. The Nelson model [9] is a typical model for assessing the reliability of a Web application. Most of the researchers, such as Kallepalli [2], Hao [11] and Tonella [12], adapt the Nelson model to estimate the reliability of a Web application. According to Kallepalli [2], the reliability R and the mean time between failures (MTBF), can be represented by the following two equations:

R = 1 - - (3)

MTBF = - (4)

In order to calculate R and MTBF, the failures - and the hits n are required. We adapt Tonella's definition of the failures of a Web application and check whether the output page is different from the expected page. However, generating the expected pages is a challenge for researchers because the element values of expected pages always depend on the data stored in the database. In our approach, we discuss two types of difference (structure difference and value difference) between output pages and expected pages. We can obtain the structure of each expected page by analyzing the design documents manually. In order to describe the value of each expected page, testers need to design SQL queries. By querying the database, we can get the value of expected pages. In our detection process, we initially check whether the output page lacks some tables defined in the corresponding expected page. Subsequently, we check the structure of each table. If we find that the structure of a table does not match the expected page, then we regard the structure of this output page as incorrect. If the structure of an output page is correct, we check whether the value matches the value of expected pages. If both the structure and value of an output page are correct, then we regard the output page as correct.

For the example transition "AdvSearch.jsp ^ Books.jsp", "Books.jsp" is the expected page displaying the books searched by customers. On this page, a result list should be shown to present the detailed information about books, such as its title, author, price, etc. The description of the expected structure of "Books.jsp" is shown as follows.

<table>

<name>Search Result</name>

<column>Title</column><column>Author</column><column>Price</column>

<column>Category</column><column>Image URL</column>

</table>

If the output page does not display "Price", the structure of this output page is incorrect. If the price of a book shown in the output page is $35.00, but the correct price is $33.00, then we consider that the value of this output page to be incorrect. Comparing both the structure and the values shown in the output pages with the expected pages, we can confirm whether the execution of a test case is correct.

4. Case Study

In this section, we present and discuss a statistical testing of a bookstore Web application. The bookstore can be downloaded from gotocode.com. We use Tomcat 6.0 as the Web server for our experiment. The Web application is deployed in a local area network (LAN), and we invited volunteers to access this Web application. The access information is stored in an access log file. We apply our approach, including the four steps mentioned in Section 3, to test this bookstore and to estimate its particular reliability under different usage patterns.

4.1. Partition Web Users

We identify the Web user by their IP address and User-Agent, and group the log entries of one user within 10 minutes into a user session. The access log contains 2,456 log entries, and 42 user sessions are found. We use one user's transitions and their occurrence probabilities to represent the characteristics of this user's usage pattern. The similarity of two users' usage patterns can be calculated by Euclidean distance.

Table 2. Specific Frequent Transitions of 5 Groups

Group No. Frequent transitions Group No. Frequent transitions

Group-1 OrderGrid ^ OrderRecord Group-3 ShoppingCart ^ ShoppingCartRecord

MemberGrid ^ MemberRecord ShoppingCartRecord ^ ShoppingCart Record_Form_ShoppingCartRecord_delete

OrderRecord ^ OrderRecord-Form_Orders -delete Default ^ Books

MemberRecord ^ MemberRecord-Form _Members_delete BookDetail ^ BookDetail_ Form_Order_insert

Group-2 AdminBooks ^ BookMaint Group-4 ShoppingCart ^ MyInfo

BookMaint ^ AdminBooks MyInfo ^ MyInfo_Form_update

BookMaint ^ BookMaint-Form_Books_update ShoppingCart ^ MyInfo

Group-5 Login ^ Registration

Registration ^ Registration_Form_Reg_insert

Registration ^ Default

We adapt the k-means clustering algorithm to partition Web users into five groups. We manually select some frequent occurrence transitions of each group to describe the characteristic of each group's usage pattern.

Table 2 shows some specific transitions which reflect the characteristics of each group's usage pattern. In Table 2, the transition "OrderRecord ^ OrderRecord_Form_Orders_delete", "Form" denotes the transition with form data submission. "Orders" is the name of a form. "Delete" is the name of a form action. We can conclude from Table 2 that the usage pattern of each group has its own characteristic. For example, the members of Group-1 are administrators who always delete some invalid orders and members of this bookstore. The members of Group-2 may be the administrators whose responsibility is to manager the information of books. The members of Group-3 are the customers who always purchase the books, but they are also likely to cancel orders. The members of Group-4 always modify their personal information. The members of Group-5 are the new users of this bookstore; they only browse few other pages after the registration.

4.2. Usage Model Construction

We build a Markov usage model for each user group because each group has its own particular usage pattern. A Markov usage model, which is used to represent the usage pattern of one group, can be built from the user sessions of one group. The approach to building the Markov usage model is introduced in Subsection 3.2. Fig.2 shows an example Markov usage model of this book store.

Fig. 2. An example Markov usage model of the book store

4.3. Test Cases Generation, Selection and Execution

We randomly generated the abstract test cases from the usage model. A threshold probability is assigned to control the size of the test suite. The initial threshold probability is equal to 0.1, and it is adjusted during

the process of generating abstract test cases. We generated 500 abstract test cases, but these abstract test cases cannot be executed directly. In order to execute these abstract test cases automatically, we adapt Sant's approach [14] to providing test data by building a data model. We use HttpClient to execute these test cases.

4.4. Reliability Measurement

We decide whether a test case execution is correct or incorrect by comparing the output pages with expected pages. If we find that one of the output pages is different from the corresponding expected page, then this execution is incorrect. We count the total hits n and the failures f for reliability measurement. Using Tonella's approach to testing the Web application statistically, we can only obtain the overall reliability. In our approach, we can estimate the particular reliability of a Web application. Fig. 3 shows the relationship between total hits n and the failures f. It is obvious that the counts of error are increasing with the cumulative hits. The slope of the curve for Group-1 is significantly larger than the overall curve. It depicts that failures occur more frequently under the usage pattern of Group-1 than under any other usage patterns.

Legend:

-©-B-

- Group-1 -Group-2 -Group-3 -Group-4 -Group-5

- Overall

0 500 1,000 1,500 2,000

Fig. 3. Hits versus Errors

20 15 10 5

200 300

Test Cases

Fig. 5. Relationship between test cases and MTBF

400 500

200 300

Test Cases

Fig. 4. Relationship between test cases and reliability

Under each group's usage pattern, the particular reliability of the bookstore can be estimated according to Equations (3) and (6). We estimate the reliability R and MTBF one time after we execute 50 test cases under different usage patterns. The test results are shown as Fig. 4 and Fig. 5.

We can conclude from Fig. 4 and Fig. 5 that the overall reliability ranges from 79% to 82.3%. However, the particular reliability for Group-1 ranges from 47.7% to 60.1%, which is much lower than the overall reliability. The overall MTBF ranges from 4.77 to 5.67 hits, but the MTBF for Group-1 ranges from 1.91 to 2.51 hits, which will cause the dissatisfaction of the users in Group-1 with the quality of this bookstore. Therefore, it is necessary for testers to estimate the particular reliability under different usage patterns.

5. Conclusion and Future work

Statistical testing is a cost-effective and a suitable approach to testing Web applications. The main contribution of this paper is an approach to estimating the particular reliability for each user group. In our approach, we adapt the k-means algorithm to partition Web users into several user groups whose members have similar usage patterns, and construct a usage model for each user group. By executing the test cases

generated from a Markov usage model, we estimate the particular reliability of a Web application. As shown in our case study, the reliability of a Web application under a particular usage pattern may be much lower than the overall reliability. Therefore, our approach is useful and meaningful for the testers and developers to fix existing errors and improve the reliability of a Web application. In our approach, testers are only required to describe the structure of expected pages and provide SQL queries to describe the value of expected pages. The comparison of expected and output pages is done automatically.

For future work, we plan to explore other efficient clustering algorithm. In addition, we ignore some special operations of Web users, such as clicking the "back" button, clicking the "forward" button, etc. These operations may affect the accuracy of our usage model, and they will be taken into account in the future.

Acknowledgment

We thank Ridha Khedri, Jason Jaskolka, and Mischa Geven for their helpful comments and suggestions. The authors C. Chen, Y. Chen, and H. Miao are partially supported by National Natural Science Foundation of China (NSFC) under grant No.61073050 and 61170044, and by Shanghai Leading Academic Discipline Project (Project Number: J50103). H. Wang is supported by IBM Canada R&D Centre and Southern Ontario Smart Computing Innovation Platform (SOSCIP) project.

References

[1] M. H. Alalfi, J. R. Cordy, and T. R. Dean, Modelling Methods for Web application Verification and Testing: State of the Art, Software Testing, Verification and Reliability, 19 (2009) 265-296.

[2] C. Kallepalli and J. Tian, Measuring and modeling usage and reliability for statistical Web testing, IEEE Transactions on Software Engineering, 27 (2001) 1023-1036.

[3] J. Musa, Operational profiles in software-reliability engineering, IEEE Software, 10 (1993) 14-32.

[4] C. Liu, D. Kung, and P. Hsia, Object-based dataflow testing of Web applications, in: proceedings of first Asia-Pacific Conference on Quality Software, 2000: pp. 7-16.

[5] S. Elbaum and G. Rothermel, Leveraging user-session data to support Web application testing, IEEE Transactions on Software Engineering, 31 (2005) 187-202.

[6] S. Elbaum, G. Rothermel and S. Karre, Improving Web Application Testing with User Session Data, in: proceedings of the 25th International Conference on Software Engineering, 2003: pp. 49-59.

[7] R. Wen, URL-Driven automated Testing, in: proceedings of second Asia-Pacific Conference on Quality Software, 2001: pp. 268-272.

[8] J. Tian, Testing the suitability of Markov chains as Web usage models, in: proceedings of the 27th Annual International Computer Software and Applications Conference, 2003: pp. 356-361.

[9] E. Nelson, Estimating Software Reliability from Test Data, Microelectronics and Reliability, 17 (1978) 67-73.

[10] A. L. Goel, Software reliability Models: Assumptions, Limitations, and Applicability, IEEE Transactions on Software Engineering, 11 (1985) 1411-1423.

[11] J. Hao and E. Mendes, Usage-based statistical testing of Web applications, in: proceedings of the 6th international conference on Web Engineering, 2006: pp. 17-24.

[12] P. Tonella and F. Ricca, Statistical testing of Web applications, Journal of Software Maintenance and Evolution: Research and Practice, 16 (2004) 103-127.

[13] P. Tonella and F. Ricca, Dynamic model extraction and statistical analysis of Web applications, in: proceedings of the Fourth International Workshop on Web Site Evolution, 2002.

[14] J. Sant, A. Souter, and L. Greenwald, An exploration of statistical models for automated test case generation, in: proceedings of the 3rd International Workshop on Dynamic analysis, 2005: pp. 1-7.

[15] S. Sprenkle, L. Pollock, and L. Simko, A Study of Usage-Based Navigation Models and Generated Abstract Test Cases for Web Applications, proceedings of 4th IEEE International Conference on Software Testing, Verification and Validation, 2011: pp. 230-239.

[16] M. M. Deza, and E. Deza, Encyclopedia of distances, Springer Berlin Heidelberg, 2009.

[17] J. Han, M. Kamber, and J. Pei, Data mining: concepts and techniques, Morgan kaufmann, 2006.

[18] Z. Li and J. Tian, Analyzing Web logs to identify common errors and improve Web reliability, in: proceedings of the International Conference e-Society, 2003: pp. 235-242.

[19] C. Chen, H. Miao, and Y. Chen, Concept Analysis based Approach to statistical Web testing, Essays in Honor of He Jifeng on the Occasion of His 70th Birthday, LNCS 8051. Springer (2013).

[20] W. J. Gutjahr, Software dependability evaluation based on Markov usage models, Performance evaluation, 40 (2000) 199-222.

[21] J. Yan, J. Wang and H. Chen, Deriving software markov chain usage model from UML model, Journal of Software, 16 (2005) 1386-1394.