`
`CONFIGIT 1007
`
`1
`
`
`
`Formal ~. GUI Model Test Case t [ ........................... ~ ............................. ] gun #~o i ~fExpected-state~ { ,~form'ff~nfrom "| Generator | executing GU1 i [ ~[r^..~'~. L iAc~al{ l~xecuuon { i._O..t~l ~'| I Stat~ ~ Monitor J J ........... ± ............................... Verdict Figure 1: An Overview of the GUI Oracle. state description. Finally, we need to design a mechanism to automatically compare the expected state with the state of the executing GUI. In this paper, we present a technique to develop an auto- mated GUI test oracle. An overview of the oracle is shown in Figure 1. The oracle uses a formal model that is de- veloped by the oracle designer from the GUI specifications. The model is composed of the GUI objects and a set of prop- erties for those objects. GUI actions are represented in the model by their preconditions and effects. The oracle auto- matically derives the expected state using the model and the actions from a test case. Likewise, the actual state is also described by a set of objects and properties typically found in a GUI toolkit or specialized GUI language. The oracle obtains the actual state information from an execution mon- itor. A verifier in the oracle then automatically compares the two states and determines if the GUI is performing as expected. We implemented our technique in our GUI testing system PATHS (the Planning Assisted Tester for graphical user interface S.ystems), and show how we were able to facil- itate automation of the GUI test oracle by exploiting the AI planning-based tools already present in PATHS. We experi- mentally evaluated the oracle on a version of Microsoft Word Pad and provide timing results that establish the feasibility of our approach. In particular, the important contributions of the method presented in this paper include the following. • We define a formal model of a GUI derived from spec- ifications that is useful in testing. In this paper we demonstrate its usefulness in developing oracles. • Our oracle is general in that it will work for any GUI as long as an appropriate model can be established. The oracle is also portable across platforms since it depends on properties that can be acquired from GUI toolkits or special programming language features. • The technique allows reuse of operator definitions that commonly appear across GUIs. These definitions can be maintained in a library and reused to help develop oracles for GUIs. • We show our oracle creation process as a natural exten- sion of our already implemented planning-based test- case generation system. We reuse the planning oper- ators defined for test-case generation and apply them in a unique way to create oracles. In the next section, we describe our GUI model. In Sec- tion 3, we show how this model is used to determine the expected state sequence of the GUI for a test case. In Sec- tion 4, we show how to compare the expected state informa- tion with the executing GUI's actual state. In Section 5, we demonstrate how the oracle is used in testing an example GUI. Section 6 describes our implementation and presents experimental results. We present related work in Section 7 and concluding remarks in Section 8. 2. MODELING THE GUI We begin by describing how a GUI can be formally modeled, and then show how that model can be used to compute expected states of the GUI. 2.1 Objects and Properties We model a GUI as a set of objects, (window, menu, but- ton, text, etc.), a set of properties of those objects (back- ground color, font, is-open, etc.), and a set of actions that change the properties of certain objects (set-background- color, etc.). Each GUI will use certain types of objects with associated properties; at any specific point in time, the GUI can be described in terms of the specific objects, or GUI elements that it currently contains, and the current values of their properties. More formally, we model a GUI at a particular time t as: its objects O = {oi, o2, ... , ore}, i.e., the objects the GUI currently contains, and the properties P = {pl, p2 .... , pl} of those objects. Each property p~ is an ni-ary Boolean relation, for ni _> I, where the first argument is an object Ol E O. If n~ > 1, the last argument may be either an object or a property value, and all the intermediate arguments are objects. The property value is a constant drawn from a set associated with the property in question: for instance, the property "background-color" has an associated set of values, (white, yellow, pink, etc.}. We assume a distinguished set of properties, the ob- ject types, which are unary relations, e.g., %¢indow" or "button". Thus we might specify the state of a (extraordinarily sim- ple) GUI at some particular time by noting that it currently has two window objects, w17 and w29, for which the following properties hold: window(wl7), window(w29), background- color(wl7, red), is-cmcrent(wl7). The state of a GUI at a particular time is everything that is currently true of it. So a description of the state would contain information about the types of all the objects currently extant in the GUI, as well as all of the properties of each of those objects. There are several points that should be noted about our description of properties. First, properties are relations, not functions, and so there may sometimes be multiple values for the same property of a given object. For example, there may be multiple objects in a window. Next, properties as we have defined them are fluents [8], i.e., relations which are true in some situations (or states of the world) and not others. An everyday example of a fluent is the relation president (US, Clinton), with the obvious meaning, where the state it is evaluated in is the state of the real world. Our fluents are evaluated with respect to a state of the GUI. Finally, note that a fluent may be undefined in some states, for example, 31
`
`2
`
`
`
`president(US, Dole) in the state of the world in the year 1567, or background-color(w24, blue) in the state of a GUI immediately after window w24 has been destroyed. In practice, we can determine the set of object types and properties for our GUI model in several different ways. One approach would be manual examination of the GUI: we look at it, and write down all the object types and properties we can discover. This approach is prone to incompleteness, es- peciaUy since GUIs may have hidden properties that must be checked during verification. For example, the tab order of windows in a GUI (the order in which windows receive input focus when the Tab key is pressed) is a property that is not visible. A second approach is to derive the objects and properties directly from the GUI's specifications, which will describe them either directly or implicitly within the descriptions of GUI actions. A third approach is to examine the language or toolkit used to develop a particular GUL For example, if the GUI was developed using the Java language, then the GUI objects would be instances of the swing GUI components of the Java swing package, and the properties would correspond to the instance variables (also called data members in C++) of each object. Visual programming envi- ronments provide a more direct interface to properties. For example, Borland's C++ Builder presents the properties as a table for the currently selected object. The third approach can lead to a larger set of object types and properties than does the second. This is because the set of object types and properties made available by a lan- guage or toolkit may not all be used in the construction of a particular GUI. For example, one might use Borland's C++ builder to construct a simple GUI in which the user is not permitted to manipulate the text color, and in which the text color does not influence the execution of any other action. (In fact, Microsoft's NotePad is like this.) Thus, if one establishes the set of properties from the GUI's specifi- cations, text color will not be amongst the properties mod- eled, whereas if one establishes it from the toolkit used for development, text color will be included as a property in the model. We thus distinguish between the complete set of properties for a GUI, which are all those that would be identified by our third (language/toolkit-based) approach, and the reduced set, which includes only those that would be identified by our second (specifications-based) approach. Note that the reduced set is always a (possibly improper) subset of the complete set of properties. 2.2 Actions The state of a GUI is not static; actions are used to change it over time. We model actions as state transducers, i.e., we define an action as follows: Definition: The actions A = {al, a% ... , an} associated with a GUI are functions from one state of the GUI to another state of the GUI. [] Actions may be parameterized, e.g., set-background-color ( w, x ). Whenever the action set-background-color( w19, yellow ) is executed in a state in which window w19 is open, the background color of w19 should become yellow (or stay yellow if it already was), and no other proper- ties of the world should change. This example illustrates that, typically, actions can only be executed in some states; set-background-color( w19, yellow )cannot be executed when window w19 is not open. We use the notation sj = [s~, a] to denote that sj is the state resulting from the execution of action a in state si. We can string actions together into sequences. We will say that al ; a2;... ; an is a legal action sequence for initial state so iff there exists a sequence of states, so; sl,... ; sn such that si = [s~-l,a~] for i = 1,... ,n. Extending the notation above, we use sj = [si,al;a2;...an], where al;a~;... ;an is a legal action sequence, to denote that sj is the state that results from executing the specified sequence of actions starting in state s~. Definition: A GUI test case is a pair < so, al ; a2; • .. an >, consisting of an initial state and a legal sequence of ac- tions for that state. [] We model actions using their descriptions in the GUI spec- ifications: after all, the purpose of verification is to ensure that the implementation of the actions matches the expected behavior promised in the specifications. In the next section, we provide further details about modeling actions. 3. DERIVING EXPECTED STATE We can now see how the model of the GUI can in principle be used to determine the expected state of a GUI after the complete or partial execution of any test case. Recall that actions are modeled as state transducers. For any test case < so, al; a2; •. • an >, the sequence of states sl; s2; • • • sn such that si = [s~-l,al] for i = 1,... ,n represents the expected state of the GUI after each action is executed, starting in so. The question is how, in practice, to compute these expected states. It is of course infeasible to give exhaustive specifications of the state mapping for each action: in principle, as there is no limit to the number of objects a GUI can contain at any point in time, there can be infinitely many states of the GUI3 Thus, we adopt the technique of modeling GUI actions using operators, which specify their preconditions and effects: Definition: An operator is a 3-tuple <Name, Precondi- tions, Effects> where: • Name identifies an action and its parameters. • Preconditions is a set of positive ground literals 2 p(argl,... ,argn), where p is an n-ary property (i.e., p E P). • Elf e ct s is also a set of positive or negative ground literals p(argl ,... , argn), where p is an n-ary prop- erty (i.e., p E P). [] 1Of course in practice, there are memory limits on the ma- chine on which the GUI is running, and hence only finitely many states actually possible, but the number of possible states will be extremely large. 2A literal is a sentence without conjunction, disjunction or implication; a literal is ground when all of its arguments are bound; and a positive literal is one that is not negated. It is straightforward to generalize the account given here to handle partially instantiated literals. However, it needlessly complicates the presentation for this paper. 32
`
`3
`
`
`
`We write Pre(Op) and EJy(Op) to represent the set of pre- conditions and effects, respectively, for operator Op. An operator is applicable in any state si in which all the literals in Pre(Op) are true. In the resulting state sj, all of the pos- itive literals in Eff(Op) will be true, as will all the literals that were true in si except for those that appear as negative literals in Eff(Op). The scheme for encoding operators we use is the same as what is standardly used in the AI plan- ning literature [14, 22, 23]; the persistence assumption built into the method for computing the result state is called the STRIPS assumption. A complete formal semantics for op- erators making the STRIPS assumption has been developed by Lifschitz [7]. The STRIPS-style of encoding operators also makes it fairly easy to derive result state sj = [si, a], via simple additions and deletions to the list of relations representing state si. For example, if we were to define an operator for the set-bac- kground-color action, then we would get the following op- erator definition: Name: set-background-color(wX: window, Col: Color) Preconditions: is-current (wX), background-col- or(wX, oldCol), oldCol ~ Col Effects: background-color(wX, Col) Going back to our simple example of the GUI in which the following properties were true: window(wl7), window(w29), background-color(wlT, red), is-curren~(wl7). If we ap- plied the above operator, with variables bound as set-back- ground-color( w17, blue ), we would get the following state: window(wlT), window(w29), background-color(wiT, blue), is-current (w17), i.e., the background color of win- dow w17 would change from red to blue. The next state is obtained from the current state Sc and the operator's effects e as follows: 1. Delete all literals in Sc that unify with a negated literal in e, and 2. add all positive literals in e. Thus, using a formal model of a GUI, we can derive the ex- pected state, given an initial state and a sequence of actions. Given that GUI specifications can describe the intended be- havior of actions in terms of their preconditions and effects [5, 4], it is relatively straightforward for the test designer to construct operators for the GUI model. In fact, as we will see later, the operators can also be used in other aspects of testing. 4. STATE COMPARISON We have just described how to model a GUI and use that model to derive the expected state. Now we turn to the question of how to compare that information to the actual state. The simplest approach is manual comparison. One manually executes a test case, and after each step, manually compares the appearance of the GUI with the expected state at that time. Manual verification has at least two problems: (1) it is labor intensive, and (2) often the GUI state includes "hidden" properties that are not visually accessible. Our goal is therefore to automate the process of extracting actual GUI state information in a form that is suitable for comparison with the expected state description. We define an execution monitor to be a process that, given an exe- cuting GUI, returns the current values of all the properties in the complete set for the GUI. Once the actual values of properties for an element or elements are known, the verifier can compare them against the expected values, to determine if they are equal. We, therefore define the verifier to be a process that compares the expected state of the GUI with the actual state and returns a verdict of equal or not equal. The remaining question, then, is what properties should be compared during the verification process. There are several possible answers to this question, and the decision amongst them establishes the level of testing performed: Changed-Properties Verification: Here, comparison is made only for those properties that were expected to change as a result of the immediately preceding action. That is, if action a was just executed, only the proper- ties that are included in Eft(a) are compared against their expected values. Although efficient, this level of testing will fail to detect changes to properties that change when they are not expected to change. For ex- ample, if the background color of a window changes, but it was not expected to change, the error would go unnoticed. Relevant-Properties Verification: Here, all the proper- ties in the reduced property set (see Section 2.1 above) are checked. Recall that the reduced property set in- cludes all the properties that the current GUI is ever supposed to access. This is, thus, a much more ex- tensive level of testing than changed-properties verifi- cation, but it may still fail when some GUI property P changed in the executing GUI, but P was not a part of the GUI specification. For example, consider a GUI for a plain-text editor, e.g., MS NotePad in which users cannot change the text color. If some action in the test case has the unintended effect of changing the text color, then this error would go unnoticed, since the color information was not encoded in the expected state. Complete-Properties Verification: Here, a check is made for all the properties that a language or toolkit pro- vides for a GUI. Recall that the verifier has access to the complete set of properties. The only problem is the absence of an expected state to compare against all these additional properties. The currently available expected state encodes only the reduced property set. To address this problem, before the test case is exe- cuted, a baseline complete expected state of the GUI is created. During test-case execution, the comparisons are done between the GUI!s actual state and the up- dated complete expected state. In practice, the test designer can choose a combination of the above levels of testing. For example, the verifier can per- form changed-properties verification after each test action and complete-properties verification after every 10 actions. 33
`
`4
`
`
`
`a a al ? ~ SelectFUe('f4,doc') ] Figure 2: The Example GUI. We now have all the necessary mechanisms to develop an automated test oracle for GUIs. 5. A GUI EXAMPLE In this section we show, through an example, how a GUI is tested using an automated test oracle. Figure 2 presents a small part of the Microsoft WordPad's GUI. This GUI can be used for loading text from files, ma- nipulating the text (by cutting and pasting) and then saving the text in another file. At the highest level, the GUI has a pull-down menu with two actions (File and Edit). The GUI user can execute the GUI actions to make other elements available. For example clicking on File opens a menu with New, Open, Save and SaveAs actions. Edit opens a menu with Cut, Copy, and Paste actions. Open and SaveAs open windows with several more actions. These actions are used to traverse the directory hierarchy and select a file. The up button moves up one level in the directory hierarchy and clicking on files and directories is used to select files or enter subdirectories respectively. The window is closed by clicking on either Open or Cancel. We assume that the GUI's test cases are given. Recall that we defined a test case as a pair (So, al; a2; a3; ...; a,), where So is the initial state and al;a2;a3;...;a,~ is an action se- quence. Consider, for example, the sequence of actions to be applied to our version of the WordPad software shown in Figure 3. This sequence of actions transforms the GUI from the initial state So shown in Figure 4(a) to the one shown in 4(b). Figure 4(a) shows a collection of files stored in a directory hierarchy. When the actions are executed on the GUI, the new document shown in Figure 4(b) is created and then stored in file f$.doc in the /Root/Latex/Samples directory. 5.1 The Oracle Designer To test the above GUI, an Oracle Designer uses the GUI specifications to develop a formal model of the GUI. The Figure 3: An Action Sequence for our Version of the WordPad Software Property in contains contains file currentFile currentFont font isCurrent onScreen selectedFile selectedText Args File, Text ParentDir, Dir Dir, File File Font, Style, Size Text, Font, Style, Size Dir Text File Text Semantics File contains Text FarentDir contains Dir Dir contains File The current file is File The current font is Font, style is Style, and size is Size Text is in Font, Style, and Size Dir is the current direc- tory Text is displayed on the screen File is selected Text is highlighted Table 1- Some Properties, their Parameters, and Semantics. rest of the process, i.e., deriving an expected state sequence for each test case, executing the test case, extracting the actual state, and verifying its outcome of the test case is handled automatically. The first step in deriving the expected state is for the oracle designer to use the GUI specifications to identify the prop- erties of the elements of the GUI. The semantics of some properties used in this example are shown in Table 1. The columns show the property name, the parameters, and the semantics of each property. The oracle designer then rep- resents the initial state (Figure 4) in terms of the identi- fied properties as shown in Figure 5. The initial state de- scribes the file structure (using the properties contains() and containsFile()), and the contents of the file fl.doc us- ing the property in(). Additional properties are used to describe the fonts, current file, and the current directory. By using the actions described in the specifications, the ora- cle designer defines the preconditions and effects of the oper- ators. Figure 6 shows an example of an operator called Open, 34
`
`5
`
`
`
`Root priv*te i a i-~ ~.doc---- i-~ Courses I' C3 Figures ~-~ ~x ~.. ":~]~ Samples L.~ f4.doc report.dcc Htrnl ~..C] gif This is the te~ that needs to be rnodele~ (a) ]~ f4°doc must be s~ored ~ ..................................................................................................................................... oo_ Figure 4: The Action Sequence of Figure 3 Trans- forms the GUI from: (a) the Initial State, to (b) the Final State representing the Open action from the File menu. The op- erator Open takes two parameters, dir and file. The oper- ator is available only if its precondition, containsf ile (dir, file) is TRUE, i.e., directory "dir" contains the file "file". The effects of applying this operator are that the currentFile value is modified, all the objects on the screen are deleted, and all the objects in the file are displayed on the screen. Quantifiers and conditional statements are used to make the notation concise and intuitive. They are later replaced with their expansions when the expected state is derived. 5.2 The Automated Oracle Using the operators defined by the oracle designer, the auto- mated oracle derives the GUI's expected state corresponding to the given test case. The expected state sequence is de- rived from So by using the method outlined in the previous section. The next expected state is automatically obtained by applying al on So, i.e., $1 = [So,a1]. The process is repeated until the entire expected state sequence has been derived. For example, consider the expected state shown in terms of properties for actions a4 and as in Figure 7. (Note that the shown subsequence is a part of the sequence shown in Figure 3) The expected state corresponding to a4 is repre- sented as $4. The GUI's state changes after action as (Open) is executed. The new state obtained is $5. The changes are highlighted using bold font. As mentioned earlier in the description of the Open operator, the currentFile value changes, and the objects from the file are now displayed on the screen (using the property onScreen()). The test case and expected state sequence shown in Figure 7 have all the necessary components to carry out a successful test run and can be used for manual testing. The tester alternates between the test-case actions and the expected initial: contains(root private) contmns(privatc LaZx) contains(Lamx Sampl~) containsfile(Samples f4.doc) containsfile(privatc fl.doc) currcntF~mt('rimes Normal 12pt) in(fl.doc "This") font('~is" Times Normal 12pt) in(fl.doc "is the") font("is the" Times Normal 12p0 in(fl.doc "text") font("text" Time~ Normal 12pt) in(fLdoc "that") font("~af' Times Normal 12pt) in(fl.doc "needs to") font("needs to" Times Normal 12pt) in(fl.doc "be modeled.") fontC'be modeled" Times Normal 12pt) Figure 5: Representing the Initial State. Operator Name Open(dir: DIRS, file : PILES) Preconditions containsfile(di r, file) Effects /* The current file is now file */ currentFile(f £ 1 e) /* Now there are no objects on the screen */ ~0nScrccn(obj) V obj e OBJEC]'S /* All objects in file are now on the screen */ onScreen(obj) V obj e OBJECTS I in(file, obj) Figure 6: An Operator. state, executing the input events in the test case and check- ing the GUI state by verifying each property. 3 However, we have fully automated test execution by implementing the execution monitor and the verifier. Now that the expected state has been automatically derived, it is compared with the actual state. The actual state of the executing GUI is obtained from the execution monitor, which maintains a list of all the properties of our version of the WordPad software. At each step in the test case, the verifier uses the values of all these properties to check them for correctness. Thus, in our example, the expected state shown in $4 and $5 will be automatically compared with the actual GUI state when the test case is executed. 6. IMPLEMENTATION In this section, we first give an algorithm that shows how the components of the test oracle are used when testing the GUI. We also show the details of how the expected state is derived from the current state. Then we describe an implementation of our oracle and results of experiments to determine the time needed to derive the expected state and execute the verifier and execution monitor. SNore that since the expected state has been derived from the specifications the names of properties may not match those in the toolkit. Renaming of properties may be needed at this step to match those used in the toolkit. 35
`
`6
`
`
`
`a4 comaias(Latex Samples) contains(t~tvate Latex) contah)s(root private) conlainsfile(private fl.doc) conlainsl~(SampMs f4.doc) currcntFon~(Times Normal 12p0 font("be raodclcd" Times Normal 12pt) font("ls the" Times Normal 12pt) font("needs to" Times Normal 12p0 font('lexC ~ Nom3a112pt) font('thaC' Times Normal 12p0 font("ThiF Times Normal 12p0 in(fl.doc "I~ mode]rA") in(fl.doc "is the") in(fl.doc "needs to') in(fl.doc "~xC) tn(fl.doc "that") Jn(fl,doc '7his") isOmeat(private) scle~edFile(fl.doc) a~ a 5 ~ ~ [ Select"text" ] "l ("'This'"') contains(Latex Samples) conUims(prlva~ Latex) contains(root private) containsfll¢~rivate fl.d0c) containsfile(Samplcs f4.doc) currentFlle(fl.doc) curtel3tFont(Timcs Normal 12p0 font(*be modeled" Times Normal 121)0 font("is the" Tim~ Normal 12p0 font("necds to" Times Nomm112pt) font("text" Times Nomm112pt) font("that" Times Normal 12pt) font("This" Times Normal 12pt) in(fl.doc "be modeled') in(fl.doc "b the") in(fl .doc "needs to") in(fl.doc "text") in(fl .dec "that") ia(fl.doc "This") izCerrent(private) onScreen("be