r28 - 04 Jan 2008 - 16:04:48 - DanSteinickeYou are here: OSAF >  Projects Web  >  QualityAssuranceTeam > TestScriptRecordingAndPlayback

Test Script Recording and Playback Project

Meeting Notes

Goals

Provide an easy way to create test scripts by recording keystrokes and mouse actions.

Usage

Script recording/ playback is accessed from the menu Tools >> Scripting. Scripts can also be played back by starting Chandler with the command line option like this:
release/RunChandler -cen  --recordedTest=TestName  

All recorded tests in the tools/cats/recorded_scripts directory can be run using rt.py like this:

tools/rt.py -r 

To run a single test use rt.py like this:

tools/rt.py -t TestNameWithoutExtension

Human readable comments in auto generated code

The top of the file for a recorded script contains a big comment that tries to describe the actions the script takes. You can still tell that this is written by a computer but its a lot easier to read than the script itself. Below is a sample of the start of a recorded script. Notice that at the end of each line in the human readable part there is a number. These numbers correspond to the numbers at the start of each line in the real script.
    """
        Choose menu 'Collection > New' (0)
        Type u'testTriageButton' (3)
        Left Mouse Down in ApplicationBar (57)
        Left Mouse Down in ApplicationBar (59)
        Choose toolbar button 'New' (61)
        Left Mouse Down in HeadlineBlockAEEditControl (63)
        Left Mouse Down in HeadlineBlockAEEditControl (65)
        Type u'Test Triage Button' (69)
        Left Mouse Down in NotesBlock (127)
        Left Mouse Down in DashboardSummaryViewGridWindow (130)
        Left Mouse Down in TriageStamp (132)
        Left Mouse Down in DashboardSummaryViewGridWindow (135)
        Left Mouse Down in TriageStamp (137)
        Left Mouse Down in DashboardSummaryViewGridWindow (140)
        Left Mouse Down in TriageStamp (142)
    """
    wx.GetApp().RunRecordedScript ([
        (0, wx.CommandEvent, {'associatedBlock':'NewCollectionItem', 'eventType':wx.EVT_MENU, 'sentTo':u'MainViewRoot'}, {}),
        (1, wx.FocusEvent, {'eventType':wx.EVT_SET_FOCUS, 'sentTo':u'SidebarAttributeEditor'}, {}),
        (2, wx.KeyEvent, {'eventType':wx.EVT_KEY_DOWN, 'sentTo':u'SidebarAttributeEditor', 'lastWidgetValue':u'Untitled'}, {'m_rawCode':84, 'm_keyCode':84, 'm_x':166, 'm_y':36, 'UnicodeKey':84}),
        (3, wx.KeyEvent, {'eventType':wx.EVT_CHAR, 'sentTo':u'SidebarAttributeEditor', 'lastWidgetValue':u'Untitled'}, {'m_rawCode':116, 'm_keyCode':116, 'm_x':166, 'm_y':36, 'UnicodeKey':116}),
        (4, wx.KeyEvent, {'eventType':wx.EVT_KEY_UP, 'sentTo':u'SidebarAttributeEditor'}, {'m_rawCode':84, 'm_keyCode':84, 'm_x':166, 'm

Active Bugzilla Bugs

6759 P3 ASSIGNED Functional test runner should fail any test that leaves a dialog up dan@osafoundation.org
9450 P3 NEW Get recorded script test framework to do collections outside of chandler python mikeal@osafoundation.org
11633 P3 NEW recTestQEDefaults fails john@osafoundation.org
11634 P3 NEW recTestAutoTriage fails john@osafoundation.org
9302 P4 REOPENED Recorded script assertion error dragging a recurring event into a new collection john@osafoundation.org

Known Issues & things you need to know to record scripts successfully

  • To reliably play back on all platforms scripts need to be recorded on Windows XP.
  • Scripts assume the starting view/ focus is the same as when the script was recorded.
    • Recommend starting with --create immediately before recording a script.
    • Be aware that the appbar 'New' button will create an event at the current date/ time - on playback this will not be the same as when the script was recorded so always set date/ time after creating events with the new button.
  • To verify a text value append some more text to it, this will cause the scripting framework to verify its text value.
  • Tabbing into Notes field (and possible other fields) does not result result in existing text getting selected. If you tab in and want to overwrite existing text use select all before typing new text.
  • Clicking multiple times on a triage button does not work (plays back too fast).
  • Date/time fields are not verified because the values are different in different locales/timezones (this could be fixed)
  • Detail view timezone value is not verified.
  • Scrolling calendar window does not work.
  • On Macs it is important for the preferences setting "preferences/ keyboard/ shortcuts/ Full keyboard access" be set to all.
  • On Parallels/winXP clipboard synchronization needs to be turned off. This is done by right clicking the parallels icon in the windows system tray.

Test development Issues

  • Need documentation
  • if action is not getting recorded/ verified correctly check the wxEventType and focusEvent lists near the top of osaf/frameworks/script_recording/__init__.py to see if the items associated with your action are in the list.
  • Need a way to dynamically generate strings/ dates. This blocks:
    • importing a .ics file because the path to the file may be in a slightly different place.
    • using dates relative to todays date
  • Need a way to dynamically generate user actions in a loop. Through testing involves running many tests with only slightly different actions, with this framework each action would have to be recorded, in a more conventional framework loops could be used to have the test repeatedly do the same thing with slight differences each time. Examples would be creating many events with different start and end times or creating recurring events with different recurrence rules. Doing these sort of thing seems very necessary to enable the automating of the TestSpecs? which contain numerous actions repeated with slight variations.
  • Need a way to test the framework itself. In order to have confidence that the tests will catch the types of errors we think it can we need to be able to have tests that by design encounter all the types of errors we expect to be able to test for and make sure the framework reports failure in all cases.
  • Need a way to slow playback down enough that someone can watch it run and tell what is happening

Things tests should be able to validate

  • existence by name
    • does a
      • collection
      • event
      • message
      • task
      • note
        • named 'some String' exist?
  • existence by date/time in calendar view
    • does an event exist at such and such date/time
  • selection
    • is a
      • collection
      • event lozenge
      • summary table row
        • selected?
  • view state:
    • is Chandler in:
      • All
      • Mail
      • Task
      • calendar
        • view state?
  • stamping state
    • is the selected item stamped as:
      • Mail
      • Task
      • Event
  • table row state
    • does the current table row show:
      • as unread
      • as read
      • as task
      • as having a reminder
      • as triaged to NOW
      • as triaged to LATER
      • as triaged to DONE
      • in the NOW section
      • in the LATER section
      • in the DONE section
  • table row column value
    • does selected table row column
      • who
      • title
      • date
        • contain the value ....
  • detail view field values
    • is value ..... in the detail view field:
      • from
      • to
      • cc
      • bcc
      • send as
      • byline
      • title
      • location
      • all-day check box
      • start date
      • start time
      • end date
      • end time
      • time zone
      • status
      • recurrence
        • recurrence end date
      • alarm
      • note
      • appears in
  • detail view field existence
    • does the detail view field:
      • from
      • to
      • cc
      • bcc
      • send as
      • location
      • all-day check box
      • start date
      • start time
      • end date
      • end time
      • time zone
      • status
      • recurrence
        • recurrence end date
      • alarm

toggleopenShow attachmentstogglecloseHide attachments
Topic attachments
I Attachment Action Size Date Who Comment
pngpng tab_settings.png manage 71.4 K 16 Nov 2007 - 17:19 DanSteinicke  
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r28 < r27 < r26 < r25 < r24 | More topic actions
 
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.