Bug 708309 - Do not use zipfile.extract in mozbase components for peptest r=jhammel a=test-only
authorAndrew Halberstadt <ahalberstadt@mozilla.com>
Tue, 13 Dec 2011 13:53:51 -0600
changeset 82517 300849c3dd10038d19a6cad6b10d8bae780ba0b7
parent 82516 0414fe2f9d735d645bfcffb27bcb2be95403e8c3
child 82518 7604f82c29d0bfcf7e1763fb31587f5cacf1c86f
child 82995 7d46a7c3ac41c3666b3421ca99220cf6e5909b9f
push id21642
push userctalbert@mozilla.com
push dateTue, 13 Dec 2011 19:55:38 +0000
treeherdermozilla-central@300849c3dd10 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersjhammel, test-only
bugs708309
milestone11.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 708309 - Do not use zipfile.extract in mozbase components for peptest r=jhammel a=test-only
testing/mozbase/manifestdestiny/README.md
testing/mozbase/mozdevice/mozdevice/devicemanagerADB.py
testing/mozbase/mozinstall/mozinstall/mozinstall.py
testing/peptest/peptest/extension/resource/pep/testsuite.js
testing/peptest/peptest/pepprocess.py
testing/peptest/peptest/peputils.py
testing/peptest/peptest/runpeptests.py
testing/peptest/setup.py
--- a/testing/mozbase/manifestdestiny/README.md
+++ b/testing/mozbase/manifestdestiny/README.md
@@ -1,15 +1,15 @@
 Universal manifests for Mozilla test harnesses
 
 # What is ManifestDestiny?
 
 What ManifestDestiny gives you:
 
-* manifests are (ordered) lists of tests
+* manifests are ordered lists of tests
 * tests may have an arbitrary number of key, value pairs
 * the parser returns an ordered list of test data structures, which
   are just dicts with some keys.  For example, a test with no
   user-specified metadata looks like this:
 
    [{'path':
      '/home/jhammel/mozmill/src/ManifestDestiny/manifestdestiny/tests/testToolbar/testBackForwardButtons.js',
      'name': 'testToolbar/testBackForwardButtons.js', 'here':
@@ -18,41 +18,49 @@ What ManifestDestiny gives you:
 
 The keys displayed here (path, name, here, and manifest) are reserved
 keys for ManifestDestiny and any consuming APIs.  You can add
 additional key, value metadata to each test.
 
 
 # Why have test manifests?
 
+It is desirable to have a unified format for test manifests for testing
+[mozilla-central](http://hg.mozilla.org/mozilla-central), etc.
+
+* It is desirable to be able to selectively enable or disable tests based on platform or other conditions. This should be easy to do. Currently, since many of the harnesses just crawl directories, there is no effective way of disabling a test except for removal from mozilla-central
+* It is desriable to do this in a universal way so that enabling and disabling tests as well as other tasks are easily accessible to a wider audience than just those intimately familiar with the specific test framework.
+* It is desirable to have other metadata on top of the test. For instance, let's say a test is marked as skipped. It would be nice to give the reason why.
+
+
 Most Mozilla test harnesses work by crawling a directory structure.
 While this is straight-forward, manifests offer several practical
 advantages::
 
 * ability to turn a test off easily: if a test is broken on m-c
   currently, the only way to turn it off, generally speaking, is just
   removing the test.  Often this is undesirable, as if the test should
   be dismissed because other people want to land and it can't be
   investigated in real time (is it a failure? is the test bad? is no
   one around that knows the test?), then backing out a test is at best
   problematic.  With a manifest, a test may be disabled without
   removing it from the tree and a bug filed with the appropriate
   reason:
 
-   [test_broken.js]
-   disabled = https://bugzilla.mozilla.org/show_bug.cgi?id=123456
+     [test_broken.js]
+     disabled = https://bugzilla.mozilla.org/show_bug.cgi?id=123456
 
 * ability to run different (subsets of) tests on different
   platforms. Traditionally, we've done a bit of magic or had the test
   know what platform it would or would not run on. With manifests, you
   can mark what platforms a test will or will not run on and change
   these without changing the test.
 
-   [test_works_on_windows_only.js]
-   run-if = os == 'win'
+     [test_works_on_windows_only.js]
+     run-if = os == 'win'
 
 * ability to markup tests with metadata. We have a large, complicated,
   and always changing infrastructure.  key, value metadata may be used
   as an annotation to a test and appropriately curated and mined.  For
   instance, we could mark certain tests as randomorange with a bug
   number, if it were desirable.
 
 * ability to have sane and well-defined test-runs. You can keep
@@ -60,42 +68,42 @@ advantages::
   (sub)manifests as appropriate to your needs.
 
 
 # Manifest Format
 
 Manifests are .ini file with the section names denoting the path
 relative to the manifest:
 
- [foo.js]
- [bar.js]
- [fleem.js]
+    [foo.js]
+    [bar.js]
+    [fleem.js]
 
 The sections are read in order. In addition, tests may include
 arbitrary key, value metadata to be used by the harness.  You may also
 have a `[DEFAULT]` section that will give key, value pairs that will
 be inherited by each test unless overridden:
 
- [DEFAULT]
- type = restart
+    [DEFAULT]
+    type = restart
 
- [lilies.js]
- color = white
+    [lilies.js]
+    color = white
 
- [daffodils.js]
- color = yellow
- type = other
- # override type from DEFAULT
+    [daffodils.js]
+    color = yellow
+    type = other
+    # override type from DEFAULT
 
- [roses.js]
- color = red
+    [roses.js]
+    color = red
 
 You can also include other manifests:
 
- [include:subdir/anothermanifest.ini]
+    [include:subdir/anothermanifest.ini]
 
 Manifests are included relative to the directory of the manifest with
 the `[include:]` directive unless they are absolute paths.
 
 
 # Data
 
 Manifest Destiny gives tests as a list of dictionaries (in python
@@ -104,17 +112,17 @@ terms).
 * path: full path to the test
 * name: short name of the test; this is the (usually) relative path
   specified in the section name
 * here: the parent directory of the manifest
 * manifest: the path to the manifest containing the test
 
 This data corresponds to a one-line manifest:
 
- [testToolbar/testBackForwardButtons.js]
+    [testToolbar/testBackForwardButtons.js]
 
 If additional key, values were specified, they would be in this dict
 as well.
 
 Outside of the reserved keys, the remaining key, values
 are up to convention to use.  There is a (currently very minimal)
 generic integration layer in ManifestDestiny for use of all harnesses,
 `manifestparser.TestManifest`.
@@ -123,35 +131,35 @@ tests without disabled (various other qu
 
 Since the system is convention-based, the harnesses may do whatever
 they want with the data.  They may ignore it completely, they may use
 the provided integration layer, or they may provide their own
 integration layer.  This should allow whatever sort of logic is
 desired.  For instance, if in yourtestharness you wanted to run only on
 mondays for a certain class of tests:
 
- tests = []
- for test in manifests.tests:
-     if 'runOnDay' in test:
-        if calendar.day_name[calendar.weekday(*datetime.datetime.now().timetuple()[:3])].lower() == test['runOnDay'].lower():
-            tests.append(test)
-     else:
-        tests.append(test)
+    tests = []
+    for test in manifests.tests:
+        if 'runOnDay' in test:
+           if calendar.day_name[calendar.weekday(*datetime.datetime.now().timetuple()[:3])].lower() == test['runOnDay'].lower():
+               tests.append(test)
+        else:
+           tests.append(test)
 
 To recap:
 * the manifests allow you to specify test data
 * the parser gives you this data
 * you can use it however you want or process it further as you need
 
 Tests are denoted by sections in an .ini file (see
 http://hg.mozilla.org/automation/ManifestDestiny/file/tip/manifestdestiny/tests/mozmill-example.ini).
 
 Additional manifest files may be included with an `[include:]` directive:
 
- [include:path-to-additional-file.manifest]
+    [include:path-to-additional-file.manifest]
 
 The path to included files is relative to the current manifest.
 
 The `[DEFAULT]` section contains variables that all tests inherit from.
 
 Included files will inherit the top-level variables but may override
 in their own `[DEFAULT]` section.
 
@@ -178,17 +186,17 @@ and
 https://github.com/mozilla/mozbase/blob/master/manifestdestiny/manifestparser.py
 in particular.
 
 
 # Using Manifests
 
 A test harness will normally call `TestManifest.active_tests`:
 
-   def active_tests(self, exists=True, disabled=True, **tags):
+    def active_tests(self, exists=True, disabled=True, **tags):
 
 The manifests are passed to the `__init__` or `read` methods with
 appropriate arguments.  `active_tests` then allows you to select the
 tests you want:
 
 - exists : return only existing tests
 - disabled : whether to return disabled tests; if not these will be
   filtered out; if True (the default), the `disabled` key of a
@@ -211,25 +219,99 @@ ManifestDestiny comes with a console scr
 may be used to create a seed manifest structure from a directory of
 files.  Run `manifestparser help create` for usage information.
 
 
 # Copying Manifests
 
 To copy tests and manifests from a source:
 
-  manifestparser [options] copy from_manifest to_directory -tag1 -tag2 --key1=value1 key2=value2 ...
+    manifestparser [options] copy from_manifest to_directory -tag1 -tag2 --key1=value1 key2=value2 ...
 
 
 # Upating Tests
 
 To update the tests associated with with a manifest from a source
 directory:
 
-  manifestparser [options] update manifest from_directory -tag1 -tag2 --key1=value1 --key2=value2 ...
+    manifestparser [options] update manifest from_directory -tag1 -tag2 --key1=value1 --key2=value2 ...
+
+
+# Usage example
+
+Here is an example of how to create manifests for a directory tree and
+update the tests listed in the manifests from an external source.
+
+## Creating Manifests
+
+Let's say you want to make a series of manifests for a given directory structure containing `.js` test files:
+
+    testing/mozmill/tests/firefox/
+    testing/mozmill/tests/firefox/testAwesomeBar/
+    testing/mozmill/tests/firefox/testPreferences/
+    testing/mozmill/tests/firefox/testPrivateBrowsing/
+    testing/mozmill/tests/firefox/testSessionStore/
+    testing/mozmill/tests/firefox/testTechnicalTools/
+    testing/mozmill/tests/firefox/testToolbar/
+    testing/mozmill/tests/firefox/restartTests
+
+You can use `manifestparser create` to do this:
+
+    $ manifestparser help create
+    Usage: manifestparser.py [options] create directory <directory> <...>
+
+         create a manifest from a list of directories
+
+    Options:
+      -p PATTERN, --pattern=PATTERN
+                            glob pattern for files
+      -i IGNORE, --ignore=IGNORE
+                            directories to ignore
+      -w IN_PLACE, --in-place=IN_PLACE
+                            Write .ini files in place; filename to write to
+
+We only want `.js` files and we want to skip the `restartTests` directory.
+We also want to write a manifest per directory, so I use the `--in-place`
+option to write the manifests:
+
+    manifestparser create . -i restartTests -p '*.js' -w manifest.ini
+
+This creates a manifest.ini per directory that we care about with the JS test files:
+
+    testing/mozmill/tests/firefox/manifest.ini
+    testing/mozmill/tests/firefox/testAwesomeBar/manifest.ini
+    testing/mozmill/tests/firefox/testPreferences/manifest.ini
+    testing/mozmill/tests/firefox/testPrivateBrowsing/manifest.ini
+    testing/mozmill/tests/firefox/testSessionStore/manifest.ini
+    testing/mozmill/tests/firefox/testTechnicalTools/manifest.ini
+    testing/mozmill/tests/firefox/testToolbar/manifest.ini
+
+The top-level `manifest.ini` merely has `[include:]` references to the sub manifests:
+
+    [include:testAwesomeBar/manifest.ini]
+    [include:testPreferences/manifest.ini]
+    [include:testPrivateBrowsing/manifest.ini]
+    [include:testSessionStore/manifest.ini]
+    [include:testTechnicalTools/manifest.ini]
+    [include:testToolbar/manifest.ini]
+
+Each sub-level manifest contains the (`.js`) test files relative to it.
+
+## Updating the tests from manifests
+
+You may need to update tests as given in manifests from a different source directory.
+`manifestparser update` was made for just this purpose:
+
+    Usage: manifestparser [options] update manifest directory -tag1 -tag2 --key1=value1 --key2=value2 ...
+
+        update the tests as listed in a manifest from a directory
+
+To update from a directory of tests in `~/mozmill/src/mozmill-tests/firefox/` run:
+
+    manifestparser update manifest.ini ~/mozmill/src/mozmill-tests/firefox/
 
 
 # Tests
 
 ManifestDestiny includes a suite of tests:
 
 https://github.com/mozilla/mozbase/tree/master/manifestdestiny/tests
 
@@ -247,30 +329,30 @@ Or contact jhammel @mozilla.org or in #a
 
 
 # CLI
 
 Run `manifestparser help` for usage information.
 
 To create a manifest from a set of directories:
 
-  manifestparser [options] create directory <directory> <...> [create-options]
+    manifestparser [options] create directory <directory> <...> [create-options]
 
 To output a manifest of tests:
 
-  manifestparser [options] write manifest <manifest> <...> -tag1 -tag2 --key1=value1 --key2=value2 ...
+    manifestparser [options] write manifest <manifest> <...> -tag1 -tag2 --key1=value1 --key2=value2 ...
 
 To copy tests and manifests from a source:
 
-  manifestparser [options] copy from_manifest to_manifest -tag1 -tag2 --key1=value1 key2=value2 ...
+    manifestparser [options] copy from_manifest to_manifest -tag1 -tag2 --key1=value1 key2=value2 ...
 
 To update the tests associated with with a manifest from a source
 directory:
 
-  manifestparser [options] update manifest from_directory -tag1 -tag2 --key1=value1 --key2=value2 ...
+    manifestparser [options] update manifest from_directory -tag1 -tag2 --key1=value1 --key2=value2 ...
 
 
 # Design Considerations
 
 Contrary to some opinion, manifestparser.py and the associated .ini
 format were not magically plucked from the sky but were descended upon
 through several design considerations.
 
@@ -304,16 +386,24 @@ through several design considerations.
   mozilla-central. This is less true these days and it is increasingly
   likely that more tests will not live in mozilla-central going
   forward.  So `manifestparser.py` should be highly consumable. To
   this end, it is a single file, as appropriate to mozilla-central,
   which is also a working python package deployed to PyPI for easy
   installation.
 
 
+# Developing ManifestDestiny
+
+ManifestDestiny is developed and maintained by Mozilla's
+[Automation and Testing Team](https://wiki.mozilla.org/Auto-tools).
+The project page is located at
+https://wiki.mozilla.org/Auto-tools/Projects/ManifestDestiny .
+
+
 # Historical Reference
 
 Date-ordered list of links about how manifests came to be where they are today::
 
 * https://wiki.mozilla.org/Auto-tools/Projects/UniversalManifest
 * http://alice.nodelman.net/blog/post/2010/05/
 * http://alice.nodelman.net/blog/post/universal-manifest-for-unit-tests-a-proposal/
 * https://elvis314.wordpress.com/2010/07/05/improving-personal-hygiene-by-adjusting-mochitests/
--- a/testing/mozbase/mozdevice/mozdevice/devicemanagerADB.py
+++ b/testing/mozbase/mozdevice/mozdevice/devicemanagerADB.py
@@ -1,23 +1,25 @@
 import subprocess
 from devicemanager import DeviceManager, DMError
 import re
 import os
 import sys
+import tempfile
 
 class DeviceManagerADB(DeviceManager):
 
   def __init__(self, host = None, port = 20701, retrylimit = 5, packageName = None):
     self.host = host
     self.port = port
     self.retrylimit = retrylimit
     self.retries = 0
     self._sock = None
     self.useRunAs = False
+    self.useZip = False
     self.packageName = None
     if packageName == None:
       if os.getenv('USER'):
         packageName = 'org.mozilla.fennec_' + os.getenv('USER')
       else:
         packageName = 'org.mozilla.fennec_'
     self.Init(packageName)
 
@@ -26,16 +28,20 @@ class DeviceManagerADB(DeviceManager):
     # successful initialization even if, for example, adb is not installed.
     try:
       self.verifyADB()
       self.verifyRunAs(packageName)
     except:
       self.useRunAs = False
       self.packageName = None
     try:
+      self.verifyZip()
+    except:
+      self.useZip = False
+    try:
       # a test to see if we have root privs
       files = self.listFiles("/data/data")
       if (len(files) == 1):
         if (files[0].find("Permission denied") != -1):
           print "NOT running as root"
           raise Exception("not running as root")
     except:
       try:
@@ -98,41 +104,51 @@ class DeviceManagerADB(DeviceManager):
   # push localDir from host to remoteDir on the device
   # external function
   # returns:
   #  success: remoteDir
   #  failure: None
   def pushDir(self, localDir, remoteDir):
     # adb "push" accepts a directory as an argument, but if the directory
     # contains symbolic links, the links are pushed, rather than the linked
-    # files; we push file-by-file to get around this limitation
+    # files; we either zip/unzip or push file-by-file to get around this 
+    # limitation
     try:
-      if (not self.dirExists(remoteDir)):
-        self.mkDirs(remoteDir+"/x")
-      for root, dirs, files in os.walk(localDir, followlinks='true'):
-        relRoot = os.path.relpath(root, localDir)
-        for file in files:
-          localFile = os.path.join(root, file)
-          remoteFile = remoteDir + "/"
-          if (relRoot!="."):
-            remoteFile = remoteFile + relRoot + "/"
-          remoteFile = remoteFile + file
-          self.pushFile(localFile, remoteFile)
-        for dir in dirs:
-          targetDir = remoteDir + "/"
-          if (relRoot!="."):
-            targetDir = targetDir + relRoot + "/"
-          targetDir = targetDir + dir
-          if (not self.dirExists(targetDir)):
-            self.mkDir(targetDir)
+      if (self.useZip):
+        localZip = tempfile.mktemp()+".zip"
+        remoteZip = remoteDir + "/adbdmtmp.zip"
+        subprocess.check_output(["zip", "-r", localZip, '.'], cwd=localDir)
+        self.pushFile(localZip, remoteZip)
+        os.remove(localZip)
+        self.checkCmdAs(["shell", "unzip", "-o", remoteZip, "-d", remoteDir])
+        self.checkCmdAs(["shell", "rm", remoteZip])
+      else:
+        if (not self.dirExists(remoteDir)):
+          self.mkDirs(remoteDir+"/x")
+        for root, dirs, files in os.walk(localDir, followlinks='true'):
+          relRoot = os.path.relpath(root, localDir)
+          for file in files:
+            localFile = os.path.join(root, file)
+            remoteFile = remoteDir + "/"
+            if (relRoot!="."):
+              remoteFile = remoteFile + relRoot + "/"
+            remoteFile = remoteFile + file
+            self.pushFile(localFile, remoteFile)
+          for dir in dirs:
+            targetDir = remoteDir + "/"
+            if (relRoot!="."):
+              targetDir = targetDir + relRoot + "/"
+            targetDir = targetDir + dir
+            if (not self.dirExists(targetDir)):
+              self.mkDir(targetDir)
       self.checkCmdAs(["shell", "chmod", "777", remoteDir])
-      return True
+      return remoteDir
     except:
       print "pushing " + localDir + " to " + remoteDir + " failed"
-      return False
+      return None
 
   # external function
   # returns:
   #  success: True
   #  failure: False
   def dirExists(self, dirname):
     return self.isDir(dirname)
 
@@ -236,21 +252,35 @@ class DeviceManagerADB(DeviceManager):
   # external function
   # returns:
   #  success: output filename
   #  failure: None
   def launchProcess(self, cmd, outputFile = "process.txt", cwd = '', env = '', failIfRunning=False):
     acmd = ["shell", "am","start"]
     cmd = ' '.join(cmd).strip()
     i = cmd.find(" ")
+    # SUT identifies the URL by looking for :\\ -- another strategy to consider
+    re_url = re.compile('^[http|file|chrome|about].*')
+    last = cmd.rfind(" ")
+    uri = ""
+    args = ""
+    if re_url.match(cmd[last:].strip()):
+      args = cmd[i:last].strip()
+      uri = cmd[last:].strip()
+    else:
+      args = cmd[i:].strip()
     acmd.append("-n")
     acmd.append(cmd[0:i] + "/.App")
     acmd.append("--es")
-    acmd.append("args")
-    acmd.append(cmd[i:])
+    if args != "":
+      acmd.append("args")
+      acmd.append(args)
+    if uri != "":
+      acmd.append("-d")
+      acmd.append(''.join(['\'',uri, '\'']));
     print acmd
     self.checkCmd(acmd)
     return outputFile;
 
   # external function
   # returns:
   #  success: output from testagent
   #  failure: None
@@ -573,8 +603,30 @@ class DeviceManagerADB(DeviceManager):
       self.checkCmd(["shell", "run-as", packageName, "cp", self.tmpDir + "/tmpfile", devroot + "/sanity"])
       if (self.fileExists(devroot + "/sanity/tmpfile")):
         print "will execute commands via run-as " + packageName
         self.packageName = packageName
         self.useRunAs = True
       self.checkCmd(["shell", "rm", devroot + "/tmp/tmpfile"])
       self.checkCmd(["shell", "run-as", packageName, "rm", "-r", devroot + "/sanity"])
       
+  def isUnzipAvailable(self):
+    data = self.runCmd(["shell", "unzip"]).stdout.read()
+    if (re.search('Usage', data)):
+      return True
+    else:
+      return False
+
+  def isLocalZipAvailable(self):
+    try:
+      subprocess.check_call(["zip", "-?"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    except:
+      return False
+    return True
+
+  def verifyZip(self):
+    # If "zip" can be run locally, and "unzip" can be run remotely, then pushDir
+    # can use these to push just one file per directory -- a significant
+    # optimization for large directories.
+    self.useZip = False
+    if (self.isUnzipAvailable() and self.isLocalZipAvailable()):
+      print "will use zip to push directories"
+      self.useZip = True
--- a/testing/mozbase/mozinstall/mozinstall/mozinstall.py
+++ b/testing/mozbase/mozinstall/mozinstall/mozinstall.py
@@ -105,41 +105,61 @@ def get_binary(path, apps=_default_apps)
 
 def _extract(path, extdir=None, delete=False):
     """
     Takes in a tar or zip file and extracts it to extdir
     If extdir is not specified, extracts to os.path.dirname(path)
     If delete is set to True, deletes the bundle at path
     Returns the list of top level files that were extracted
     """
+    assert not os.path.isfile(extdir), "extdir cannot be a file"
+    if extdir is None:
+        extdir = os.path.dirname(path)
+    elif not os.path.isdir(extdir):
+        os.makedirs(extdir)
     if zipfile.is_zipfile(path):
         bundle = zipfile.ZipFile(path)
         namelist = bundle.namelist()
+        if hasattr(bundle, 'extractall'):
+            bundle.extractall(path=extdir)
+        # zipfile.extractall doesn't exist in Python 2.5
+        else:
+            for name in namelist:
+                filename = os.path.realpath(os.path.join(extdir, name))
+                if name.endswith("/"):
+                    os.makedirs(filename)
+                else:
+                    path = os.path.dirname(filename)
+                    if not os.path.isdir(path):
+                        os.makedirs(path)
+                    dest = open(filename, "wb")
+                    dest.write(bundle.read(name))
+                    dest.close()
     elif tarfile.is_tarfile(path):
         bundle = tarfile.open(path)
         namelist = bundle.getnames()
+        if hasattr(bundle, 'extractall'):
+            bundle.extractall(path=extdir)
+        # tarfile.extractall doesn't exist in Python 2.4
+        else:
+            for name in namelist:
+                bundle.extract(name, path=extdir)
     else:
         return
-    if extdir is None:
-        extdir = os.path.dirname(path)
-    elif not os.path.exists(extdir):
-        os.makedirs(extdir)
-    bundle.extractall(path=extdir)
     bundle.close()
     if delete:
         os.remove(path)
     # namelist returns paths with forward slashes even in windows
     top_level_files = [os.path.join(extdir, name) for name in namelist
                              if len(name.rstrip('/').split('/')) == 1]
-    # namelist doesn't include folders in windows, append these to the list
-    if mozinfo.isWin:
-        for name in namelist:
-            root = name[:name.find('/')]
-            if root not in top_level_files:
-                top_level_files.append(root)
+    # namelist doesn't include folders, append these to the list
+    for name in namelist:
+        root = os.path.join(extdir, name[:name.find('/')])
+        if root not in top_level_files:
+            top_level_files.append(root)
     return top_level_files
 
 def _install_dmg(src, dest):
     proc = subprocess.Popen("hdiutil attach " + src,
                             shell=True,
                             stdout=subprocess.PIPE)
     try:
         for data in proc.communicate()[0].split():
--- a/testing/peptest/peptest/extension/resource/pep/testsuite.js
+++ b/testing/peptest/peptest/extension/resource/pep/testsuite.js
@@ -95,15 +95,17 @@ TestSuite.prototype.loadTest = function(
 
     // post-test
     let runTime = Date.now() - startTime;
     let fThreshold = test['failThreshold'] === undefined ?
                           '' : ' ' + test['failThreshold'];
     log.log('TEST-END', test.name + ' ' + runTime + fThreshold);
   } catch (e) {
     log.error(test.name + ' | ' + e);
-    log.debug(test.name + ' | Traceback:');
-    lines = e.stack.split('\n');
-    for (let i = 0; i < lines.length - 1; ++i) {
-      log.debug('\t' + lines[i]);
+    if (e['stack'] !== undefined) {
+      log.debug(test.name + ' | Traceback:');
+      lines = e.stack.split('\n');
+      for (let i = 0; i < lines.length - 1; ++i) {
+        log.debug('\t' + lines[i]);
+      }
     }
   }
 };
--- a/testing/peptest/peptest/pepprocess.py
+++ b/testing/peptest/peptest/pepprocess.py
@@ -31,16 +31,18 @@
 # and other provisions required by the GPL or the LGPL. If you do not delete
 # the provisions above, a recipient may use your version of this file under
 # the terms of any one of the MPL, the GPL or the LGPL.
 #
 # ***** END LICENSE BLOCK *****
 
 from mozprocess import ProcessHandler
 from pepresults import Results
+from time import sleep
+from threading import Thread
 import mozlog
 import os
 
 results = Results()
 
 class PepProcess(ProcessHandler):
     """
     Process handler for running peptests
@@ -52,45 +54,58 @@ class PepProcess(ProcessHandler):
                        **kwargs):
 
         ProcessHandler.__init__(self, cmd, args=args, cwd=cwd, env=env,
                                 ignore_children=ignore_children, **kwargs)
 
         self.logger = mozlog.getLogger('PEP')
         results.fails[str(None)] = []
 
+    def waitForQuit(self, timeout=5):
+        for i in range(1, timeout):
+            if self.proc.returncode != None:
+                return
+            sleep(1)
+        self.proc.kill()
+
     def processOutputLine(self, line):
         """
         Callback called on each line of output
         Responsible for determining which output lines are relevant
         and writing them to a log
         """
         tokens = line.split(' ')
         if len(tokens) > 1 and tokens[0] == 'PEP':
             # The output is generated from the Peptest extension
             # Format is 'PEP <LEVEL> <MSG>' where <MSG> can have multiple tokens
             # The content of <MSG> depends on the <LEVEL>
+            if line.find('Test Suite Finished') != -1:
+                thread = Thread(target=self.waitForQuit)
+                thread.setDaemon(True) # don't hang on quit
+                thread.start()
+
             level = tokens[1]
             if level == 'TEST-START':
                 results.currentTest = tokens[2].rstrip()
                 results.fails[results.currentTest] = []
                 self.logger.testStart(results.currentTest)
             elif level == 'TEST-END':
                 metric = results.get_metric(results.currentTest)
                 if len(tokens) > 4:
                     threshold = float(tokens[4].rstrip())
                 else:
                     threshold = 0.0
 
                 msg = results.currentTest \
-                      + ' | fail threshold: ' + str(threshold) \
-                      + ' | metric: ' + str(metric)
+                      + ' | fail threshold: ' + str(threshold)
                 if metric > threshold:
+                    msg += ' < metric: ' + str(metric)
                     self.logger.testFail(msg)
                 else:
+                    msg += ' >= metric: ' + str(metric)
                     self.logger.testPass(msg)
 
                 self.logger.testEnd(
                         results.currentTest +
                         ' | finished in: ' + tokens[3].rstrip() + ' ms')
                 results.currentTest = None
             elif level == 'ACTION-START':
                 results.currentAction = tokens[3].rstrip()
--- a/testing/peptest/peptest/peputils.py
+++ b/testing/peptest/peptest/peputils.py
@@ -63,30 +63,58 @@ def isURL(path):
     """Return True if path looks like a URL."""
     if path is not None:
         return urlparse.urlparse(path).scheme != ''
     return False
 
 def extract(path, extdir=None, delete=False):
     """
     Takes in a tar or zip file and extracts it to extdir
-    If extdir is not specified, extracts to path
+    If extdir is not specified, extracts to os.path.dirname(path)
     If delete is set to True, deletes the bundle at path
     Returns the list of top level files that were extracted
     """
+    assert not os.path.isfile(extdir), "extdir cannot be a file"
+    if extdir is None:
+        extdir = os.path.dirname(path)
+    elif not os.path.isdir(extdir):
+        os.makedirs(extdir)
     if zipfile.is_zipfile(path):
         bundle = zipfile.ZipFile(path)
         namelist = bundle.namelist()
+        if hasattr(bundle, 'extractall'):
+            bundle.extractall(path=extdir)
+        # zipfile.extractall doesn't exist in Python 2.5
+        else:
+            for name in namelist:
+                filename = os.path.realpath(os.path.join(extdir, name))
+                if name.endswith("/"):
+                    os.makedirs(filename)
+                else:
+                    path = os.path.dirname(filename)
+                    if not os.path.isdir(path):
+                        os.makedirs(path)
+                    dest = open(filename, "wb")
+                    dest.write(bundle.read(name))
+                    dest.close()
     elif tarfile.is_tarfile(path):
         bundle = tarfile.open(path)
         namelist = bundle.getnames()
+        if hasattr(bundle, 'extractall'):
+            bundle.extractall(path=extdir)
+        # tarfile.extractall doesn't exist in Python 2.4
+        else:
+            for name in namelist:
+                bundle.extract(name, path=extdir)
     else:
         return
-    if extdir is None:
-        extdir = os.path.dirname(path)
-    elif not os.path.exists(extdir):
-        os.makedirs(extdir)
-    bundle.extractall(path=extdir)
     bundle.close()
     if delete:
         os.remove(path)
-    return [os.path.join(extdir, name) for name in namelist
-                if len(name.rstrip(os.sep).split(os.sep)) == 1]
+    # namelist returns paths with forward slashes even in windows
+    top_level_files = [os.path.join(extdir, name) for name in namelist
+                             if len(name.rstrip('/').split('/')) == 1]
+    # namelist doesn't include folders, append these to the list
+    for name in namelist:
+        root = os.path.join(extdir, name[:name.find('/')])
+        if root not in top_level_files:
+            top_level_files.append(root)
+    return top_level_files
--- a/testing/peptest/peptest/runpeptests.py
+++ b/testing/peptest/peptest/runpeptests.py
@@ -77,16 +77,17 @@ class Peptest():
 
         tests = []
         # TODO is there a better way of doing this?
         if self.options.testPath.endswith('.js'):
             # a single test file was passed in
             testObj = {}
             testObj['path'] = os.path.realpath(self.options.testPath)
             testObj['name'] = os.path.basename(self.options.testPath)
+            testObj['here'] = os.path.dirname(testObj['path'])
             tests.append(testObj)
         else:
             # a test manifest was passed in
             # open and convert the manifest to json
             manifest = TestManifest()
             manifest.read(self.options.testPath)
             tests = manifest.get()
 
--- a/testing/peptest/setup.py
+++ b/testing/peptest/setup.py
@@ -39,22 +39,25 @@ import os
 from setuptools import setup, find_packages
 
 try:
     here = os.path.dirname(os.path.abspath(__file__))
     description = file(os.path.join(here, 'README.md')).read()
 except IOError:
     description = ''
 
-version = "0.0"
+version = "0.1"
 
-dependencies = ['mozprofile',
+dependencies = ['ManifestDestiny',
+                'mozhttpd',
+                'mozlog',
+                'mozprofile >= 0.1',
                 'mozprocess',
                 'mozrunner >= 3.0b3',
-                'mozlog']
+               ]
 
 setup(name='peptest',
       version=version,
       description="""
                   Peptest is an automated testing framework designed to test whether or not the browser's
                   UI thread remains responsive while performing a variety of actions. Tests are simple
                   Javascript files which can optionally import Mozmill's driver to manipulate the user
                   interface in an automated fashion.