Bug 1505086: use `mach python` to submit build telemetry r=ted
authorConnor Sheehan <sheehan@mozilla.com>
Mon, 12 Nov 2018 14:57:57 +0000
changeset 445885 9a2f0d443bb1d94c38da120b790fcad6245b77d7
parent 445884 12c35ad7e844ec1d1c555ef57acac71e5c97d656
child 445886 96046dff0022da18603a359733514a4ff0efc799
push id35028
push usercsabou@mozilla.com
push dateMon, 12 Nov 2018 21:54:15 +0000
treeherdermozilla-central@05331fb8f533 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersted
bugs1505086
milestone65.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1505086: use `mach python` to submit build telemetry r=ted While attempting to improve the build telemetry submission logic, I found a bug in the way telemetry submission works. Essentially the submission script was failing to import any of the required packages (specifically `mozbuild.telemetry` in this case) as the method used to modify path was incorrect and the script was running outside of the virtualenv. The invocation is also sending stdout and stderr to `/dev/null`, making this problem even less obvious. When I fixed the path modifications, I realized that `mozbuild` imports will require a long chain of other imports (and transitively, more `sys.path` modifications) such as `which`, `mach`, `mozautomation`, etc to complete. When I tested the submission script, I did so by running `mach python build/submit_telemetry_data.py`, which runs the script in a virtualenv with all required packages installed. That's likely part of the reasons I overlooked this issue in testing. Rather than go through the process of importing every dependency of `mozbuild`, this commit changes the invocation of the submission script to go through `mach python`. Things seem to work as expected with this change. Differential Revision: https://phabricator.services.mozilla.com/D11278
build/mach_bootstrap.py
build/submit_telemetry_data.py
--- a/build/mach_bootstrap.py
+++ b/build/mach_bootstrap.py
@@ -277,18 +277,20 @@ def bootstrap(topsrcdir, mozilla_dir=Non
 
         if should_skip_telemetry_submission(handler):
             return True
 
         # But only submit about every n-th operation
         if random.randint(1, TELEMETRY_SUBMISSION_FREQUENCY) != 1:
             return
 
+        machpath = os.path.join(instance.topsrcdir, 'mach')
         with open(os.devnull, 'wb') as devnull:
-            subprocess.Popen([sys.executable,
+            subprocess.Popen([machpath, 'python',
+                              '--no-virtualenv',
                               os.path.join(topsrcdir, 'build',
                                            'submit_telemetry_data.py'),
                               get_state_dir()[0]],
                              stdout=devnull, stderr=devnull)
 
     def populate_context(context, key=None):
         if key is None:
             return
--- a/build/submit_telemetry_data.py
+++ b/build/submit_telemetry_data.py
@@ -5,24 +5,16 @@
 from __future__ import print_function
 
 import datetime
 import json
 import logging
 import os
 import sys
 
-HERE = os.path.abspath(os.path.dirname(__file__))
-PYTHIRDPARTY = os.path.join(HERE, '..', 'third_party', 'python')
-
-# Add some required files to $PATH to ensure they are available
-sys.path.append(os.path.join(HERE, '..', 'python', 'mozbuild', 'mozbuild'))
-sys.path.append(os.path.join(PYTHIRDPARTY, 'requests'))
-sys.path.append(os.path.join(PYTHIRDPARTY, 'voluptuous'))
-
 import requests
 import voluptuous
 import voluptuous.humanize
 
 from mozbuild.telemetry import (
     schema as build_telemetry_schema,
     verify_statedir,
 )