Bug 1305804 - Resolve hostname before attempting to use pip link; r=ted
authorGregory Szorc <gps@mozilla.com>
Tue, 27 Sep 2016 13:44:27 -0700
changeset 315530 e1763ef7d6b96ab3ff6bc4f94c6e3cdd6b5e79bb
parent 315529 d5b3c15bbbdf8776320ed316c186f3e638a5a411
child 315531 733a77530961deeed0690c1b0993b8c064b07da4
push id30751
push usercbook@mozilla.com
push dateThu, 29 Sep 2016 09:43:23 +0000
treeherdermozilla-central@b67dc49095dc [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersted
bugs1305804, 1304176
milestone52.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1305804 - Resolve hostname before attempting to use pip link; r=ted In automation, we try to use pypi.pvt.build.mozilla.org nearly everywhere. This hostname doesn't resolve in TaskCluster and outside of buildbot automation. A consequence of work in bug 1304176 and using a modern pip is that we attempt to connect to all defined pip links. This was resulting in pip attempting connections to pypi.pvt.build.mozilla.org. And since pip was attempting retries, this resulted in a several second delay for all pip operations if that host didn't resolve. This commit adds a DNS lookup in mozharness before using a pip --link. We spend a little bit of overhead in mozharness doing a DNS lookup. In return, we guarantee we'll avoid a multiple second pause if any links don't resolve. This is somewhat hacky. But it seems like the easiest solution. MozReview-Commit-ID: EecqBQSW75R
testing/mozharness/mozharness/base/python.py
--- a/testing/mozharness/mozharness/base/python.py
+++ b/testing/mozharness/mozharness/base/python.py
@@ -8,16 +8,17 @@
 '''
 
 import distutils.version
 import os
 import subprocess
 import sys
 import time
 import json
+import socket
 import traceback
 import urlparse
 
 import mozharness
 from mozharness.base.script import (
     PostScriptAction,
     PostScriptRun,
     PreScriptAction,
@@ -261,18 +262,26 @@ class VirtualenvMixin(object):
             self.fatal("install_module() doesn't understand an install_method of %s!" % install_method)
 
         # Add --find-links pages to look at. Add --trusted-host automatically if
         # the host isn't secure. This allows modern versions of pip to connect
         # without requiring an override.
         proxxy = Proxxy(self.config, self.log_obj)
         trusted_hosts = set()
         for link in proxxy.get_proxies_and_urls(c.get('find_links', [])):
+            parsed = urlparse.urlparse(link)
+
+            try:
+                socket.gethostbyname(parsed.hostname)
+            except socket.gaierror as e:
+                self.info('error resolving %s (ignoring): %s' %
+                          (parsed.hostname, e.message))
+                continue
+
             command.extend(["--find-links", link])
-            parsed = urlparse.urlparse(link)
             if parsed.scheme != 'https':
                 trusted_hosts.add(parsed.hostname)
 
         if self.pip_version >= distutils.version.LooseVersion('6.0'):
             for host in sorted(trusted_hosts):
                 command.extend(['--trusted-host', host])
 
         # module_url can be None if only specifying requirements files