Bug 1305804 - Resolve hostname before attempting to use pip link; r?ted draft
authorGregory Szorc <gps@mozilla.com>
Tue, 27 Sep 2016 13:44:27 -0700
changeset 418127 c0f2203143dbd64827ff2809b83bcb4a32300d2e
parent 417914 66a77b9bfe5dcacd50eccf85de7c0e7e15ce0ffd
child 532275 ccd40515adbfc6bbdf3fe8b1eeae1ca63ab21e36
push id30606
push userbmo:gps@mozilla.com
push dateTue, 27 Sep 2016 20:44:40 +0000
bugs1305804, 1304176
Bug 1305804 - Resolve hostname before attempting to use pip link; r?ted In automation, we try to use pypi.pvt.build.mozilla.org nearly everywhere. This hostname doesn't resolve in TaskCluster and outside of buildbot automation. A consequence of work in bug 1304176 and using a modern pip is that we attempt to connect to all defined pip links. This was resulting in pip attempting connections to pypi.pvt.build.mozilla.org. And since pip was attempting retries, this resulted in a several second delay for all pip operations if that host didn't resolve. This commit adds a DNS lookup in mozharness before using a pip --link. We spend a little bit of overhead in mozharness doing a DNS lookup. In return, we guarantee we'll avoid a multiple second pause if any links don't resolve. This is somewhat hacky. But it seems like the easiest solution. MozReview-Commit-ID: EecqBQSW75R
--- a/testing/mozharness/mozharness/base/python.py
+++ b/testing/mozharness/mozharness/base/python.py
@@ -8,16 +8,17 @@
 import distutils.version
 import os
 import subprocess
 import sys
 import time
 import json
+import socket
 import traceback
 import urlparse
 import mozharness
 from mozharness.base.script import (
@@ -261,18 +262,26 @@ class VirtualenvMixin(object):
             self.fatal("install_module() doesn't understand an install_method of %s!" % install_method)
         # Add --find-links pages to look at. Add --trusted-host automatically if
         # the host isn't secure. This allows modern versions of pip to connect
         # without requiring an override.
         proxxy = Proxxy(self.config, self.log_obj)
         trusted_hosts = set()
         for link in proxxy.get_proxies_and_urls(c.get('find_links', [])):
+            parsed = urlparse.urlparse(link)
+            try:
+                socket.gethostbyname(parsed.hostname)
+            except socket.gaierror as e:
+                self.info('error resolving %s (ignoring): %s' %
+                          (parsed.hostname, e.message))
+                continue
             command.extend(["--find-links", link])
-            parsed = urlparse.urlparse(link)
             if parsed.scheme != 'https':
         if self.pip_version >= distutils.version.LooseVersion('6.0'):
             for host in sorted(trusted_hosts):
                 command.extend(['--trusted-host', host])
         # module_url can be None if only specifying requirements files