Bug 1509962: Vendor redo==2.0.2; r=mtabara
authorTom Prince <mozilla@hocat.ca>
Mon, 03 Dec 2018 06:43:52 +0000
changeset 508405 53bb18cc23abfddd18b90efedb0f2596ba5293dd
parent 508404 b1c9fa45f90ee5f26843749557a2fda84581f406
child 508406 fe7c1bcf044acc3644adbba6e7b9039d1ce78eb4
push id1905
push userffxbld-merge
push dateMon, 21 Jan 2019 12:33:13 +0000
treeherdermozilla-release@c2fca1944d8c [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersmtabara
bugs1509962
milestone65.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1509962: Vendor redo==2.0.2; r=mtabara Differential Revision: https://phabricator.services.mozilla.com/D12973
third_party/python/redo/PKG-INFO
third_party/python/redo/README
third_party/python/redo/README.md
third_party/python/redo/redo.egg-info/PKG-INFO
third_party/python/redo/redo.egg-info/SOURCES.txt
third_party/python/redo/redo/__init__.py
third_party/python/redo/redo/cmd.py
third_party/python/redo/setup.cfg
third_party/python/redo/setup.py
third_party/python/requirements.in
third_party/python/requirements.txt
--- a/third_party/python/redo/PKG-INFO
+++ b/third_party/python/redo/PKG-INFO
@@ -1,10 +1,10 @@
 Metadata-Version: 1.0
 Name: redo
-Version: 1.6
+Version: 2.0.2
 Summary: Utilities to retry Python callables.
 Home-page: https://github.com/bhearsum/redo
 Author: Ben Hearsum
 Author-email: ben@hearsum.ca
 License: UNKNOWN
 Description: UNKNOWN
 Platform: UNKNOWN
deleted file mode 100644
--- a/third_party/python/redo/README
+++ /dev/null
@@ -1,4 +0,0 @@
-Redo - Utilities to retry Python callables
-******************************************
-
-Redo provides various means to add seamless retriability to any Python callable. Redo includes a plain function (redo.retry), a decorator (redo.retriable), and a context manager (redo.retrying) to enable you to integrate it in the best possible way for your project. As a bonus, a standalone interface is also included ("retry"). For details and sample invocations have a look at the docstrings in redo/__init__.py.
new file mode 100644
--- /dev/null
+++ b/third_party/python/redo/README.md
@@ -0,0 +1,147 @@
+
+# Redo - Utilities to retry Python callables
+******************************************
+
+## Introduction
+
+Redo provides various means to add seamless ability to retry to any Python callable. Redo includes a plain function `(redo.retry)`, a decorator `(redo.retriable)`, and a context manager `(redo.retrying)` to enable you to integrate it in the best possible way for your project. As a bonus, a standalone interface is also included `("retry")`.
+
+## Installation
+
+For installing with pip, run following commands
+> pip install redo
+
+## How To Use
+Below is the list of functions available
+* retrier
+* retry
+* retriable
+* retrying (contextmanager)
+
+### retrier(attempts=5, sleeptime=10, max_sleeptime=300, sleepscale=1.5, jitter=1)
+A generator function that sleeps between retries, handles exponential back off and jitter. The action you are retrying is meant to run after retrier yields. At each iteration, we sleep for `sleeptime + random.randint(-jitter, jitter)`. Afterwards sleeptime is multiplied by sleepscale for the next iteration.
+
+**Arguments Detail:**    
+
+1. **attempts (int):** maximum number of times to try; defaults to 5
+2. **sleeptime (float):** how many seconds to sleep between tries; defaults to 60s (one minute)
+3. **max_sleeptime (float):** the longest we'll sleep, in seconds; defaults to 300s (five minutes)
+4. **sleepscale (float):** how much to multiply the sleep time by each iteration; defaults to 1.5
+5. **jitter (int):** random jitter to introduce to sleep time each iteration. the amount is chosen at random between `[-jitter, +jitter]` defaults to 1
+
+**Output:**
+None, a maximum of `attempts` number of times
+
+**Example:**
+
+    >>> n = 0
+    >>> for _ in retrier(sleeptime=0, jitter=0):
+    ...     if n == 3:
+    ...         # We did the thing!
+    ...         break
+    ...     n += 1
+    >>> n
+    3
+    >>> n = 0
+    >>> for _ in retrier(sleeptime=0, jitter=0):
+    ...     if n == 6:
+    ...         # We did the thing!
+    ...         break
+    ...     n += 1
+    ... else:
+    ...     print("max tries hit")
+    max tries hit
+
+### retry(action, attempts=5, sleeptime=60, max_sleeptime=5 * 60, sleepscale=1.5, jitter=1, retry_exceptions=(Exception,), cleanup=None, args=(), kwargs={})  
+Calls an action function until it succeeds, or we give up.
+
+**Arguments Detail:**  
+
+1. **action (callable):** the function to retry
+2. **attempts (int):** maximum number of times to try; defaults to 5
+3. **sleeptime (float):** how many seconds to sleep between tries; defaults to 60s (one minute)
+4. **max_sleeptime (float):** the longest we'll sleep, in seconds; defaults to 300s (five minutes)
+5. **sleepscale (float):** how much to multiply the sleep time by each iteration; defaults to 1.5
+6. **jitter (int):** random jitter to introduce to sleep time each iteration. The amount is chosen at random between `[-jitter, +jitter]` defaults to 1
+7. **retry_exceptions (tuple):** tuple of exceptions to be caught. If other exceptions are raised by `action()`, then these are immediately re-raised to the caller.
+8. **cleanup (callable):** optional; called if one of `retry_exceptions` is caught. No arguments are passed to the cleanup function; if your cleanup requires arguments, consider using `functools.partial` or a `lambda` function.
+9. **args (tuple):** positional arguments to call `action` with
+10. **kwargs (dict):** keyword arguments to call `action` with
+
+**Output:**
+ Whatever action`(*args, **kwargs)` returns
+ 
+ **Output:**
+        Whatever action(*args, **kwargs) raises. `retry_exceptions` are caught
+        up until the last attempt, in which case they are re-raised.
+
+**Example:**
+
+    >>> count = 0
+    >>> def foo():
+    ...     global count
+    ...     count += 1
+    ...     print(count)
+    ...     if count < 3:
+    ...         raise ValueError("count is too small!")
+    ...     return "success!"
+    >>> retry(foo, sleeptime=0, jitter=0)
+    1
+    2
+    3
+    'success!'
+
+### retriable(*retry_args, **retry_kwargs)
+A decorator factory for `retry()`. Wrap your function in `@retriable(...)` to give it retry powers!
+
+**Arguments Detail:**  
+        Same as for `retry`, with the exception of `action`, `args`, and `kwargs`,
+        which are left to the normal function definition.
+
+**Output:**
+A function decorator
+
+**Example:**
+
+    >>> count = 0
+    >>> @retriable(sleeptime=0, jitter=0)
+    ... def foo():
+    ...     global count
+    ...     count += 1
+    ...     print(count)
+    ...     if count < 3:
+    ...         raise ValueError("count too small")
+    ...     return "success!"
+    >>> foo()
+    1
+    2
+    3
+    'success!'
+
+### retrying(func, *retry_args, **retry_kwargs)
+A context manager for wrapping functions with retry functionality.
+
+**Arguments Detail:**   
+
+1. **func (callable):** the function to wrap
+other arguments as per `retry`
+
+**Output:**
+A context manager that returns `retriable(func)` on `__enter__`
+
+**Example:**
+
+    >>> count = 0
+    >>> def foo():
+    ...     global count
+    ...     count += 1
+    ...     print(count)
+    ...     if count < 3:
+    ...         raise ValueError("count too small")
+    ...     return "success!"
+    >>> with retrying(foo, sleeptime=0, jitter=0) as f:
+    ...     f()
+    1
+    2
+    3
+    'success!'
\ No newline at end of file
--- a/third_party/python/redo/redo.egg-info/PKG-INFO
+++ b/third_party/python/redo/redo.egg-info/PKG-INFO
@@ -1,10 +1,10 @@
 Metadata-Version: 1.0
 Name: redo
-Version: 1.6
+Version: 2.0.2
 Summary: Utilities to retry Python callables.
 Home-page: https://github.com/bhearsum/redo
 Author: Ben Hearsum
 Author-email: ben@hearsum.ca
 License: UNKNOWN
 Description: UNKNOWN
 Platform: UNKNOWN
--- a/third_party/python/redo/redo.egg-info/SOURCES.txt
+++ b/third_party/python/redo/redo.egg-info/SOURCES.txt
@@ -1,10 +1,10 @@
+README.md
 setup.cfg
 setup.py
 redo/__init__.py
 redo/cmd.py
 redo.egg-info/PKG-INFO
 redo.egg-info/SOURCES.txt
 redo.egg-info/dependency_links.txt
 redo.egg-info/entry_points.txt
-redo.egg-info/pbr.json
 redo.egg-info/top_level.txt
\ No newline at end of file
--- a/third_party/python/redo/redo/__init__.py
+++ b/third_party/python/redo/redo/__init__.py
@@ -13,28 +13,28 @@ log = logging.getLogger(__name__)
 
 
 def retrier(attempts=5, sleeptime=10, max_sleeptime=300, sleepscale=1.5, jitter=1):
     """
     A generator function that sleeps between retries, handles exponential
     backoff and jitter. The action you are retrying is meant to run after
     retrier yields.
 
-    At each iteration, we sleep for sleeptime + random.randint(-jitter, jitter).
+    At each iteration, we sleep for sleeptime + random.uniform(-jitter, jitter).
     Afterwards sleeptime is multiplied by sleepscale for the next iteration.
 
     Args:
         attempts (int): maximum number of times to try; defaults to 5
         sleeptime (float): how many seconds to sleep between tries; defaults to
-                           60s (one minute)
+                           10 seconds
         max_sleeptime (float): the longest we'll sleep, in seconds; defaults to
                                300s (five minutes)
         sleepscale (float): how much to multiply the sleep time by each
                             iteration; defaults to 1.5
-        jitter (int): random jitter to introduce to sleep time each iteration.
+        jitter (float): random jitter to introduce to sleep time each iteration.
                       the amount is chosen at random between [-jitter, +jitter]
                       defaults to 1
 
     Yields:
         None, a maximum of `attempts` number of times
 
     Example:
         >>> n = 0
@@ -63,61 +63,63 @@ def retrier(attempts=5, sleeptime=10, ma
 
     sleeptime_real = sleeptime
     for _ in range(attempts):
         log.debug("attempt %i/%i", _ + 1, attempts)
 
         yield sleeptime_real
 
         if jitter:
-            sleeptime_real = sleeptime + random.randint(-jitter, jitter)
+            sleeptime_real = sleeptime + random.uniform(-jitter, jitter)
             # our jitter should scale along with the sleeptime
-            jitter = int(jitter * sleepscale)
+            jitter = jitter * sleepscale
         else:
             sleeptime_real = sleeptime
 
         sleeptime *= sleepscale
 
         if sleeptime_real > max_sleeptime:
             sleeptime_real = max_sleeptime
 
         # Don't need to sleep the last time
         if _ < attempts - 1:
             log.debug("sleeping for %.2fs (attempt %i/%i)", sleeptime_real, _ + 1, attempts)
             time.sleep(sleeptime_real)
 
 
 def retry(action, attempts=5, sleeptime=60, max_sleeptime=5 * 60,
           sleepscale=1.5, jitter=1, retry_exceptions=(Exception,),
-          cleanup=None, args=(), kwargs={}):
+          cleanup=None, args=(), kwargs={}, log_args=True):
     """
     Calls an action function until it succeeds, or we give up.
 
     Args:
         action (callable): the function to retry
         attempts (int): maximum number of times to try; defaults to 5
         sleeptime (float): how many seconds to sleep between tries; defaults to
                            60s (one minute)
         max_sleeptime (float): the longest we'll sleep, in seconds; defaults to
                                300s (five minutes)
         sleepscale (float): how much to multiply the sleep time by each
                             iteration; defaults to 1.5
-        jitter (int): random jitter to introduce to sleep time each iteration.
+        jitter (float): random jitter to introduce to sleep time each iteration.
                       the amount is chosen at random between [-jitter, +jitter]
                       defaults to 1
         retry_exceptions (tuple): tuple of exceptions to be caught. If other
                                   exceptions are raised by action(), then these
                                   are immediately re-raised to the caller.
         cleanup (callable): optional; called if one of `retry_exceptions` is
                             caught. No arguments are passed to the cleanup
                             function; if your cleanup requires arguments,
                             consider using functools.partial or a lambda
                             function.
         args (tuple): positional arguments to call `action` with
         kwargs (dict): keyword arguments to call `action` with
+        log_args (bool): whether or not to include args and kwargs in log
+                         messages. Defaults to True.
 
     Returns:
         Whatever action(*args, **kwargs) returns
 
     Raises:
         Whatever action(*args, **kwargs) raises. `retry_exceptions` are caught
         up until the last attempt, in which case they are re-raised.
 
@@ -135,42 +137,43 @@ def retry(action, attempts=5, sleeptime=
         2
         3
         'success!'
     """
     assert callable(action)
     assert not cleanup or callable(cleanup)
 
     action_name = getattr(action, '__name__', action)
-    if args or kwargs:
-        log_attempt_format = ("retry: calling %s with args: %s,"
-                              " kwargs: %s, attempt #%%d"
-                              % (action_name, args, kwargs))
+    if log_args and (args or kwargs):
+        log_attempt_args = ("retry: calling %s with args: %s,"
+                            " kwargs: %s, attempt #%d",
+                            action_name, args, kwargs)
     else:
-        log_attempt_format = ("retry: calling %s, attempt #%%d"
-                              % action_name)
+        log_attempt_args = ("retry: calling %s, attempt #%d",
+                            action_name)
 
     if max_sleeptime < sleeptime:
-        log.debug("max_sleeptime %d less than sleeptime %d" % (
-            max_sleeptime, sleeptime))
+        log.debug("max_sleeptime %d less than sleeptime %d",
+                  max_sleeptime, sleeptime)
 
     n = 1
     for _ in retrier(attempts=attempts, sleeptime=sleeptime,
                      max_sleeptime=max_sleeptime, sleepscale=sleepscale,
                      jitter=jitter):
         try:
             logfn = log.info if n != 1 else log.debug
-            logfn(log_attempt_format, n)
+            log_attempt_args += (n, )
+            logfn(*log_attempt_args)
             return action(*args, **kwargs)
         except retry_exceptions:
             log.debug("retry: Caught exception: ", exc_info=True)
             if cleanup:
                 cleanup()
             if n == attempts:
-                log.info("retry: Giving up on %s" % action_name)
+                log.info("retry: Giving up on %s", action_name)
                 raise
             continue
         finally:
             n += 1
 
 
 def retriable(*retry_args, **retry_kwargs):
     """
--- a/third_party/python/redo/redo/cmd.py
+++ b/third_party/python/redo/redo/cmd.py
@@ -7,33 +7,33 @@ import logging
 from subprocess import check_call, CalledProcessError
 import sys
 
 from redo import retrying
 
 log = logging.getLogger(__name__)
 
 
-def main():
-    from argparse import ArgumentParser
+def main(argv):
+    from argparse import ArgumentParser, REMAINDER
 
     parser = ArgumentParser()
     parser.add_argument(
         "-a", "--attempts", type=int, default=5,
         help="How many times to retry.")
     parser.add_argument(
         "-s", "--sleeptime", type=int, default=60,
         help="How long to sleep between attempts. Sleeptime doubles after each attempt.")
     parser.add_argument(
         "-m", "--max-sleeptime", type=int, default=5*60,
         help="Maximum length of time to sleep between attempts (limits backoff length).")
     parser.add_argument("-v", "--verbose", action="store_true", default=False)
-    parser.add_argument("cmd", nargs="+", help="Command to run. Eg: wget http://blah")
+    parser.add_argument("cmd", nargs=REMAINDER, help="Command to run. Eg: wget http://blah")
 
-    args = parser.parse_args()
+    args = parser.parse_args(argv[1:])
 
     if args.verbose:
         logging.basicConfig(level=logging.INFO)
         logging.getLogger("retry").setLevel(logging.INFO)
     else:
         logging.basicConfig(level=logging.ERROR)
         logging.getLogger("retry").setLevel(logging.ERROR)
 
@@ -44,10 +44,11 @@ def main():
             r_check_call(args.cmd)
     except KeyboardInterrupt:
         sys.exit(-1)
     except Exception as e:
         log.error("Unable to run command after %d attempts" % args.attempts, exc_info=True)
         rc = getattr(e, "returncode", -2)
         sys.exit(rc)
 
+
 if __name__ == "__main__":
-    main()
+    main(sys.argv)
--- a/third_party/python/redo/setup.cfg
+++ b/third_party/python/redo/setup.cfg
@@ -1,8 +1,7 @@
 [wheel]
 universal = 1
 
 [egg_info]
 tag_build = 
 tag_date = 0
-tag_svn_revision = 0
 
--- a/third_party/python/redo/setup.py
+++ b/third_party/python/redo/setup.py
@@ -1,17 +1,17 @@
 try:
     from setuptools import setup
 except ImportError:
     from distutils.core import setup
 
 
 setup(
     name="redo",
-    version="1.6",
+    version="2.0.2",
     description="Utilities to retry Python callables.",
     author="Ben Hearsum",
     author_email="ben@hearsum.ca",
     packages=["redo"],
     entry_points={
         "console_scripts": ["retry = redo.cmd:main"],
     },
     url="https://github.com/bhearsum/redo",
--- a/third_party/python/requirements.in
+++ b/third_party/python/requirements.in
@@ -4,13 +4,14 @@ jsmin==2.1.0
 json-e==2.7.0
 mozilla-version==0.3.0
 pathlib2==2.3.2
 pip-tools==3.0.0
 pipenv==2018.5.18
 psutil==5.4.3
 pytest==3.6.2
 python-hglib==2.4
+redo==2.0.2
 requests==2.9.1
 six==1.10.0
 taskcluster==4.0.1
 virtualenv==15.2.0
 voluptuous==0.11.5
--- a/third_party/python/requirements.txt
+++ b/third_party/python/requirements.txt
@@ -70,16 +70,19 @@ py==1.5.4 \
     --hash=sha256:3fd59af7435864e1a243790d322d763925431213b6b8529c6ca71081ace3bbf7 \
     --hash=sha256:e31fb2767eb657cbde86c454f02e99cb846d3cd9d61b318525140214fdc0e98e \
     # via pytest
 pytest==3.6.2 \
     --hash=sha256:8ea01fc4fcc8e1b1e305252b4bc80a1528019ab99fd3b88666c9dc38d754406c \
     --hash=sha256:90898786b3d0b880b47645bae7b51aa9bbf1e9d1e4510c2cfd15dd65c70ea0cd
 python-hglib==2.4 \
     --hash=sha256:693d6ed92a6566e78802c7a03c256cda33d08c63ad3f00fcfa11379b184b9462
+redo==2.0.2 \
+    --hash=sha256:29159a717454e1f276c7c509b81357e167a0b9218c68adf8ca8b0499363877ad \
+    --hash=sha256:703603d61b4ae7fa14a9dce3db22d8789284e99be997f558137612e847ead3cb
 requests==2.9.1 \
     --hash=sha256:113fbba5531a9e34945b7d36b33a084e8ba5d0664b703c81a7c572d91919a5b8 \
     --hash=sha256:c577815dd00f1394203fc44eb979724b098f88264a9ef898ee45b8e5e9cf587f
 scandir==1.9.0 \
     --hash=sha256:04b8adb105f2ed313a7c2ef0f1cf7aff4871aa7a1883fa4d8c44b5551ab052d6 \
     --hash=sha256:1444134990356c81d12f30e4b311379acfbbcd03e0bab591de2696a3b126d58e \
     --hash=sha256:1b5c314e39f596875e5a95dd81af03730b338c277c54a454226978d5ba95dbb6 \
     --hash=sha256:346619f72eb0ddc4cf355ceffd225fa52506c92a2ff05318cfabd02a144e7c4e \