Bug 794509 - Part 1: mach subcommands are now defined through Python decorators; r=jhammel
authorGregory Szorc <gps@mozilla.com>
Fri, 05 Oct 2012 12:12:51 -0700
changeset 115640 15c2bcb1a982e748f9f88be3976096cf5d31b07c
parent 115639 f4c2a6f2b838ea1a59c6484355bcfbdc4923501d
child 115641 d04782dc009156e114f27da3a468321a99edc47b
push id1708
push userakeybl@mozilla.com
push dateMon, 19 Nov 2012 21:10:21 +0000
treeherdermozilla-beta@27b14fe50103 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersjhammel
bugs794509
milestone18.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 794509 - Part 1: mach subcommands are now defined through Python decorators; r=jhammel
python/mach/README.rst
python/mach/mach/base.py
python/mach/mach/build.py
python/mach/mach/main.py
python/mach/mach/registrar.py
python/mach/mach/settings.py
python/mach/mach/testing.py
python/mach/mach/warnings.py
--- a/python/mach/README.rst
+++ b/python/mach/README.rst
@@ -6,79 +6,57 @@ The *mach* driver is the command line in
 The *mach* driver is invoked by running the *mach* script or from
 instantiating the *Mach* class from the *mach.main* module.
 
 Implementing mach Commands
 --------------------------
 
 The *mach* driver follows the convention of popular tools like Git,
 Subversion, and Mercurial and provides a common driver for multiple
-sub-commands.
+subcommands.
+
+Subcommands are implemented by decorating a class inheritting from
+mozbuild.base.MozbuildObject and by decorating methods that act as subcommand
+handlers.
+
+Relevant decorators are defined in the *mach.base* module. There are the
+*Command* and *CommandArgument* decorators, which should be used on methods
+to denote that a specific method represents a handler for a mach subcommand.
+There is also the *CommandProvider* decorator, which is applied to a class
+to denote that it contains mach subcommands.
+
+Here is a complete example:
+
+    from mozbuild.base import MozbuildObject
 
-Modules inside *mach* typically contain 1 or more classes which
-inherit from *mach.base.ArgumentProvider*. Modules that inherit from
-this class are hooked up to the *mach* CLI driver. So, to add a new
-sub-command/action to *mach*, one simply needs to create a new class in
-the *mach* package which inherits from *ArgumentProvider*.
+    from mach.base import CommandArgument
+    from mach.base import CommandProvider
+    from mach.base import Command
+
+    @CommandProvider
+    class MyClass(MozbuildObject):
+
+        @Command('doit', help='Do ALL OF THE THINGS.')
+        @CommandArgument('--force', '-f', action='store_true',
+            help='Force doing it.')
+        def doit(self, force=False):
+            # Do stuff here.
+
+
+When the module is loaded, the decorators tell mach about all handlers. When
+mach runs, it takes the assembled metadata from these handlers and hooks it
+up to the command line driver. Under the hood, arguments passed to the
+decorators are being used as arguments to *argparse.ArgumentParser.add_parser()*
+and *argparse.ArgumentParser.add_argument()*. See the documentation for
+*argparse* for more.
 
 Currently, you also need to hook up some plumbing in
 *mach.main.Mach*. In the future, we hope to have automatic detection
 of submodules.
 
-Your command class performs the role of configuring the *mach* frontend
-argument parser as well as providing the methods invoked if a command is
-requested. These methods will take the user-supplied input, do something
-(likely by calling a backend function in a separate module), then format
-output to the terminal.
-
-The plumbing to hook up the arguments to the *mach* driver involves
-light magic. At *mach* invocation time, the driver creates a new
-*argparse* instance. For each registered class that provides commands,
-it calls the *populate_argparse* static method, passing it the parser
-instance.
-
-Your class's *populate_argparse* function should register sub-commands
-with the parser.
-
-For example, say you want to provide the *doitall* command. e.g. *mach
-doitall*. You would create the module *mach.doitall* and this
-module would contain the following class:
-
-    from mach.base import ArgumentProvider
-
-    class DoItAll(ArgumentProvider):
-        def run(self, more=False):
-            print 'I did it!'
-
-        @staticmethod
-        def populate_argparse(parser):
-            # Create the parser to handle the sub-command.
-            p = parser.add_parser('doitall', help='Do it all!')
-
-            p.add_argument('more', action='store_true', default=False,
-                help='Do more!')
-
-            # Tell driver that the handler for this sub-command is the
-            # method *run* on the class *DoItAll*.
-            p.set_defaults(cls=DoItAll, method='run')
-
-The most important line here is the call to *set_defaults*.
-Specifically, the *cls* and *method* parameters, which tell the driver
-which class to instantiate and which method to execute if this command
-is requested.
-
-The specified method will receive all arguments parsed from the command.
-It is important that you use named - not positional - arguments for your
-handler functions or things will blow up. This is because the mach driver
-is using the ``**kwargs`` notation to call the defined method.
-
-In the future, we may provide additional syntactical sugar to make all
-this easier. For example, we may provide decorators on methods to hook
-up commands and handlers.
-
 Minimizing Code in Mach
 -----------------------
 
 Mach is just a frontend. Therefore, code in this package should pertain to
 one of 3 areas:
 
 1. Obtaining user input (parsing arguments, prompting, etc)
 2. Calling into some other Python package
--- a/python/mach/mach/base.py
+++ b/python/mach/mach/base.py
@@ -1,13 +1,94 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import unicode_literals
 
+import types
 
-class ArgumentProvider(object):
-    """Base class for classes wishing to provide CLI arguments to mach."""
+from mach.registrar import register_method_handler
+
+
+def CommandProvider(cls):
+    """Class decorator to denote that it provides subcommands for Mach.
+
+    When this decorator is present, mach looks for commands being defined by
+    methods inside the class.
+    """
+
+    # The implementation of this decorator relies on the parse-time behavior of
+    # decorators. When the module is imported, the method decorators (like
+    # @Command and @CommandArgument) are called *before* this class decorator.
+    # The side-effect of the method decorators is to store specifically-named
+    # attributes on the function types. We just scan over all functions in the
+    # class looking for the side-effects of the method decorators.
+
+    # We scan __dict__ because we only care about the classes own attributes,
+    # not inherited ones. If we did inherited attributes, we could potentially
+    # define commands multiple times. We also sort keys so commands defined in
+    # the same class are grouped in a sane order.
+    for attr in sorted(cls.__dict__.keys()):
+        value = cls.__dict__[attr]
+
+        if not isinstance(value, types.FunctionType):
+            continue
+
+        parser_args = getattr(value, '_mach_command', None)
+        if parser_args is None:
+            continue
+
+        arguments = getattr(value, '_mach_command_args', None)
+
+        register_method_handler(cls, attr, (parser_args[0], parser_args[1]),
+            arguments or [])
+
+    return cls
+
+
+class Command(object):
+    """Decorator for functions or methods that provide a mach subcommand.
 
-    @staticmethod
-    def populate_argparse(parser):
-        raise Exception("populate_argparse not implemented.")
+    The decorator accepts arguments that would be passed to add_parser() of an
+    ArgumentParser instance created via add_subparsers(). Essentially, it
+    accepts the arguments one would pass to add_argument().
+
+    For example:
+
+        @Command('foo', help='Run the foo action')
+        def foo(self):
+            pass
+    """
+    def __init__(self, *args, **kwargs):
+        self._command_args = (args, kwargs)
+
+    def __call__(self, func):
+        func._mach_command = self._command_args
+
+        return func
+
+
+class CommandArgument(object):
+    """Decorator for additional arguments to mach subcommands.
+
+    This decorator should be used to add arguments to mach commands. Arguments
+    to the decorator are proxied to ArgumentParser.add_argument().
+
+    For example:
+
+        @Command('foo', help='Run the foo action')
+        @CommandArgument('-b', '--bar', action='store_true', default=False,
+            help='Enable bar mode.')
+        def foo(self):
+            pass
+    """
+    def __init__(self, *args, **kwargs):
+        self._command_args = (args, kwargs)
+
+    def __call__(self, func):
+        command_args = getattr(func, '_mach_command_args', [])
+
+        command_args.append(self._command_args)
+
+        func._mach_command_args = command_args
+
+        return func
--- a/python/mach/mach/build.py
+++ b/python/mach/mach/build.py
@@ -2,23 +2,26 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import unicode_literals
 
 import logging
 import os
 
-from mach.base import ArgumentProvider
+from mach.base import CommandProvider
+from mach.base import Command
 from mozbuild.base import MozbuildObject
 
 
-class Build(MozbuildObject, ArgumentProvider):
+@CommandProvider
+class Build(MozbuildObject):
     """Interface to build the tree."""
 
+    @Command('build', help='Build the tree.')
     def build(self):
         # This code is only meant to be temporary until the more robust tree
         # building code in bug 780329 lands.
         from mozbuild.compilation.warnings import WarningsCollector
         from mozbuild.compilation.warnings import WarningsDatabase
 
         warnings_path = self._get_state_filename('warnings.json')
         warnings_database = WarningsDatabase()
@@ -44,13 +47,8 @@ class Build(MozbuildObject, ArgumentProv
         self._run_make(srcdir=True, filename='client.mk', line_handler=on_line,
             log=False, print_directory=False)
 
         self.log(logging.WARNING, 'warning_summary',
             {'count': len(warnings_collector.database)},
             '{count} compiler warnings present.')
 
         warnings_database.save_to_file(warnings_path)
-
-    @staticmethod
-    def populate_argparse(parser):
-        build = parser.add_parser('build', help='Build the tree.')
-        build.set_defaults(cls=Build, method='build')
--- a/python/mach/mach/main.py
+++ b/python/mach/mach/main.py
@@ -12,31 +12,26 @@ import codecs
 import logging
 import os
 import sys
 
 from mozbuild.base import BuildConfig
 from mozbuild.config import ConfigSettings
 from mozbuild.logger import LoggingManager
 
+from mach.registrar import populate_argument_parser
+
 # Import sub-command modules
 # TODO Bug 794509 do this via auto-discovery. Update README once this is
 # done.
 from mach.build import Build
 from mach.settings import Settings
 from mach.testing import Testing
 from mach.warnings import Warnings
 
-# Classes inheriting from ArgumentProvider that provide commands.
-HANDLERS = [
-    Build,
-    Settings,
-    Testing,
-    Warnings,
-]
 
 # Classes inheriting from ConfigProvider that provide settings.
 # TODO this should come from auto-discovery somehow.
 SETTINGS_PROVIDERS = [
     BuildConfig,
 ]
 
 # Settings for argument parser that don't get proxied to sub-module. i.e. these
@@ -279,13 +274,11 @@ To see more help for a specific command,
             metavar='FILENAME', type=argparse.FileType('ab'),
             help='Filename to write log data to.')
         global_group.add_argument('--log-interval', dest='log_interval',
             action='store_true', default=False,
             help='Prefix log line with interval from last message rather '
                 'than relative time. Note that this is NOT execution time '
                 'if there are parallel operations.')
 
-        # Register argument action providers with us.
-        for cls in HANDLERS:
-            cls.populate_argparse(subparser)
+        populate_argument_parser(subparser)
 
         return parser
new file mode 100644
--- /dev/null
+++ b/python/mach/mach/registrar.py
@@ -0,0 +1,21 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import unicode_literals
+
+
+class_handlers = []
+
+def register_method_handler(cls, method, parser_args, arguments):
+    class_handlers.append((cls, method, parser_args, arguments))
+
+
+def populate_argument_parser(parser):
+    for cls, method, parser_args, arguments in class_handlers:
+        p = parser.add_parser(*parser_args[0], **parser_args[1])
+
+        for arg in arguments:
+            p.add_argument(*arg[0], **arg[1])
+
+        p.set_defaults(cls=cls, method=method)
--- a/python/mach/mach/settings.py
+++ b/python/mach/mach/settings.py
@@ -2,50 +2,43 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import print_function, unicode_literals
 
 from textwrap import TextWrapper
 
 from mozbuild.base import MozbuildObject
-from mach.base import ArgumentProvider
+from mach.base import CommandProvider
+from mach.base import Command
 
-class Settings(MozbuildObject, ArgumentProvider):
+@CommandProvider
+class Settings(MozbuildObject):
     """Interact with settings for mach.
 
     Currently, we only provide functionality to view what settings are
     available. In the future, this module will be used to modify settings, help
     people create configs via a wizard, etc.
     """
+    @Command('settings-list', help='Show available config settings.')
     def list_settings(self):
         """List available settings in a concise list."""
         for section in sorted(self.settings):
             for option in sorted(self.settings[section]):
                 short, full = self.settings.option_help(section, option)
                 print('%s.%s -- %s' % (section, option, short))
 
+    @Command('settings-create',
+        help='Print a new settings file with usage info.')
     def create(self):
         """Create an empty settings file with full documentation."""
         wrapper = TextWrapper(initial_indent='# ', subsequent_indent='# ')
 
         for section in sorted(self.settings):
             print('[%s]' % section)
             print('')
 
             for option in sorted(self.settings[section]):
                 short, full = self.settings.option_help(section, option)
 
                 print(wrapper.fill(full))
                 print(';%s =' % option)
                 print('')
-
-    @staticmethod
-    def populate_argparse(parser):
-        lst = parser.add_parser('settings-list',
-            help='Show available config settings.')
-
-        lst.set_defaults(cls=Settings, method='list_settings')
-
-        create = parser.add_parser('settings-create',
-            help='Print a new settings file with usage info.')
-
-        create.set_defaults(cls=Settings, method='create')
--- a/python/mach/mach/testing.py
+++ b/python/mach/mach/testing.py
@@ -1,108 +1,90 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import unicode_literals
 
-from mach.base import ArgumentProvider
+from mach.base import CommandArgument
+from mach.base import CommandProvider
+from mach.base import Command
 from mozbuild.base import MozbuildObject
 
 
 generic_help = 'Test to run. Can be specified as a single JS file, a ' +\
 'directory, or omitted. If omitted, the entire test suite is executed.'
 
+suites = ['mochitest-plain', 'mochitest-chrome', 'mochitest-browser', 'all']
 
-class Testing(MozbuildObject, ArgumentProvider):
+
+@CommandProvider
+class Testing(MozbuildObject):
     """Provides commands for running tests."""
 
+    @Command('test', help='Perform tests.')
+    @CommandArgument('suite', default='all', choices=suites, nargs='?',
+        help='Test suite to run.')
     def run_suite(self, suite):
         from mozbuild.testing.suite import Suite
 
         s = self._spawn(Suite)
         s.run_suite(suite)
 
+    @Command('mochitest-plain', help='Run a plain mochitest.')
+    @CommandArgument('test_file', default=None, nargs='?', metavar='TEST',
+        help=generic_help)
+    def run_mochitest_plain(self, test_file):
+        self.run_mochitest(test_file, 'plain')
+
+    @Command('mochitest-chrome', help='Run a chrome mochitest.')
+    @CommandArgument('test_file', default=None, nargs='?', metavar='TEST',
+        help=generic_help)
+    def run_mochitest_chrome(self, test_file):
+        self.run_mochitest(test_file, 'chrome')
+
+    @Command('mochitest-browser', help='Run a mochitest with browser chrome.')
+    @CommandArgument('test_file', default=None, nargs='?', metavar='TEST',
+        help=generic_help)
+    def run_mochitest_browser(self, test_file):
+        self.run_mochitest(test_file, 'browser')
+
+    @Command('mochitest-a11y', help='Run an a11y mochitest.')
+    @CommandArgument('test_file', default=None, nargs='?', metavar='TEST',
+        help=generic_help)
+    def run_mochitest_a11y(self, test_file):
+        self.run_mochitest(test_file, 'a11y')
+
     def run_mochitest(self, test_file, flavor):
         from mozbuild.testing.mochitest import MochitestRunner
 
         mochitest = self._spawn(MochitestRunner)
         mochitest.run_mochitest_test(test_file, flavor)
 
-    def run_reftest(self, test_file, flavor):
+    @Command('reftest', help='Run a reftest.')
+    @CommandArgument('test_file', default=None, nargs='?', metavar='TEST',
+        help=generic_help)
+    def run_reftest(self, test_file):
+        self._run_reftest(test_file, 'reftest')
+
+    @Command('crashtest', help='Run a crashtest.')
+    @CommandArgument('test_file', default=None, nargs='?', metavar='TEST',
+        help=generic_help)
+    def run_crashtest(self, test_file):
+        self._run_reftest(test_file, 'crashtest')
+
+    def _run_reftest(self, test_file, flavor):
         from mozbuild.testing.reftest import ReftestRunner
 
         reftest = self._spawn(ReftestRunner)
         reftest.run_reftest_test(test_file, flavor)
 
+    @Command('xpcshell-test', help='Run an xpcshell test.')
+    @CommandArgument('test_file', default='all', nargs='?', metavar='TEST',
+        help=generic_help)
+    @CommandArgument('--debug', '-d', action='store_true',
+        help='Run test in a debugger.')
     def run_xpcshell_test(self, **params):
         from mozbuild.testing.xpcshell import XPCShellRunner
 
         xpcshell = self._spawn(XPCShellRunner)
         xpcshell.run_test(**params)
 
-    @staticmethod
-    def populate_argparse(parser):
-        # Whole suites.
-        group = parser.add_parser('test', help="Perform tests.")
-
-        suites = set(['xpcshell', 'mochitest-plain', 'mochitest-chrome',
-            'mochitest-browser', 'all'])
-
-        group.add_argument('suite', default='all', choices=suites, nargs='?',
-            help="Test suite to run.")
-
-        group.set_defaults(cls=Testing, method='run_suite', suite='all')
-
-        # Mochitest-style
-        mochitest_plain = parser.add_parser('mochitest-plain',
-            help='Run a plain mochitest.')
-        mochitest_plain.add_argument('test_file', default=None, nargs='?',
-            metavar='TEST', help=generic_help)
-        mochitest_plain.set_defaults(cls=Testing, method='run_mochitest',
-            flavor='plain')
-
-        mochitest_chrome = parser.add_parser('mochitest-chrome',
-            help='Run a chrome mochitest.')
-        mochitest_chrome.add_argument('test_file', default=None, nargs='?',
-            metavar='TEST', help=generic_help)
-        mochitest_chrome.set_defaults(cls=Testing, method='run_mochitest',
-            flavor='chrome')
-
-        mochitest_browser = parser.add_parser('mochitest-browser',
-            help='Run a mochitest with browser chrome.')
-        mochitest_browser.add_argument('test_file', default=None, nargs='?',
-            metavar='TEST', help=generic_help)
-        mochitest_browser.set_defaults(cls=Testing, method='run_mochitest',
-            flavor='browser')
-
-        mochitest_a11y = parser.add_parser('mochitest-a11y',
-            help='Run an a11y mochitest.')
-        mochitest_a11y.add_argument('test_file', default=None, nargs='?',
-            metavar='TEST', help=generic_help)
-        mochitest_a11y.set_defaults(cls=Testing, method='run_mochitest',
-            flavor='a11y')
-
-        # Reftest-style
-        reftest = parser.add_parser('reftest',
-            help='Run a reftest.')
-        reftest.add_argument('test_file', default=None, nargs='?',
-            metavar='TEST', help=generic_help)
-        reftest.set_defaults(cls=Testing, method='run_reftest',
-            flavor='reftest')
-
-        crashtest = parser.add_parser('crashtest',
-            help='Run a crashtest.')
-        crashtest.add_argument('test_file', default=None, nargs='?',
-            metavar='TEST', help=generic_help)
-        crashtest.set_defaults(cls=Testing, method='run_reftest',
-            flavor='crashtest')
-
-        # XPCShell-style
-        xpcshell = parser.add_parser('xpcshell-test',
-            help="Run an individual xpcshell test.")
-
-        xpcshell.add_argument('test_file', default='all', nargs='?',
-            metavar='TEST', help=generic_help)
-        xpcshell.add_argument('--debug', '-d', action='store_true',
-            help='Run test in debugger.')
-
-        xpcshell.set_defaults(cls=Testing, method='run_xpcshell_test')
--- a/python/mach/mach/warnings.py
+++ b/python/mach/mach/warnings.py
@@ -2,21 +2,24 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import print_function, unicode_literals
 
 import operator
 import os
 
-from mach.base import ArgumentProvider
+from mach.base import CommandArgument
+from mach.base import CommandProvider
+from mach.base import Command
 from mozbuild.base import MozbuildObject
 
 
-class Warnings(MozbuildObject, ArgumentProvider):
+@CommandProvider
+class Warnings(MozbuildObject):
     """Provide commands for inspecting warnings."""
 
     @property
     def database_path(self):
         return self._get_state_filename('warnings.json')
 
     @property
     def database(self):
@@ -26,30 +29,39 @@ class Warnings(MozbuildObject, ArgumentP
 
         database = WarningsDatabase()
 
         if os.path.exists(path):
             database.load_from_file(path)
 
         return database
 
+    @Command('warnings-summary',
+        help='Show a summary of compiler warnings.')
+    @CommandArgument('report', default=None, nargs='?',
+        help='Warnings report to display. If not defined, show the most '
+            'recent report.')
     def summary(self, report=None):
         database = self.database
 
         type_counts = database.type_counts
         sorted_counts = sorted(type_counts.iteritems(),
             key=operator.itemgetter(1))
 
         total = 0
         for k, v in sorted_counts:
             print('%d\t%s' % (v, k))
             total += v
 
         print('%d\tTotal' % total)
 
+    @Command('warnings-list', help='Show a list of compiler warnings.')
+    @CommandArgument('report', default=None, nargs='?',
+        help='Warnings report to display. If not defined, show the most '
+            'recent report.')
     def list(self, report=None):
         database = self.database
 
         by_name = sorted(database.warnings)
 
         for warning in by_name:
             filename = warning['filename']
 
@@ -58,26 +70,8 @@ class Warnings(MozbuildObject, ArgumentP
 
             if warning['column'] is not None:
                 print('%s:%d:%d [%s] %s' % (filename, warning['line'],
                     warning['column'], warning['flag'], warning['message']))
             else:
                 print('%s:%d [%s] %s' % (filename, warning['line'],
                     warning['flag'], warning['message']))
 
-    @staticmethod
-    def populate_argparse(parser):
-        summary = parser.add_parser('warnings-summary',
-            help='Show a summary of compiler warnings.')
-
-        summary.add_argument('report', default=None, nargs='?',
-            help='Warnings report to display. If not defined, show '
-                 'the most recent report')
-
-        summary.set_defaults(cls=Warnings, method='summary', report=None)
-
-        lst = parser.add_parser('warnings-list',
-            help='Show a list of compiler warnings')
-        lst.add_argument('report', default=None, nargs='?',
-            help='Warnings report to display. If not defined, show '
-                 'the most recent report.')
-
-        lst.set_defaults(cls=Warnings, method='list', report=None)