Bug 784841 - Part 3: Implement sandbox to data translation layer; r=ted,glandium
authorGregory Szorc <gps@mozilla.com>
Tue, 15 Jan 2013 22:21:23 -0800
changeset 119047 d8ea5b8be44d077721bd272cbe9e027d892eeedc
parent 119046 ec072cee05025335f0637de7e1dab95623fe8a56
child 119048 ee04a251a5efd3837033efe2e087fc0a000f63fe
push id24188
push usergszorc@mozilla.com
push dateWed, 16 Jan 2013 20:38:49 +0000
treeherdermozilla-central@ff2e30afa205 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersted, glandium
bugs784841
milestone21.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 784841 - Part 3: Implement sandbox to data translation layer; r=ted,glandium This patch takes MozbuildSandbox instances emitted from BuildReader and converts them to data structures.
python/mozbuild/mozbuild/frontend/README.rst
python/mozbuild/mozbuild/frontend/data.py
python/mozbuild/mozbuild/frontend/emitter.py
python/mozbuild/mozbuild/test/frontend/test_emitter.py
--- a/python/mozbuild/mozbuild/frontend/README.rst
+++ b/python/mozbuild/mozbuild/frontend/README.rst
@@ -4,30 +4,33 @@ mozbuild.frontend
 
 The mozbuild.frontend package is of sufficient importance and complexity
 to warrant its own README file. If you are looking for documentation on
 how the build system gets started, you've come to the right place.
 
 Overview
 ========
 
-The build system is defined by a bunch of files in the source tree called
-*mozbuild* files. Each *mozbuild* file defines a unique part of the overall
-build system. This includes information like "compile file X," "copy this
-file here," "link these files together to form a library." Together,
-all the *mozbuild* files define how the entire build system works.
+Tree metadata (including the build system) is defined by a collection of
+files in the source tree called *mozbuild* files. These typically are files
+named *moz.build*. But, the actual name can vary.
+
+Each *mozbuild* file defines basic metadata about the part of the tree
+(typically directory scope) it resides in. This includes build system
+configuration, such as the list of C++ files to compile or headers to install
+or libraries to link together.
 
 *mozbuild* files are actually Python scripts. However, their execution
 is governed by special rules. This will be explained later.
 
 Once a *mozbuild* file has executed, it is converted into a set of static
 data structures.
 
 The set of all data structures from all relevant *mozbuild* files
-constitutes the current build configuration.
+constitute all of the metadata from the tree.
 
 How *mozbuild* Files Work
 =========================
 
 As stated above, *mozbuild* files are actually Python scripts. However,
 their behavior is very different from what you would expect if you executed
 the file using the standard Python interpreter from the command line.
 
@@ -81,57 +84,54 @@ How Sandboxes are Converted into Data St
 ================================================
 
 The output of a *mozbuild* file execution is essentially a dict of all
 the special UPPERCASE variables populated during its execution. While these
 dicts are data structures, they aren't the final data structures that
 represent the build configuration.
 
 We feed the *mozbuild* execution output (actually *reader.MozbuildSandbox*
-instances) into a *BuildDefinitionEmitter* class instance. This class is
-defined in the *emitter* module. *BuildDefinitionEmitter* converts the
-*MozbuildSandbox* instances into instances of the *BuildDefinition*-derived
+instances) into a *TreeMetadataEmitter* class instance. This class is
+defined in the *emitter* module. *TreeMetadataEmitter* converts the
+*MozbuildSandbox* instances into instances of the *TreeMetadata*-derived
 classes from the *data* module.
 
 All the classes in the *data* module define a domain-specific
-component of the build configuration. File compilation and IDL generation
-are separate classes, for example. The only thing these classes have in
-common is that they inherit from *BuildDefinition*, which is merely an
-abstract base class.
+component of the tree metadata, including build configuration. File compilation
+and IDL generation are separate classes, for example. The only thing these
+classes have in common is that they inherit from *TreeMetadata*, which is
+merely an abstract base class.
 
-The set of all emitted *BuildDefinition* instances (converted from executed
-*mozbuild* files) constitutes the aggregate build configuration. This is
-the authoritative definition of the build system and is what's used by
-all downstream consumers, such as backends. There is no monolithic build
-system configuration class. Instead, the build system configuration is
-modeled as a collection/iterable of *BuildDefinition*.
+The set of all emitted *TreeMetadata* instances (converted from executed
+*mozbuild* files) constitutes the aggregate tree metadata. This is the
+the authoritative definition of the build system, etc and is what's used by
+all downstream consumers, such as build backends. There is no monolithic
+class or data structure. Instead, the tree metadata is modeled as a collection
+of *TreeMetadata* instances.
 
 There is no defined mapping between the number of
-*MozbuildSandbox*/*moz.build* instances and *BuildDefinition* instances.
-Some *mozbuild* files will emit only 1 *BuildDefinition* instance. Some
+*MozbuildSandbox*/*moz.build* instances and *TreeMetadata* instances.
+Some *mozbuild* files will emit only 1 *TreeMetadata* instance. Some
 will emit 7. Some may even emit 0!
 
 The purpose of this *emitter* layer between the raw *mozbuild* execution
-result and *BuildDefinition* is to facilitate additional normalization and
-verification of the output. The downstream consumer of the build
-configuration are build backends. And, there are several of these. There
-are common functions shared by backends related to examining the build
-configuration. It makes sense to move this functionality upstream as part
-of a shared pipe. Thus, *BuildDefinitionEmitter* exists.
+result and *TreeMetadata* is to facilitate additional normalization and
+verification of the output. There are multiple downstream consumers of
+this data and there is common functionality shared between them. An
+abstraction layer that provides high-level filtering is a useful feature.
+Thus *TreeMetadataEmitter* exists.
 
 Other Notes
 ===========
 
-*reader.BuildReader* and *emitter.BuildDefinitionEmitter* have a nice
+*reader.BuildReader* and *emitter.TreeMetadataEmitter* have a nice
 stream-based API courtesy of generators. When you hook them up properly,
-*BuildDefinition* instances can be consumed before all *mozbuild* files have
+*TreeMetadata* instances can be consumed before all *mozbuild* files have
 been read. This means that errors down the pipe can trigger before all
 upstream tasks (such as executing and converting) are complete. This should
 reduce the turnaround time in the event of errors. This likely translates to
 a more rapid pace for implementing backends, which require lots of iterative
 runs through the entire system.
 
-In theory, the frontend to the build system is generic and could be used
-by any project. In practice, parts are specifically tailored towards
-Mozilla's needs. With a little work, the core build system bits could be
-separated into its own package, independent of the Mozilla bits. Or, one
-could simply replace the Mozilla-specific pieces in the *variables*, *data*,
-and *emitter* modules to reuse the core logic.
+Lots of code in this sub-module is applicable to other systems, not just
+Mozilla's. However, some of the code is tightly coupled. If there is a will
+to extract the generic bits for re-use in other projects, that can and should
+be done.
new file mode 100644
--- /dev/null
+++ b/python/mozbuild/mozbuild/frontend/data.py
@@ -0,0 +1,82 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+r"""Data structures representing Mozilla's source tree.
+
+The frontend files are parsed into static data structures. These data
+structures are defined in this module.
+
+All data structures of interest are children of the TreeMetadata class.
+
+Logic for populating these data structures is not defined in this class.
+Instead, what we have here are dumb container classes. The emitter module
+contains the code for converting executed mozbuild files into these data
+structures.
+"""
+
+from __future__ import unicode_literals
+
+from collections import OrderedDict
+
+
+class TreeMetadata(object):
+    """Base class for all data being captured."""
+
+
+class SandboxDerived(TreeMetadata):
+    """Build object derived from a single MozbuildSandbox instance.
+
+    It holds fields common to all sandboxes. This class is likely never
+    instantiated directly but is instead derived from.
+    """
+
+    __slots__ = (
+        'objdir',
+        'relativedir',
+        'srcdir',
+        'topobjdir',
+        'topsrcdir',
+    )
+
+    def __init__(self, sandbox):
+        self.topsrcdir = sandbox['TOPSRCDIR']
+        self.topobjdir = sandbox['TOPOBJDIR']
+
+        self.relativedir = sandbox['RELATIVEDIR']
+        self.srcdir = sandbox['SRCDIR']
+        self.objdir = sandbox['OBJDIR']
+
+
+class DirectoryTraversal(SandboxDerived):
+    """Describes how directory traversal for building should work.
+
+    This build object is likely only of interest to the recursive make backend.
+    Other build backends should (ideally) not attempt to mimic the behavior of
+    the recursive make backend. The only reason this exists is to support the
+    existing recursive make backend while the transition to mozbuild frontend
+    files is complete and we move to a more optimal build backend.
+
+    Fields in this class correspond to similarly named variables in the
+    frontend files.
+    """
+    __slots__ = (
+        'dirs',
+        'parallel_dirs',
+        'tool_dirs',
+        'test_dirs',
+        'test_tool_dirs',
+        'tier_dirs',
+        'tier_static_dirs',
+    )
+
+    def __init__(self, sandbox):
+        SandboxDerived.__init__(self, sandbox)
+
+        self.dirs = []
+        self.parallel_dirs = []
+        self.tool_dirs = []
+        self.test_dirs = []
+        self.test_tool_dirs = []
+        self.tier_dirs = OrderedDict()
+        self.tier_static_dirs = OrderedDict()
new file mode 100644
--- /dev/null
+++ b/python/mozbuild/mozbuild/frontend/emitter.py
@@ -0,0 +1,59 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import unicode_literals
+
+from .data import DirectoryTraversal
+from .reader import MozbuildSandbox
+
+
+class TreeMetadataEmitter(object):
+    """Converts the executed mozbuild files into data structures.
+
+    This is a bridge between reader.py and data.py. It takes what was read by
+    reader.BuildReader and converts it into the classes defined in the data
+    module.
+    """
+
+    def __init__(self, config):
+        self.config = config
+
+    def emit(self, output):
+        """Convert the BuildReader output into data structures.
+
+        The return value from BuildReader.read_topsrcdir() (a generator) is
+        typically fed into this function.
+        """
+        for out in output:
+            if isinstance(out, MozbuildSandbox):
+                for o in self.emit_from_sandbox(out):
+                    yield o
+            else:
+                raise Exception('Unhandled output type: %s' % out)
+
+    def emit_from_sandbox(self, sandbox):
+        """Convert a MozbuildSandbox to tree metadata objects.
+
+        This is a generator of mozbuild.frontend.data.SandboxDerived instances.
+        """
+
+        # We always emit a directory traversal descriptor. This is needed by
+        # the recursive make backend.
+        for o in self._emit_directory_traversal_from_sandbox(sandbox): yield o
+
+    def _emit_directory_traversal_from_sandbox(self, sandbox):
+        o = DirectoryTraversal(sandbox)
+        o.dirs = sandbox.get('DIRS', [])
+        o.parallel_dirs = sandbox.get('PARALLEL_DIRS', [])
+        o.tool_dirs = sandbox.get('TOOL_DIRS', [])
+        o.test_dirs = sandbox.get('TEST_DIRS', [])
+        o.test_tool_dirs = sandbox.get('TEST_TOOL_DIRS', [])
+
+        if 'TIERS' in sandbox:
+            for tier in sandbox['TIERS']:
+                o.tier_dirs[tier] = sandbox['TIERS'][tier]['regular']
+                o.tier_static_dirs[tier] = sandbox['TIERS'][tier]['static']
+
+        yield o
+
new file mode 100644
--- /dev/null
+++ b/python/mozbuild/mozbuild/test/frontend/test_emitter.py
@@ -0,0 +1,85 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import unicode_literals
+
+import os
+import unittest
+
+from mozbuild.frontend.data import DirectoryTraversal
+from mozbuild.frontend.emitter import TreeMetadataEmitter
+from mozbuild.frontend.reader import BuildReader
+
+from mozbuild.test.common import MockConfig
+
+
+data_path = os.path.abspath(os.path.dirname(__file__))
+data_path = os.path.join(data_path, 'data')
+
+
+class TestEmitterBasic(unittest.TestCase):
+    def reader(self, name):
+        config = MockConfig(os.path.join(data_path, name))
+        config.substs['ENABLE_TESTS'] = '1'
+
+        return BuildReader(config)
+
+    def test_dirs_traversal_simple(self):
+        reader = self.reader('traversal-simple')
+        emitter = TreeMetadataEmitter(reader.config)
+
+        objs = list(emitter.emit(reader.read_topsrcdir()))
+
+        self.assertEqual(len(objs), 4)
+
+        for o in objs:
+            self.assertIsInstance(o, DirectoryTraversal)
+            self.assertEqual(o.parallel_dirs, [])
+            self.assertEqual(o.tool_dirs, [])
+            self.assertEqual(o.test_dirs, [])
+            self.assertEqual(o.test_tool_dirs, [])
+            self.assertEqual(len(o.tier_dirs), 0)
+            self.assertEqual(len(o.tier_static_dirs), 0)
+
+        reldirs = [o.relativedir for o in objs]
+        self.assertEqual(reldirs, ['', 'foo', 'foo/biz', 'bar'])
+
+        dirs = [o.dirs for o in objs]
+        self.assertEqual(dirs, [['foo', 'bar'], ['biz'], [], []])
+
+    def test_traversal_all_vars(self):
+        reader = self.reader('traversal-all-vars')
+        emitter = TreeMetadataEmitter(reader.config)
+
+        objs = list(emitter.emit(reader.read_topsrcdir()))
+        self.assertEqual(len(objs), 6)
+
+        for o in objs:
+            self.assertIsInstance(o, DirectoryTraversal)
+
+        reldirs = set([o.relativedir for o in objs])
+        self.assertEqual(reldirs, set(['', 'parallel', 'regular', 'test',
+            'test_tool', 'tool']))
+
+        for o in objs:
+            reldir = o.relativedir
+
+            if reldir == '':
+                self.assertEqual(o.dirs, ['regular'])
+                self.assertEqual(o.parallel_dirs, ['parallel'])
+                self.assertEqual(o.test_dirs, ['test'])
+                self.assertEqual(o.test_tool_dirs, ['test_tool'])
+                self.assertEqual(o.tool_dirs, ['tool'])
+
+    def test_tier_simple(self):
+        reader = self.reader('traversal-tier-simple')
+        emitter = TreeMetadataEmitter(reader.config)
+
+        objs = list(emitter.emit(reader.read_topsrcdir()))
+        self.assertEqual(len(objs), 6)
+
+        reldirs = [o.relativedir for o in objs]
+        self.assertEqual(reldirs, ['', 'foo', 'foo/biz', 'foo_static', 'bar',
+            'baz'])
+