deleted file mode 100644 --- a/build/docs/files-metadata.rst +++ /dev/null @@ -1,178 +0,0 @@ -.. _mozbuild_files_metadata: - -============== -Files Metadata -============== - -:ref:`mozbuild-files` provide a mechanism for attaching metadata to -files. Essentially, you define some flags to set on a file or file -pattern. Later, some tool or process queries for metadata attached to a -file of interest and it does something intelligent with that data. - -Defining Metadata -================= - -Files metadata is defined by using the -:ref:`Files Sub-Context <mozbuild_subcontext_Files>` in ``moz.build`` -files. e.g.:: - - with Files('**/Makefile.in'): - BUG_COMPONENT = ('Core', 'Build Config') - -This working example says, *for all Makefile.in files in all directories -in this one and underneath it, set the Bugzilla component to -Core :: Build Config*. - -For more info, read the -:ref:`docs on Files <mozbuild_subcontext_Files>`. - -How Metadata is Read -==================== - -``Files`` metadata is extracted in :ref:`mozbuild_fs_reading_mode`. - -Reading starts by specifying a set of files whose metadata you are -interested in. For each file, the filesystem is walked to the root -of the source directory. Any ``moz.build`` encountered during this -walking are marked as relevant to the file. - -Let's say you have the following filesystem content:: - - /moz.build - /root_file - /dir1/moz.build - /dir1/foo - /dir1/subdir1/foo - /dir2/foo - -For ``/root_file``, the relevant ``moz.build`` files are just -``/moz.build``. - -For ``/dir1/foo`` and ``/dir1/subdir1/foo``, the relevant files are -``/moz.build`` and ``/dir1/moz.build``. - -For ``/dir2``, the relevant file is just ``/moz.build``. - -Once the list of relevant ``moz.build`` files is obtained, each -``moz.build`` file is evaluated. Root ``moz.build`` file first, -leaf-most files last. This follows the rules of -:ref:`mozbuild_fs_reading_mode`, with the set of evaluated ``moz.build`` -files being controlled by filesystem content, not ``DIRS`` variables. - -The file whose metadata is being resolved maps to a set of ``moz.build`` -files which in turn evaluates to a list of contexts. For file metadata, -we only care about one of these contexts: -:ref:`Files <mozbuild_subcontext_Files>`. - -We start with an empty ``Files`` instance to represent the file. As -we encounter a *files sub-context*, we see if it is appropriate to -this file. If it is, we apply its values. This process is repeated -until all *files sub-contexts* have been applied or skipped. The final -state of the ``Files`` instance is used to represent the metadata for -this particular file. - -It may help to visualize this. Say we have 2 ``moz.build`` files:: - - # /moz.build - with Files('*.cpp'): - BUG_COMPONENT = ('Core', 'XPCOM') - - with Files('**/*.js'): - BUG_COMPONENT = ('Firefox', 'General') - - # /foo/moz.build - with Files('*.js'): - BUG_COMPONENT = ('Another', 'Component') - -Querying for metadata for the file ``/foo/test.js`` will reveal 3 -relevant ``Files`` sub-contexts. They are evaluated as follows: - -1. ``/moz.build - Files('*.cpp')``. Does ``/*.cpp`` match - ``/foo/test.js``? **No**. Ignore this context. -2. ``/moz.build - Files('**/*.js')``. Does ``/**/*.js`` match - ``/foo/test.js``? **Yes**. Apply ``BUG_COMPONENT = ('Firefox', 'General')`` - to us. -3. ``/foo/moz.build - Files('*.js')``. Does ``/foo/*.js`` match - ``/foo/test.js``? **Yes**. Apply - ``BUG_COMPONENT = ('Another', 'Component')``. - -At the end of execution, we have -``BUG_COMPONENT = ('Another', 'Component')`` as the metadata for -``/foo/test.js``. - -One way to look at file metadata is as a stack of data structures. -Each ``Files`` sub-context relevant to a given file is applied on top -of the previous state, starting from an empty state. The final state -wins. - -.. _mozbuild_files_metadata_finalizing: - -Finalizing Values -================= - -The default behavior of ``Files`` sub-context evaluation is to apply new -values on top of old. In most circumstances, this results in desired -behavior. However, there are circumstances where this may not be -desired. There is thus a mechanism to *finalize* or *freeze* values. - -Finalizing values is useful for scenarios where you want to prevent -wildcard matches from overwriting previously-set values. This is useful -for one-off files. - -Let's take ``Makefile.in`` files as an example. The build system module -policy dictates that ``Makefile.in`` files are part of the ``Build -Config`` module and should be reviewed by peers of that module. However, -there exist ``Makefile.in`` files in many directories in the source -tree. Without finalization, a ``*`` or ``**`` wildcard matching rule -would match ``Makefile.in`` files and overwrite their metadata. - -Finalizing of values is performed by setting the ``FINAL`` variable -on ``Files`` sub-contexts. See the -:ref:`Files documentation <mozbuild_subcontext_Files>` for more. - -Here is an example with ``Makefile.in`` files, showing how it is -possible to finalize the ``BUG_COMPONENT`` value.:: - - # /moz.build - with Files('**/Makefile.in'): - BUG_COMPONENT = ('Core', 'Build Config') - FINAL = True - - # /foo/moz.build - with Files('**'): - BUG_COMPONENT = ('Another', 'Component') - -If we query for metadata of ``/foo/Makefile.in``, both ``Files`` -sub-contexts match the file pattern. However, since ``BUG_COMPONENT`` is -marked as finalized by ``/moz.build``, the assignment from -``/foo/moz.build`` is ignored. The final value for ``BUG_COMPONENT`` -is ``('Core', 'Build Config')``. - -Here is another example:: - - with Files('*.cpp'): - BUG_COMPONENT = ('One-Off', 'For C++') - FINAL = True - - with Files('**'): - BUG_COMPONENT = ('Regular', 'Component') - -For every files except ``foo.cpp``, the bug component will be resolved -as ``Regular :: Component``. However, ``foo.cpp`` has its value of -``One-Off :: For C++`` preserved because it is finalized. - -.. important:: - - ``FINAL`` only applied to variables defined in a context. - - If you want to mark one variable as finalized but want to leave - another mutable, you'll need to use 2 ``Files`` contexts. - -Guidelines for Defining Metadata -================================ - -In general, values defined towards the root of the source tree are -generic and become more specific towards the leaves. For example, -the ``BUG_COMPONENT`` for ``/browser`` might be ``Firefox :: General`` -whereas ``/browser/components/preferences`` would list -``Firefox :: Preferences``.
--- a/build/docs/mozbuild-files.rst +++ b/build/docs/mozbuild-files.rst @@ -25,18 +25,16 @@ different, it's doubtful most ``moz.buil error if executed by a vanilla Python interpreter (e.g. ``python moz.build``. The following properties make execution of ``moz.build`` files special: 1. The execution environment exposes a limited subset of Python. 2. There is a special set of global symbols and an enforced naming convention of symbols. -3. Some symbols are inherited from previously-executed ``moz.build`` - files. The limited subset of Python is actually an extremely limited subset. Only a few symbols from ``__builtins__`` are exposed. These include ``True``, ``False``, and ``None``. Global functions like ``import``, ``print``, and ``open`` aren't available. Without these, ``moz.build`` files can do very little. *This is by design*. The execution sandbox treats all ``UPPERCASE`` variables specially. Any @@ -65,87 +63,26 @@ data structures in this module are consu the sandbox. There are tests to ensure that the set of symbols exposed to an empty sandbox are all defined in the ``context`` module. This module also contains documentation for each symbol, so nothing can sneak into the sandbox without being explicitly defined and documented. Reading and Traversing moz.build Files ====================================== -The process for reading ``moz.build`` files roughly consists of: - -1. Start at the root ``moz.build`` (``<topsrcdir>/moz.build``). -2. Evaluate the ``moz.build`` file in a new sandbox. -3. Emit the main *context* and any *sub-contexts* from the executed - sandbox. -4. Extract a set of ``moz.build`` files to execute next. -5. For each additional ``moz.build`` file, goto #2 and repeat until all - referenced files have executed. - -From the perspective of the consumer, the output of reading is a stream -of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each -``Context`` defines a particular aspect of data. Consumers iterate over -these objects and do something with the data inside. Each object is -essentially a dictionary of all the ``UPPERCASE`` variables populated -during its execution. - -.. note:: - - Historically, there was only one ``context`` per ``moz.build`` file. - As the number of things tracked by ``moz.build`` files grew and more - and more complex processing was desired, it was necessary to split these - contexts into multiple logical parts. It is now common to emit - multiple contexts per ``moz.build`` file. - -Build System Reading Mode -------------------------- - -The traditional mode of evaluation of ``moz.build`` files is what's -called *build system traversal mode.* In this mode, the ``CONFIG`` -variable in each ``moz.build`` sandbox is populated from data coming -from ``config.status``, which is produced by ``configure``. +The process responsible for reading ``moz.build`` files simply starts at +a root ``moz.build`` file, processes it, emits the globals namespace to +a consumer, and then proceeds to process additional referenced +``moz.build`` files from the original file. The consumer then examines +the globals/``UPPERCASE`` variables set as part of execution and then +converts the data therein to Python class instances. -During evaluation, ``moz.build`` files often make decisions conditional -on the state of the build configuration. e.g. *only compile foo.cpp if -feature X is enabled*. - -In this mode, traversal of ``moz.build`` files is governed by variables -like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child -directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build`` -file and ``foo/moz.build`` would be evaluated. - -.. _mozbuild_fs_reading_mode: - -Filesystem Reading Mode ------------------------ - -There is an alternative reading mode that doesn't involve the build -system and doesn't use ``DIRS`` variables to control traversal into -child directories. This mode is called *filesystem reading mode*. - -In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty -object. Accessing all but a few special variables will return an empty -value. This means that nearly all ``if CONFIG['FOO']:`` branches will -not be taken. - -Instead of using content from within the evaluated ``moz.build`` -file to drive traversal into subsequent ``moz.build`` files, the set -of files to evaluate is controlled by the thing doing the reading. - -A single ``moz.build`` file is not guaranteed to be executable in -isolation. Instead, we must evaluate all *parent* ``moz.build`` files -first. For example, in order to evaluate ``/foo/moz.build``, one must -execute ``/moz.build`` and have its state influence the execution of -``/foo/moz.build``. - -Filesystem reading mode is utilized to power the -:ref:`mozbuild_files_metadata` feature. - -Technical Details ------------------ +The executed Python sandbox is essentially represented as a dictionary +of all the special ``UPPERCASE`` variables populated during its +execution. The code for reading ``moz.build`` files lives in :py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results (:py:class:`mozbuild.frontend.context.Context`) are passed into :py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a domain-specific component of tree metdata. e.g. there will be separate classes that represent a JavaScript file vs a compiled C++ file or test @@ -158,16 +95,19 @@ each. Depending on the content of the `` object derived or 100. The purpose of the ``emitter`` layer between low-level sandbox execution and metadata representation is to facilitate a unified normalization and verification step. There are multiple downstream consumers of the ``moz.build``-derived data and many will perform the same actions. This logic can be complicated, so we have a component dedicated to it. +Other Notes +=========== + :py:class:`mozbuild.frontend.reader.BuildReader`` and :py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a stream-based API courtesy of generators. When you hook them up properly, the :py:mod:`mozbuild.frontend.data` classes are emitted before all ``moz.build`` files have been read. This means that downstream errors are raised soon after sandbox execution. Lots of the code for evaluating Python sandboxes is applicable to
--- a/build/gyp.mozbuild +++ b/build/gyp.mozbuild @@ -90,17 +90,17 @@ flavors = { 'Darwin': 'mac' if CONFIG['MOZ_WIDGET_TOOLKIT'] == 'cocoa' else 'ios', 'SunOS': 'solaris', 'GNU/kFreeBSD': 'freebsd', 'DragonFly': 'dragonfly', 'FreeBSD': 'freebsd', 'NetBSD': 'netbsd', 'OpenBSD': 'openbsd', } -gyp_vars['OS'] = flavors.get(os) +gyp_vars['OS'] = flavors[os] arches = { 'x86_64': 'x64', 'x86': 'ia32', } gyp_vars['target_arch'] = arches.get(CONFIG['CPU_ARCH'], CONFIG['CPU_ARCH'])
--- a/config/moz.build +++ b/config/moz.build @@ -25,14 +25,13 @@ if CONFIG['HOST_OS_ARCH'] != 'WINNT': if CONFIG['GKMEDIAS_SHARED_LIBRARY']: DEFINES['GKMEDIAS_SHARED_LIBRARY'] = True if CONFIG['MOZ_SHARED_ICU']: DEFINES['MOZ_SHARED_ICU'] = True PYTHON_UNIT_TESTS += [ - 'tests/test_mozbuild_reading.py', 'tests/unit-expandlibs.py', 'tests/unit-mozunit.py', 'tests/unit-nsinstall.py', 'tests/unit-printprereleasesuffix.py', ]
deleted file mode 100644 --- a/config/tests/test_mozbuild_reading.py +++ /dev/null @@ -1,69 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -from __future__ import unicode_literals - -import os -import unittest - -from mozunit import main - -from mozbuild.base import MozbuildObject -from mozbuild.frontend.reader import ( - BuildReader, - EmptyConfig, -) - - -class TestMozbuildReading(unittest.TestCase): - # This hack is needed to appease running in automation. - def setUp(self): - self._old_env = dict(os.environ) - os.environ.pop('MOZCONFIG', None) - os.environ.pop('MOZ_OBJDIR', None) - - def tearDown(self): - os.environ.clear() - os.environ.update(self._old_env) - - def _mozbuilds(self, reader): - if not hasattr(self, '_mozbuild_paths'): - self._mozbuild_paths = set(reader.all_mozbuild_paths()) - - return self._mozbuild_paths - - def test_filesystem_traversal_reading(self): - """Reading moz.build according to filesystem traversal works. - - We attempt to read every known moz.build file via filesystem traversal. - - If this test fails, it means that metadata extraction will fail. - """ - mb = MozbuildObject.from_environment(detect_virtualenv_mozinfo=False) - config = mb.config_environment - reader = BuildReader(config) - all_paths = self._mozbuilds(reader) - paths, contexts = reader.read_relevant_mozbuilds(all_paths) - self.assertEqual(set(paths), all_paths) - self.assertGreaterEqual(len(contexts), len(paths)) - - def test_filesystem_traversal_no_config(self): - """Reading moz.build files via filesystem traversal mode with no build config. - - This is similar to the above test except no build config is applied. - This will likely fail in more scenarios than the above test because a - lot of moz.build files assumes certain variables are present. - """ - here = os.path.abspath(os.path.dirname(__file__)) - root = os.path.normpath(os.path.join(here, '..', '..')) - config = EmptyConfig(root) - reader = BuildReader(config) - all_paths = self._mozbuilds(reader) - paths, contexts = reader.read_relevant_mozbuilds(all_paths) - self.assertEqual(set(paths.keys()), all_paths) - self.assertGreaterEqual(len(contexts), len(paths)) - - -if __name__ == '__main__': - main()
--- a/ipc/glue/moz.build +++ b/ipc/glue/moz.build @@ -82,17 +82,17 @@ else: if CONFIG['OS_ARCH'] == 'Linux': UNIFIED_SOURCES += [ 'ProcessUtils_linux.cpp', ] elif CONFIG['OS_ARCH'] in ('DragonFly', 'FreeBSD', 'NetBSD', 'OpenBSD'): UNIFIED_SOURCES += [ 'ProcessUtils_bsd.cpp' ] -elif CONFIG['OS_ARCH'] == 'Darwin': +elif CONFIG['OS_ARCH'] in ('Darwin'): UNIFIED_SOURCES += [ 'ProcessUtils_mac.mm' ] else: UNIFIED_SOURCES += [ 'ProcessUtils_none.cpp', ]
--- a/moz.build +++ b/moz.build @@ -61,13 +61,13 @@ if CONFIG['COMPILE_ENVIRONMENT'] and not ] if CONFIG['BUILD_CTYPES']: DIRS += ['config/external/ffi'] if CONFIG['USE_ICU']: DIRS += ['config/external/icu'] DIRS += ['js/src'] -if not CONFIG['JS_STANDALONE'] and CONFIG['MOZ_BUILD_APP']: +if not CONFIG['JS_STANDALONE']: # Bring in the configuration for the configured application. include('/' + CONFIG['MOZ_BUILD_APP'] + '/app.mozbuild') include('build/templates.mozbuild')
--- a/netwerk/build/moz.build +++ b/netwerk/build/moz.build @@ -28,18 +28,17 @@ LOCAL_INCLUDES += [ '/netwerk/protocol/about', '/netwerk/protocol/app', '/netwerk/socket', '/netwerk/streamconv', '/netwerk/streamconv/converters', ] protocols = CONFIG['NECKO_PROTOCOLS'].copy() -if 'about' in protocols: - protocols.remove('about') +protocols.remove("about") LOCAL_INCLUDES += sorted([ '/netwerk/protocol/%s' % d for d in protocols ]) if CONFIG['OS_ARCH'] == 'WINNT': LOCAL_INCLUDES += [ '/netwerk/system/win32', ]
--- a/python/mozbuild/mozbuild/frontend/reader.py +++ b/python/mozbuild/mozbuild/frontend/reader.py @@ -24,24 +24,20 @@ import logging import os import sys import textwrap import time import tokenize import traceback import types -from collections import ( - defaultdict, - OrderedDict, -) +from collections import OrderedDict from io import StringIO from mozbuild.util import ( - EmptyValue, memoize, ReadOnlyDefaultDict, ReadOnlyDict, ) from mozbuild.backend.configenvironment import ConfigEnvironment from mozpack.files import FileFinder @@ -62,74 +58,31 @@ from .sandbox import ( from .context import ( Context, ContextDerivedValue, FUNCTIONS, VARIABLES, DEPRECATION_HINTS, SPECIAL_VARIABLES, SUBCONTEXTS, - SubContext, TemplateContext, ) if sys.version_info.major == 2: text_type = unicode type_type = types.TypeType else: text_type = str type_type = type def log(logger, level, action, params, formatter): logger.log(level, formatter, extra={'action': action, 'params': params}) -class EmptyConfig(object): - """A config object that is empty. - - This config object is suitable for using with a BuildReader on a vanilla - checkout, without any existing configuration. The config is simply - bootstrapped from a top source directory path. - """ - class PopulateOnGetDict(ReadOnlyDefaultDict): - """A variation on ReadOnlyDefaultDict that populates during .get(). - - This variation is needed because CONFIG uses .get() to access members. - Without it, None (instead of our EmptyValue types) would be returned. - """ - def get(self, key, default=None): - return self[key] - - def __init__(self, topsrcdir): - self.topsrcdir = topsrcdir - self.topobjdir = '' - - self.substs = self.PopulateOnGetDict(EmptyValue, { - # These 2 variables are used semi-frequently and it isn't worth - # changing all the instances. - b'MOZ_APP_NAME': b'empty', - b'MOZ_CHILD_PROCESS_NAME': b'empty', - # Set manipulations are performed within the moz.build files. But - # set() is not an exposed symbol, so we can't create an empty set. - b'NECKO_PROTOCOLS': set(), - # Needed to prevent js/src's config.status from loading. - b'JS_STANDALONE': b'1', - }) - udict = {} - for k, v in self.substs.items(): - if isinstance(v, str): - udict[k.decode('utf-8')] = v.decode('utf-8') - else: - udict[k] = v - self.substs_unicode = self.PopulateOnGetDict(EmptyValue, udict) - self.defines = self.substs - self.external_source_dir = None - - def is_read_allowed(path, config): """Whether we are allowed to load a mozbuild file at the specified path. This is used as cheap security to ensure the build is isolated to known source directories. We are allowed to read from the main source directory and any defined external source directories. The latter is to allow 3rd party applications @@ -177,35 +130,30 @@ class MozbuildSandbox(Sandbox): self._log = logging.getLogger(__name__) self.metadata = dict(metadata) exports = self.metadata.get('exports', {}) self.exports = set(exports.keys()) context.update(exports) self.templates = self.metadata.setdefault('templates', {}) - self.special_variables = self.metadata.setdefault('special_variables', - SPECIAL_VARIABLES) - self.functions = self.metadata.setdefault('functions', FUNCTIONS) - self.subcontext_types = self.metadata.setdefault('subcontexts', - SUBCONTEXTS) def __getitem__(self, key): - if key in self.special_variables: - return self.special_variables[key][0](self._context) - if key in self.functions: - return self._create_function(self.functions[key]) - if key in self.subcontext_types: - return self._create_subcontext(self.subcontext_types[key]) + if key in SPECIAL_VARIABLES: + return SPECIAL_VARIABLES[key][0](self._context) + if key in FUNCTIONS: + return self._create_function(FUNCTIONS[key]) + if key in SUBCONTEXTS: + return self._create_subcontext(SUBCONTEXTS[key]) if key in self.templates: return self._create_template_function(self.templates[key]) return Sandbox.__getitem__(self, key) def __setitem__(self, key, value): - if key in self.special_variables or key in self.functions or key in self.subcontext_types: + if key in SPECIAL_VARIABLES or key in FUNCTIONS or key in SUBCONTEXTS: raise KeyError() if key in self.exports: self._context[key] = value self.exports.remove(key) return Sandbox.__setitem__(self, key, value) def exec_file(self, path): @@ -403,30 +351,22 @@ class MozbuildSandbox(Sandbox): sandbox needs a function to execute. This is what this method returns. That function creates a new sandbox for execution of the template. After the template is executed, the data from its execution is merged with the context of the calling sandbox. """ func, code, path = template def template_function(*args, **kwargs): - context = TemplateContext(self._context._allowed_variables, - self._context.config) + context = TemplateContext(VARIABLES, self._context.config) context.add_source(self._context.current_path) for p in self._context.all_paths: context.add_source(p) - sandbox = MozbuildSandbox(context, metadata={ - # We should arguably set these defaults to something else. - # Templates, for example, should arguably come from the state - # of the sandbox from when the template was declared, not when - # it was instantiated. Bug 1137319. - 'functions': self.metadata.get('functions', {}), - 'special_variables': self.metadata.get('special_variables', {}), - 'subcontexts': self.metadata.get('subcontexts', {}), + sandbox = MozbuildSandbox(context, { 'templates': self.metadata.get('templates', {}) }) for k, v in inspect.getcallargs(func, *args, **kwargs).items(): sandbox[k] = v sandbox.exec_source(code, path) # This is gross, but allows the merge to happen. Eventually, the @@ -1027,21 +967,16 @@ class BuildReader(object): config.topobjdir = topobjdir config.external_source_dir = None context = Context(VARIABLES, config) sandbox = MozbuildSandbox(context, metadata=metadata) sandbox.exec_file(path) context.execution_time = time.time() - time_start - # Yield main context before doing any processing. This gives immediate - # consumers an opportunity to change state before our remaining - # processing is performed. - yield context - # We first collect directories populated in variables. dir_vars = ['DIRS'] if context.config.substs.get('ENABLE_TESTS', False) == '1': dir_vars.append('TEST_DIRS') dirs = [(v, context[v]) for v in dir_vars if v in context] @@ -1076,16 +1011,18 @@ class BuildReader(object): non_unified_sources = non_unified_sources): gyp_context.update(gyp_dir.sandbox_vars) gyp_contexts.append(gyp_context) for gyp_context in gyp_contexts: context['DIRS'].append(mozpath.relpath(gyp_context.objdir, context.objdir)) sandbox.subcontexts.append(gyp_context) + yield context + for subcontext in sandbox.subcontexts: yield subcontext # Traverse into referenced files. # It's very tempting to use a set here. Unfortunately, the recursive # make backend needs order preserved. Once we autogenerate all backend # files, we should be able to convert this to a set. @@ -1093,21 +1030,22 @@ class BuildReader(object): for var, var_dirs in dirs: for d in var_dirs: if d in recurse_info: raise SandboxValidationError( 'Directory (%s) registered multiple times in %s' % ( mozpath.relpath(d, context.srcdir), var), context) recurse_info[d] = {} - for key in sandbox.metadata: - if key == 'exports': - sandbox.recompute_exports() - - recurse_info[d][key] = dict(sandbox.metadata[key]) + if 'templates' in sandbox.metadata: + recurse_info[d]['templates'] = dict( + sandbox.metadata['templates']) + if 'exports' in sandbox.metadata: + sandbox.recompute_exports() + recurse_info[d]['exports'] = dict(sandbox.metadata['exports']) for path, child_metadata in recurse_info.items(): child_path = path.join('moz.build') # Ensure we don't break out of the topsrcdir. We don't do realpath # because it isn't necessary. If there are symlinks in the srcdir, # that's not our problem. We're not a hosted application: we don't # need to worry about security too much. @@ -1119,121 +1057,8 @@ class BuildReader(object): if not descend: continue for res in self.read_mozbuild(child_path, context.config, metadata=child_metadata): yield res self._execution_stack.pop() - - def _find_relevant_mozbuilds(self, paths): - """Given a set of filesystem paths, find all relevant moz.build files. - - We assume that a moz.build file in the directory ancestry of a given path - is relevant to that path. Let's say we have the following files on disk:: - - moz.build - foo/moz.build - foo/baz/moz.build - foo/baz/file1 - other/moz.build - other/file2 - - If ``foo/baz/file1`` is passed in, the relevant moz.build files are - ``moz.build``, ``foo/moz.build``, and ``foo/baz/moz.build``. For - ``other/file2``, the relevant moz.build files are ``moz.build`` and - ``other/moz.build``. - - Returns a dict of input paths to a list of relevant moz.build files. - The root moz.build file is first and the leaf-most moz.build is last. - """ - root = self.config.topsrcdir - result = {} - - @memoize - def exists(path): - return os.path.exists(path) - - def itermozbuild(path): - subpath = '' - yield 'moz.build' - for part in mozpath.split(path): - subpath = mozpath.join(subpath, part) - yield mozpath.join(subpath, 'moz.build') - - for path in sorted(paths): - path = mozpath.normpath(path) - if os.path.isabs(path): - if not mozpath.basedir(path, [root]): - raise Exception('Path outside topsrcdir: %s' % path) - path = mozpath.relpath(path, root) - - result[path] = [p for p in itermozbuild(path) - if exists(mozpath.join(root, p))] - - return result - - def read_relevant_mozbuilds(self, paths): - """Read and process moz.build files relevant for a set of paths. - - For an iterable of relative-to-root filesystem paths ``paths``, - find all moz.build files that may apply to them based on filesystem - hierarchy and read those moz.build files. - - The return value is a 2-tuple. The first item is a dict mapping each - input filesystem path to a list of Context instances that are relevant - to that path. The second item is a list of all Context instances. Each - Context instance is in both data structures. - """ - relevants = self._find_relevant_mozbuilds(paths) - - topsrcdir = self.config.topsrcdir - - # Source moz.build file to directories to traverse. - dirs = defaultdict(set) - # Relevant path to absolute paths of relevant contexts. - path_mozbuilds = {} - - # There is room to improve this code (and the code in - # _find_relevant_mozbuilds) to better handle multiple files in the same - # directory. Bug 1136966 tracks. - for path, mbpaths in relevants.items(): - path_mozbuilds[path] = [mozpath.join(topsrcdir, p) for p in mbpaths] - - for i, mbpath in enumerate(mbpaths[0:-1]): - source_dir = mozpath.dirname(mbpath) - target_dir = mozpath.dirname(mbpaths[i + 1]) - - d = mozpath.normpath(mozpath.join(topsrcdir, mbpath)) - dirs[d].add(mozpath.relpath(target_dir, source_dir)) - - # Exporting doesn't work reliably in tree traversal mode. Override - # the function to no-op. - functions = dict(FUNCTIONS) - def export(sandbox): - return lambda varname: None - functions['export'] = tuple([export] + list(FUNCTIONS['export'][1:])) - - metadata = { - 'functions': functions, - } - - contexts = defaultdict(list) - all_contexts = [] - for context in self.read_mozbuild(mozpath.join(topsrcdir, 'moz.build'), - self.config, metadata=metadata): - # Explicitly set directory traversal variables to override default - # traversal rules. - if not isinstance(context, SubContext): - for v in ('DIRS', 'GYP_DIRS', 'TEST_DIRS'): - context[v][:] = [] - - context['DIRS'] = sorted(dirs[context.main_path]) - - contexts[context.main_path].append(context) - all_contexts.append(context) - - result = {} - for path, paths in path_mozbuilds.items(): - result[path] = reduce(lambda x, y: x + y, (contexts[p] for p in paths), []) - - return result, all_contexts
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/bad-assignment/moz.build +++ /dev/null @@ -1,2 +0,0 @@ -with Files('*'): - BUG_COMPONENT = 'bad value'
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/different-matchers/moz.build +++ /dev/null @@ -1,4 +0,0 @@ -with Files('*.jsm'): - BUG_COMPONENT = ('Firefox', 'JS') -with Files('*.cpp'): - BUG_COMPONENT = ('Firefox', 'C++')
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/final/moz.build +++ /dev/null @@ -1,3 +0,0 @@ -with Files('**/Makefile.in'): - BUG_COMPONENT = ('Core', 'Build Config') - FINAL = True
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/final/subcomponent/moz.build +++ /dev/null @@ -1,2 +0,0 @@ -with Files('**'): - BUG_COMPONENT = ('Another', 'Component')
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/moz.build +++ /dev/null @@ -1,2 +0,0 @@ -with Files('**'): - BUG_COMPONENT = ('default_product', 'default_component')
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/simple/moz.build +++ /dev/null @@ -1,2 +0,0 @@ -with Files('*'): - BUG_COMPONENT = ('Core', 'Build Config')
deleted file mode 100644 --- a/python/mozbuild/mozbuild/test/frontend/data/files-info/bug_component/static/moz.build +++ /dev/null @@ -1,5 +0,0 @@ -with Files('foo'): - BUG_COMPONENT = ('FooProduct', 'FooComponent') - -with Files('bar'): - BUG_COMPONENT = ('BarProduct', 'BarComponent')
--- a/python/mozbuild/mozbuild/test/frontend/test_reader.py +++ b/python/mozbuild/mozbuild/test/frontend/test_reader.py @@ -232,87 +232,11 @@ class TestBuildReader(unittest.TestCase) contexts = list(reader.read_topsrcdir()) self.assertEqual(len(contexts), 4) self.assertEqual([context.relsrcdir for context in contexts], ['', 'foo', 'foo/baz', 'bar']) self.assertEqual([context['XPIDL_MODULE'] for context in contexts], ['foobar', 'foobar', 'baz', 'foobar']) - def test_find_relevant_mozbuilds(self): - reader = self.reader('reader-relevant-mozbuild') - - # Absolute paths outside topsrcdir are rejected. - with self.assertRaises(Exception): - reader._find_relevant_mozbuilds(['/foo']) - - # File in root directory. - paths = reader._find_relevant_mozbuilds(['file']) - self.assertEqual(paths, {'file': ['moz.build']}) - - # File in child directory. - paths = reader._find_relevant_mozbuilds(['d1/file1']) - self.assertEqual(paths, {'d1/file1': ['moz.build', 'd1/moz.build']}) - - # Multiple files in same directory. - paths = reader._find_relevant_mozbuilds(['d1/file1', 'd1/file2']) - self.assertEqual(paths, { - 'd1/file1': ['moz.build', 'd1/moz.build'], - 'd1/file2': ['moz.build', 'd1/moz.build']}) - - # Missing moz.build from missing intermediate directory. - paths = reader._find_relevant_mozbuilds( - ['d1/no-intermediate-moz-build/child/file']) - self.assertEqual(paths, { - 'd1/no-intermediate-moz-build/child/file': [ - 'moz.build', 'd1/moz.build', 'd1/no-intermediate-moz-build/child/moz.build']}) - - # Lots of empty directories. - paths = reader._find_relevant_mozbuilds([ - 'd1/parent-is-far/dir1/dir2/dir3/file']) - self.assertEqual(paths, { - 'd1/parent-is-far/dir1/dir2/dir3/file': - ['moz.build', 'd1/moz.build', 'd1/parent-is-far/moz.build']}) - - # Lots of levels. - paths = reader._find_relevant_mozbuilds([ - 'd1/every-level/a/file', 'd1/every-level/b/file']) - self.assertEqual(paths, { - 'd1/every-level/a/file': [ - 'moz.build', - 'd1/moz.build', - 'd1/every-level/moz.build', - 'd1/every-level/a/moz.build', - ], - 'd1/every-level/b/file': [ - 'moz.build', - 'd1/moz.build', - 'd1/every-level/moz.build', - 'd1/every-level/b/moz.build', - ], - }) - - # Different root directories. - paths = reader._find_relevant_mozbuilds(['d1/file', 'd2/file', 'file']) - self.assertEqual(paths, { - 'file': ['moz.build'], - 'd1/file': ['moz.build', 'd1/moz.build'], - 'd2/file': ['moz.build', 'd2/moz.build'], - }) - - def test_read_relevant_mozbuilds(self): - reader = self.reader('reader-relevant-mozbuild') - - paths, contexts = reader.read_relevant_mozbuilds(['d1/every-level/a/file', - 'd1/every-level/b/file', 'd2/file']) - self.assertEqual(len(paths), 3) - self.assertEqual(len(contexts), 6) - - self.assertEqual([ctx.relsrcdir for ctx in paths['d1/every-level/a/file']], - ['', 'd1', 'd1/every-level', 'd1/every-level/a']) - self.assertEqual([ctx.relsrcdir for ctx in paths['d1/every-level/b/file']], - ['', 'd1', 'd1/every-level', 'd1/every-level/b']) - self.assertEqual([ctx.relsrcdir for ctx in paths['d2/file']], - ['', 'd2']) - if __name__ == '__main__': main()
--- a/python/mozbuild/mozbuild/test/test_util.py +++ b/python/mozbuild/mozbuild/test/test_util.py @@ -27,17 +27,16 @@ from mozbuild.util import ( memoized_property, resolve_target_to_make, MozbuildDeletionError, HierarchicalStringList, HierarchicalStringListWithFlagsFactory, StrictOrderingOnAppendList, StrictOrderingOnAppendListWithFlagsFactory, TypedList, - TypedNamedTuple, UnsortedError, ) if sys.version_info[0] == 3: str_type = 'str' else: str_type = 'unicode' @@ -659,43 +658,16 @@ class TypedTestStrictOrderingOnAppendLis with self.assertRaises(UnsortedError): cls(['c', 'b', 'a']) with self.assertRaises(ValueError): cls(['a', 'b', 3]) self.assertEqual(len(l), 3) - -class TestTypedNamedTuple(unittest.TestCase): - def test_simple(self): - FooBar = TypedNamedTuple('FooBar', [('foo', unicode), ('bar', int)]) - - t = FooBar(foo='foo', bar=2) - self.assertEquals(type(t), FooBar) - self.assertEquals(t.foo, 'foo') - self.assertEquals(t.bar, 2) - self.assertEquals(t[0], 'foo') - self.assertEquals(t[1], 2) - - FooBar('foo', 2) - - with self.assertRaises(TypeError): - FooBar('foo', 'not integer') - with self.assertRaises(TypeError): - FooBar(2, 4) - - # Passing a tuple as the first argument is the same as passing multiple - # arguments. - t1 = ('foo', 3) - t2 = FooBar(t1) - self.assertEquals(type(t2), FooBar) - self.assertEqual(FooBar(t1), FooBar('foo', 3)) - - class TestGroupUnifiedFiles(unittest.TestCase): FILES = ['%s.cpp' % letter for letter in string.ascii_lowercase] def test_multiple_files(self): mapping = list(group_unified_files(self.FILES, 'Unified', 'cpp', 5)) def check_mapping(index, expected_num_source_files): (unified_file, source_files) = mapping[index]
--- a/python/mozbuild/mozbuild/util.py +++ b/python/mozbuild/mozbuild/util.py @@ -13,17 +13,16 @@ import difflib import errno import functools import hashlib import itertools import os import stat import sys import time -import types from collections import ( defaultdict, OrderedDict, ) from StringIO import StringIO @@ -46,27 +45,16 @@ def hash_file(path, hasher=None): if not len(data): break h.update(data) return h.hexdigest() -class EmptyValue(unicode): - """A dummy type that behaves like an empty string and sequence. - - This type exists in order to support - :py:class:`mozbuild.frontend.reader.EmptyConfig`. It should likely not be - used elsewhere. - """ - def __init__(self): - super(EmptyValue, self).__init__() - - class ReadOnlyDict(dict): """A read-only dictionary.""" def __init__(self, *args, **kwargs): dict.__init__(self, *args, **kwargs) def __delitem__(self, key): raise Exception('Object does not support deletion.') @@ -261,28 +249,28 @@ class ListMixin(object): def __setslice__(self, i, j, sequence): if not isinstance(sequence, list): raise ValueError('List can only be sliced with other list instances.') return super(ListMixin, self).__setslice__(i, j, sequence) def __add__(self, other): - # Allow None and EmptyValue is a special case because it makes undefined - # variable references in moz.build behave better. - other = [] if isinstance(other, (types.NoneType, EmptyValue)) else other + # Allow None is a special case because it makes undefined variable + # references in moz.build behave better. + other = [] if other is None else other if not isinstance(other, list): raise ValueError('Only lists can be appended to lists.') new_list = self.__class__(self) new_list.extend(other) return new_list def __iadd__(self, other): - other = [] if isinstance(other, (types.NoneType, EmptyValue)) else other + other = [] if other is None else other if not isinstance(other, list): raise ValueError('Only lists can be appended to lists.') return super(ListMixin, self).__iadd__(other) class List(ListMixin, list): """A list specialized for moz.build environments. @@ -824,66 +812,16 @@ class memoized_property(object): def __get__(self, instance, cls): name = '_%s' % self.func.__name__ if not hasattr(instance, name): setattr(instance, name, self.func(instance)) return getattr(instance, name) -def TypedNamedTuple(name, fields): - """Factory for named tuple types with strong typing. - - Arguments are an iterable of 2-tuples. The first member is the - the field name. The second member is a type the field will be validated - to be. - - Construction of instances varies from ``collections.namedtuple``. - - First, if a single tuple argument is given to the constructor, this is - treated as the equivalent of passing each tuple value as a separate - argument into __init__. e.g.:: - - t = (1, 2) - TypedTuple(t) == TypedTuple(1, 2) - - This behavior is meant for moz.build files, so vanilla tuples are - automatically cast to typed tuple instances. - - Second, fields in the tuple are validated to be instances of the specified - type. This is done via an ``isinstance()`` check. To allow multiple types, - pass a tuple as the allowed types field. - """ - cls = collections.namedtuple(name, (name for name, typ in fields)) - - class TypedTuple(cls): - __slots__ = () - - def __new__(klass, *args, **kwargs): - if len(args) == 1 and not kwargs and isinstance(args[0], tuple): - args = args[0] - - return super(TypedTuple, klass).__new__(klass, *args, **kwargs) - - def __init__(self, *args, **kwargs): - for i, (fname, ftype) in enumerate(self._fields): - value = self[i] - - if not isinstance(value, ftype): - raise TypeError('field in tuple not of proper type: %s; ' - 'got %s, expected %s' % (fname, - type(value), ftype)) - - super(TypedTuple, self).__init__(*args, **kwargs) - - TypedTuple._fields = fields - - return TypedTuple - - class TypedListMixin(object): '''Mixin for a list with type coercion. See TypedList.''' def _ensure_type(self, l): if isinstance(l, self.__class__): return l def normalize(e):