777723b6b06c26e7bbb11ecdacdc8fbe1ba42497: Bug 1417684 - Reimplement make detection in moz.configure; r?build draft
Gregory Szorc <gps@mozilla.com> - Tue, 14 Nov 2017 16:06:35 -0800 - rev 707810
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Reimplement make detection in moz.configure; r?build Previously, moz.configure assembled a list of possible make binary names and did a check_prog, taking the first one that resolved. This logic was inadequate and littered with bugs. Until very recently, moz.configure was always invoked with client.mk. And client.mk exposed MAKE to the configure invocation. And this MAKE was derived by Python code in mozbuild that knows how to find a suitable make. So even though moz.configure had logic for resolving MAKE, chances are that mozbuild was really doing all the work. A bug in the old implementation was that the MAKE environment variable was supplemented by other candidate binaries. Generally speaking, if a variable is passed into configure, it should be used explicitly. But the previous code didn't do that. The new code only looks at MAKE if it is defined. The old code was also failing to look for mozmake on Windows. We actually require mozmake on Windows. Basically, the previous code was relying on the fact that MAKE was defined and pointing to mozmake for it to work. If we were to undefine MAKE and invoke the old code, builds on Windows would break. The old code was also not validating the make binary. The Python code in base.py for locating a make binary (and client.mk until recently) invokes baseconfig.mk to validate the make binary. So, it was possible for the old moz.configure code to use a make binary that can't actually be used with the build system. The new moz.configure code is much more robust. If MAKE is defined, it only looks at that binary. Otherwise, it falls back to searching for various make binary names. The search order is similar to base.py. The lone exception is mozmake is last instead of between "make" and "gnumake." Since base.py's logic has been invoking client.mk and that make becomes MAKE and is used by moz.configure, I'm not anticipating any significant change in behavior. We also validate the make binary in moz.configure. The logic is very similar to that in base.py. Basically, we invoke baseconfig.mk (which does most validation) and special case the Xcode license check failure. Eventually, we can likely move more logic from baseconfig.mk into moz.configure. And once we no longer run client.mk to invoke the build system, we can remove the "find make" code from base.py and drive everything from moz.configure. But these are for other commits. To prove the new code works, we stop passing the mozbuild derived MAKE into configure. moz.configure (or the mozconfig) is now in the driver's seat for finding a make binary. We still may not actually use the make found by moz.configure for invoking client.mk or the build backend. But this was always a problem (at least for `mach build`). We'll need to fix this in subsequent commits. MozReview-Commit-ID: 1qEdWSkdmGf
e82320b9b80aab7f215932e22af38fd8f4c0016c: Bug 1417684 - Stop looking for mingw32-make; r?build draft
Gregory Szorc <gps@mozilla.com> - Tue, 14 Nov 2017 16:00:47 -0800 - rev 707809
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Stop looking for mingw32-make; r?build Support for mingw32-make was added in bug 1100925 in order to support msys2. That project is effectively dead. I'm about to refactor this code and don't want to deal with extra requirements. So delete references to mingw32-make. MozReview-Commit-ID: FMl4OzpLHwv
3f9bc9023053067601fe13dd4cbb78f7cb45a73b: Bug 1417684 - Port client.mk code to run configure to Python draft
Gregory Szorc <gps@mozilla.com> - Fri, 17 Nov 2017 20:35:18 -0800 - rev 707808
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Port client.mk code to run configure to Python This is a large patch and there's a lot going on. I'm sorry for anyone who has to read it. The build system is effectively a pipeline of stages that complete one after the other. Here are the stages relevant to this commit: `mach` -> env config -> configure -> config.status -> build backend Before this commit, everything after "env config" (finding topsrcdir, topobjdir, and extracting specific data from an active mozconfig) was implemented in client.mk. (And until recently pretty much everything was implemented in client.mk and `mach` effectively ran `make -f client.mk`.) The goal of this commit is to move the configure pipeline stage from client.mk to Python. The structure of configure is kinda wonky. A lot of that is for historical reasons. The "configure" and "js/src/configure" files derive from "configure.in" files. In the old days, we used autoconf for the derivation. These days, client.mk just copies the file. (But configure.in is also a valid m4 file, so autoconf can still be used!) The "configure" files today are very thin shell scripts that invoke the new Python-based configure: "configure.py." When configure/configure.py runs, its main output is a "config.status" file in the objdir. This file - an executable Python script - contains all the data obtained by running configure. When "config.status" runs ("configure" will run it automatically after writing it), it reads moz.build files and generates files to be consumed by a build backend to build the configured application. Since we always produce a make backend today, the most visible output from config.status is a "Makefile" in the objdir. This commit ports the logic for invoking configure from client.mk to Python. When reviewing the code, it may help to look at removals from client.mk from bottom to top. Here's a summary of what was going on. client.mk has two primary targets: "build" and "configure." The recipe for "configure::" is pretty basic: run $(topsrcdir)/configure from the objdir and touch $(objdir)/Makefile. Of course, there are a lot of dependencies required to get there. More on those later. The relationship between the "build" and "configure" targets is a bit hard to read (original lines 142-155 in client.mk). But the gist of it is: * If Makefile exists, Makefile depends on config.status and config.status depends on $(CONFIG_STATUS_DEPS). * If Makefile doesn't exist, Makefile depends on $(CONFIG_STATUS_DEPS). * Makefile target is produced by running "make -f client.mk configure" * "build" depends on Makefile and config.status. Essentially, this is expressing that Makefile is built by config.status and these both depend on $(CONFIG_STATUS_DEPS). When "build" is evaluated, both Makefile and config.status must exist and be up to date. If not, we evaluate the "configure" target of client.mk in a separate invocation of client.mk (I suspect it isn't an "internal" target dependency for legacy reasons and to keep things simple since it avoids issues with phony targets). The "configure" target itself has a couple of dependencies: * The configure files themselves (which are copied from configure.in) * .mozconfig.json (which is derived from the active mozconfig) The "config.status" target is a bit more interesting. It lists a lot of random files throughout the repo. Basically things that when changed have a significant impact on the build configuration. In addition, the included configure.d file also extended the dependencies for config.status. Logically, the "configure" and "config.status" dependencies should probably be the same. I think the reason they are separate is historical reasons. If you look at $(EXTRA_CONFIG_DEPS), those are files related to m4. Those made sense as dependencies when configure was produced by running autoconf. They should have been removed when we started producing configure via `cp`. But they weren't. And, it appears moz.configure isn't tracking these m4 files in its auto-generated configure.d file. So we need to carry those files with us in the Python port (I intend to fix this in another commit). The new Python code is hopefully easier to read. Because invoking configure represents a distinct stage in the build pipeline and because building.py was already quite large, I put the code in a new module. Because I don't like the class-centric design of the MozbuildObject base class (I wish I had a time machine to uninvent that "god" object), I've stepped out of my way to avoid using it explicitly in the new code. The new code in configure.py is relatively straightforward. We have a single new high-level API: ensure_configure(). By default, it runs configure it it needs to. You can also pass `force=True` to force configure to run. There is definitely some wonkiness in ensure_configure() and _run_configure(), such as having to set the MAKE environment variable (we'll fix this later). Otherwise, it is pretty straightforward with regards to capturing file dependencies and invoking configure. There are a few notable changes in behavior. First, we now pull in all loaded .py modules as dependencies for configure. This will include python.mozbuild.*, which is desirable. Less desirable is the inclusion of other modules, like random mach_commands.py files. I think we'll eventually want to move the "run configure if necessary" code to a separate process to mitigate this. We may even want to move the logic to configure itself and just always invoke configure, having it no-op as appropriate. Second, since we no longer run the configure process from the context of client.mk, variables set in client.mk - including from the included mozconfig-derived make files - are no longer set in the configure environment. I think this is fine. configure has logic for locating the mozconfig that matches what mach is doing. So I'm pretty sure the end result is the same. The only variable we do need to set though is MAKE. That's because moz.configure's make detection logic is poor. This will be improved in subsequent commits and the MAKE hack will go away. The new code documents a handful of known areas for improvement. I don't think any of these are show stoppers. Perfect is the enemy of good. Subsequent commits will improve matters significantly... MozReview-Commit-ID: ZCpTdLFur3
33f306eaf972544032d19f7fdd0be41dc40300c9: Bug 1417684 - Move configure file generation to Python; r?build draft
Gregory Szorc <gps@mozilla.com> - Mon, 20 Nov 2017 12:17:12 -0800 - rev 707807
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Move configure file generation to Python; r?build Thus begins the process of moving the logic for invoking configure from client.mk to Python. Because building.py is getting quite large and the logic for configure is somewhat standalone, we've established a new module to contain logic related to configure. The new module has a single public API: ensure_configure(). This will (eventually) be the single function that is called to run configure. The new code in configure.py does not attempt to use the MozbuildObject "god" object. I'm not a fan of this type and pattern. Its existence stems from a time when I was putting everything in Python inside classes. I have since gravitated away from such heavy class usage and don't want to cargo cult this pattern further. The added code in building.py definitely has some redundancy. While we could consolidate the code, the end state of this series will result in direct calls to ensure_configure() everywhere that needs to occur. Rather than go through excessive refactoring later, we incur the minimal redundancy today. The only logic ported to Python so far is the generation of the configure files. However, the API of ensure_configure() and the checking of its return value foreshadow functionality to come. It is quite possible that this commit in isolation subtly breaks dependencies in client.mk. I'm not inclined to exert too much effort to validate this because the client.mk mechanism for invoking configure is not long for this world.
e200a2b9a68e1f8d3d98f1c490d801b72d1b6b86: Bug 1417684 - Prepare the object directory earlier; r?build draft
Gregory Szorc <gps@mozilla.com> - Thu, 16 Nov 2017 18:49:15 -0800 - rev 707806
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Prepare the object directory earlier; r?build The code in _run_client_mk() was extracted from client.mk. So far, the bulk of that code deals with initializing the objdir. Future patches uncovered some wonkiness with the ordering of the execution of this code. In this commit, we split out the code for preparing the object directory into its own function. We then call this function very early in the build process. This ensures that files like .mozconfig.json are present and accessible more often. FWIW, I heard glandium say he would like configure to create the objdir. This would presumably also entail configure managing files like .mozconfig.json. I'm not opposed to this idea. However, changing that right now is difficult since things are changing so fast. Let's get functionality out of client.mk then clean things up later. MozReview-Commit-ID: JyvhwJK51Vp
636e11616302c4e39e70eabf3d2dac07bb9be1e8: Bug 1417684 - Always evaluate config environment; r?build draft
Gregory Szorc <gps@mozilla.com> - Fri, 27 Oct 2017 17:14:00 -0700 - rev 707805
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Always evaluate config environment; r?build I /think/ the only scenario where we wouldn't be able to evaluate a config environment after configure was the case where MOZ_BUILD_PROJECTS was in play. Since we removed support for that feature, let's drop support for treating the config environment as optional. MozReview-Commit-ID: 4sz9dOwaA3y
7fd4a9c17e3cad7e7ac95f15f1b6cabe6be168d8: Bug 1417684 - Stop writing FOUND_MOZCONFIG variable; r?build draft
Gregory Szorc <gps@mozilla.com> - Thu, 16 Nov 2017 17:46:56 -0800 - rev 707804
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Stop writing FOUND_MOZCONFIG variable; r?build The last consumer of FOUND_MOZCONFIG was removed in 41492b2ba4e1 (bug 1417264). client.mk only takes the content of mozconfig.json into account (which is derived from the content of the mozconfig file) for dependency checking. So it should be safe to stop writing this to .mozconfig.mk. MozReview-Commit-ID: 2bBUvaaiWsz
9ab154d77574e29b840aa35bfd90845789a3dbc2: Bug 1417684 - Functionality for evaluating file-based mtime dependencies; r?build draft
Gregory Szorc <gps@mozilla.com> - Wed, 15 Nov 2017 17:53:28 -0800 - rev 707803
Push 92225 by bmo:gps@mozilla.com at Tue, 05 Dec 2017 21:43:29 +0000
Bug 1417684 - Functionality for evaluating file-based mtime dependencies; r?build We want to get rid of client.mk and drive the build system from Python instead of make. This requires reinventing aspects of a build system to determine if targets are up to date. And since our initial intent is to port logic from client.mk to Python, we must implement aspects of make's logic to Python. Make works by looking at target/file presence and mtime to determine if it is up to date. In this commit, we implement some utility Python code for evaluating a file-based "is up to date" check using mtimes. I figured it was worth making this code generic instead of inlining a bunch of I/O operations in to-be-written Python code. If nothing else, it makes the final Python code easier to read. And, who knows, perhaps we'll reuse this dependency checking code in other operations in the future. MozReview-Commit-ID: 3rmUQJMehMn
03162c5fb4425219491a15191daf5a1663670f99: Bug 1423282 - Remove last remenants of frame.Manager draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 21:28:59 +0000 - rev 707802
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Remove last remenants of frame.Manager This removes the last remenants of frame.Manager and testing/marionette/frame.js from Marionette. The preceding commits in this changeset has gradually removed the unused and duplicated features that it implemented. The only remaining pieces are the registrating of some chrome-side IPC message handlers which we can leave attached for the duration of the Marionette session. MozReview-Commit-ID: EYjrJBeTybz
0372c53ea6cbdc39734c0865f3a4d1933486d946: Bug 1423282 - Remove unused IPC listener Marionette:getImportedScripts draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 21:21:26 +0000 - rev 707801
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Remove unused IPC listener Marionette:getImportedScripts MozReview-Commit-ID: EwRaq9ljYxo
0e885b46ed86bbe78dd2be354a92b119a54f84d9: Bug 1423282 - Drop Marionette:emitTouchEvent IPC message and related infra draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 21:08:48 +0000 - rev 707800
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop Marionette:emitTouchEvent IPC message and related infra This removes the Marionette:emitTouchEvent IPC message which is currently not in use by any tests. Along with removing this message listener we can get rid of a tonne of complicated infrastructure in testing/marionette/frame.js. On switching the content frame we no longer await frame scripts to register themselves because they implicitly inherit the parent's frame script in Firefox/Fennec. This was a relic from the B2G days when each frame was OOP. MozReview-Commit-ID: 5vxrWHjzd68
4b8ad1eb9d20a9a30cdb66f9298b035da3bc15d5: Bug 1423282 - Remove legacy action chain browser close guard draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 21:05:37 +0000 - rev 707799
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Remove legacy action chain browser close guard It turns out that we no longer need to guard against the browser/frame closing when using the legacy actions module. This means we can get rid of GeckoDriver#addFrameClsoeListener, which again populates mozBrowserClose and adds handlers for the related mozbrowserclose event. The mozbrowsercloseevent was set for every case of Marionette:ok, Marionette:done, and Marionette:error IPC messages. These events are still in use in testing/marionette/proxy.js, but with this patch we stop listening for these events in testing/marionette/driver.js. MozReview-Commit-ID: jp34kh7nqD
2cb142fda70950311088a9e89f56b1070560bde7: Bug 1423282 - Remove aliveCheck to frame message manager draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 20:50:14 +0000 - rev 707798
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Remove aliveCheck to frame message manager The IPC message "aliveCheck" will always fail because there is no such message handler in Marionette and because it swallows all thrown errors. MozReview-Commit-ID: JISuK65ZcGM
fe80a863f6687a93948d0648fecd643b7fac8b83: Bug 1423282 - Drop MarionetteFrame:getCurrentFrameId IPC message draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 19:40:29 +0000 - rev 707797
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop MarionetteFrame:getCurrentFrameId IPC message The MarionetteFrame:getCurrentFrameId IPC message was used for B2G applications that needed access to chrome-scoped APIs for emitting touch events. Now that actions happen either in chrome _or_ in content this is no longer necessary functionality to maintain. MozReview-Commit-ID: Bk9LRAOxjAw
9add30fdc5a5f14ba9daf92688f28f8a51574139: Bug 1423282 - Drop MarionetteFrame:getInterruptedState IPC message draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 19:22:53 +0000 - rev 707796
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop MarionetteFrame:getInterruptedState IPC message For similar reasons to MarionetteFrame:handleModal we can remove getInterruptedState. MozReview-Commit-ID: 72g0GlYy21T
804eec3f4ce1b3e290a2a916950924bcdddd890b: Bug 1423282 - Drop MarionetteFrame:handleModal IPC message draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 18:52:07 +0000 - rev 707795
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop MarionetteFrame:handleModal IPC message The MarionetteFrame:handleModal IPC message is not used in listener.js, and it is no longer a requirement that this is done through a call from the content frame script. MozReview-Commit-ID: Bn40b1VT7Da
8349f77c584dca5eccf6a47f1a889703e6767fce: Bug 1423282 - Drop Marionette:shareData IPC message draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 18:48:37 +0000 - rev 707794
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop Marionette:shareData IPC message The Marionette:shareData IPC message was used by the simpletest harness to share test logs with the main process. This is no longer a requirement. MozReview-Commit-ID: 4nn7FefCdJ8
f29d23a0818fd73bd4647868c9274562c25c6a54: Bug 1423282 - Drop Marionette:log IPC message draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 18:46:49 +0000 - rev 707793
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop Marionette:log IPC message We used to transport log messages to the main process for logging. This is no longer required and the IPC message has not been in use for some time. MozReview-Commit-ID: F5thqDOJADd
434b4aeba13d5d39b055b182403dcf6c122074c4: Bug 1423282 - Drop Marionette:switchToModalOrigin IPC message draft
Andreas Tolfsen <ato@sny.no> - Tue, 05 Dec 2017 18:44:24 +0000 - rev 707792
Push 92224 by bmo:ato@sny.no at Tue, 05 Dec 2017 21:39:16 +0000
Bug 1423282 - Drop Marionette:switchToModalOrigin IPC message In B2G, when a frame was interrupted by a modal dialogue, the Marionette:switchToModalOrigin IPC message allowed you to switch back to the frame that was interrupted. It got called by the interrupted frame once the dialogue got dismissed and the frame resumed its process. This functionality is no longer requried. MozReview-Commit-ID: DtCOzeW45qP
77d2fce695bbc79889b659210815689eeff0e78d: Bug 1415536 - Extend NotifyNetworkActivity to get sent/received bytes r?valentin r?baku draft
Tarek Ziadé <tarek@mozilla.com> - Thu, 23 Nov 2017 09:37:54 +0100 - rev 707791
Push 92223 by tziade@mozilla.com at Tue, 05 Dec 2017 21:35:12 +0000
Bug 1415536 - Extend NotifyNetworkActivity to get sent/received bytes r?valentin r?baku MozReview-Commit-ID: Afdvz0lktY8
(0) -300000 -100000 -30000 -10000 -3000 -1000 -300 -100 -50 -20 +20 +50 +100 +300 +1000 +3000 +10000 +30000 +100000 tip