Bug 1661434 - added debugging and taskcluster pages from MDN to in-tree. r=sylvestre
authorEmil Farisan <emil.farisan@gmail.com>
Mon, 31 Aug 2020 11:16:52 +0000
changeset 546984 f2b8b67af40168e925b79255c92f7d102bb0d1ab
parent 546983 e6ef511d20c9127463138b3ae7aaea233cd50756
child 546985 6e69b5d68c1e247ee9b83ecea582d1a47cc837d9
push id37743
push usernbeleuzu@mozilla.com
push dateMon, 31 Aug 2020 21:52:15 +0000
treeherdermozilla-central@93d8458a0c4d [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerssylvestre
bugs1661434
milestone82.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1661434 - added debugging and taskcluster pages from MDN to in-tree. r=sylvestre Differential Revision: https://phabricator.services.mozilla.com/D88580
docs/contributing/debugging/debugging_a_hang_on_os_x.rst
docs/contributing/debugging/debugging_on_mac_os_x.rst
docs/contributing/debugging/debugging_on_windows.rst
taskcluster/docs/index.rst
taskcluster/docs/release-promotion.rst
taskcluster/docs/setting-up-an-update-server.rst
taskcluster/docs/task-graph.rst
new file mode 100644
--- /dev/null
+++ b/docs/contributing/debugging/debugging_a_hang_on_os_x.rst
@@ -0,0 +1,63 @@
+Debugging A Hang On OS X
+========================
+
+This article contains historical information about older versions of OS X.
+See `How to Report a Hung
+Firefox <https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox>`__ instead.
+
+If you find a hang in an application, it is very useful for the
+developer to know where in the code this hang happens, especially if he
+or she can't reproduce it. Below are steps you can use to attach
+so-called "Samples" to bug reports.
+
+Creating the sample on Mac OS X 10.5 (XCode < 4.2)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. When the application is still hung, open up Spin Control.app (it’s in
+   your <tt>/Developer/Applications/Performance Tools/</tt> folder; if
+   it is missing, install the latest `Computer Hardware Understanding
+   Development (CHUD)
+   Tools <http://developer.apple.com/tools/download/>`__ from Apple).
+#. By default, sampling of any hanging application will begin
+   automatically.
+#. After about 3-4 seconds, select the hanging application in the
+   “Detected Hangs” window and click the “Interrupt Sampling” button.
+   Note that sampling **will quickly hog up a lot of memory** if you
+   leave it on for too long!
+#. When it's done parsing the data, click the "Show text report" button;
+   a new window will open with a couple of rows with stacktraces for all
+   the threads in the sampled application.
+#. If you need to upload the sample to Bugzilla, select all the sample
+   text, copy it into your favorite text editor, and save as a
+   plain-text file.
+#. For a more detailed view of the sample data, make sure the sample you
+   just recorded is selected in the “Detected Hangs” window and click on
+   the “Open…” button.
+
+Creating the sample on Mac OS X 10.4
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. When the application is still hung, open up Sampler.app (it’s in your
+   <tt>/Developer/Applications/Performance Tools/</tt> folder; if it is
+   missing, install the latest `Computer Hardware Understanding
+   Development (CHUD)
+   Tools <http://developer.apple.com/tools/download/>`__ from Apple.).
+#. Choose File > Attach...
+#. Select "firefox-bin" or the hung application you want a sample from.
+#. Now, click "Start recording". Note that only 3-4 seconds usually
+   suffice, note that **this will quickly hog up a lot of memory** if
+   you leave it on for too long!
+#. When it's done parsing the data, you should now have a couple of rows
+   with stacktraces for all the threads in the sampled application.
+#. When you have the raw Sample data, you can't just save that and
+   attach it to a bug, because the format is not very usable (unless the
+   developer is a Mac hacker). Export it using Tools > Generate Report,
+   and attach this as a text file to the bug report.
+
+The final data will show where the application spent all its time when
+it was hung, and hopefully your bug will be easier to fix now!
+
+See also
+~~~~~~~~
+
+`Debugging on Mac OS X <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_Mac_OS_X>`__
new file mode 100644
--- /dev/null
+++ b/docs/contributing/debugging/debugging_on_mac_os_x.rst
@@ -0,0 +1,351 @@
+Debugging On Mac Os X
+=====================
+
+This document explains how to debug Mozilla-derived applications such as
+Firefox, Thunderbird, and SeaMonkey on macOS using Xcode. If you want to
+debug from the terminal see :ref:`Debugging Mozilla with
+lldb <Debugging Firefox with LLDB>`. For specific
+information on a way to debug hangs, see `Debugging a hang on OS
+X <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_hang_on_OS_X>`__.
+
+Creating a debuggable build
+---------------------------
+
+First, you need to build the application you're going to debug using
+``--disable-optimize``\ ``--enable-debug-symbols`` in your .mozconfig
+(also add ``--enable-debug`` if you want assertions etc. compiled in).
+See :ref:`Building Firefox for macOS <Building Firefox On MacOS>`
+if you need help creating your own build.
+
+Debugging Firefox on macOS 10.14+
+---------------------------------
+
+macOS 10.14 introduced Notarization and Hardened Runtime features for
+improved application security. macOS 10.15 went further, requiring
+applications to be Notarized with Hardened Runtime enabled in order to
+launch (ignoring workarounds.) When run on earlier macOS versions,
+Notarization and Hardened Runtime settings have no effect.
+
+Official Builds
+~~~~~~~~~~~~~~~
+
+At this time, official builds of Firefox 69 and later are Notarized.
+Uplifts to Firefox 68 and ESR 68 to enable Notarization are planned.
+**As a result, it is not possible to attach a debugger to these official
+Firefox releases on macOS 10.14+ without disabling System Integrity
+Protection (SIP).** This is due to Notarization requiring Hardened
+Runtime to be enabled with the ``com.apple.security.get-task-allow``
+entitlement disallowed. **Rather than disabling SIP (which has security
+implications), it is recommended to debug with try builds or local
+builds. The differences are explained below.**
+
+try Server Builds
+~~~~~~~~~~~~~~~~~
+
+In most cases, developers needing to debug a build as close as possible
+to the production environment should use a `try
+build <https://wiki.mozilla.org/ReleaseEngineering/TryServer>`__. These
+builds enable Hardened Runtime and only differ from production builds in
+that they are not Notarized which should not otherwise affect
+functionality, (other than the ability to easily launch the browser on
+macOS 10.15+ -- see quarantine note below). At this time, developers can
+obtain a Hardened Runtime build with the
+``com.apple.security.get-task-allow`` entitlement allowed by submitting
+a try build and downloading the dmg generated by the "Rpk" shippable
+build job. A debugger can be attached to Firefox processes of these
+builds. try builds use the ``developer.entitlements.xml`` file from the
+source tree while production builds use ``production.entitlements.xml``.
+**On macOS 10.15+, downloaded try builds will not launch by default
+because Notarization is required. To workaround this problem, remove the
+quarantine extended attribute from the downloaded Nightly:**
+
+  ``$ xattr -r -d com.apple.quarantine /Path/to/Nightly.app``
+
+Local Builds
+~~~~~~~~~~~~
+
+Local builds of mozilla-central do not enable Hardened Runtime and hence
+do not have debugging restrictions. As a result, some functionality will
+be permitted on local builds, but blocked on production builds which
+have Hardened Runtime enabled. `Bug
+1522409 <https://bugzilla.mozilla.org/show_bug.cgi?id=1522409>`__ was
+filed to automate codesigning local builds to enable Hardened Runtime by
+default and eliminate this discrepancy.
+
+To obtain a Hardened Runtime build without using try infrastructure, a
+developer can manually codesign builds using the macOS ``codesign(1)``
+command with the ``developer.entitlements.xml`` file from the tree. This
+requires creating a codesigning identity.
+
+Disabling System Integrity Protection (SIP)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If debugging a production build is required, follow Apple's documented
+steps for disabling System Integrity Protection (SIP). Note that
+disabling SIP bypasses Hardened Runtime restrictions which can mask some
+bugs that only occur with Hardened Runtime so it is recommended to test
+fixes with SIP enabled. **Disabling SIP has system security implications
+that should be understood before taking this step.**
+
+Creating an Xcode project
+-------------------------
+
+If you try to create a new Xcode project in an existing directory
+then Xcode will delete its existing contents (Xcode will warn you
+beforehand). To work around that, the steps below have you initialize
+the project outside the Mozilla source tree, close the project, copy
+the .xcodeproj project "file" into the source tree, and then reopen
+the project to finish setting it up.
+
+Note also that since Xcode 7.3.1 it doesn't seem to be possible to
+have the Xcode project live outside the source tree. If you try to do
+that then Xcode will simply **copy** the source files under the
+project directory rather than link to them (still the case in Xcode
+10?) which breaks debugging and the possibility to
+modify-rebuild-relaunch from inside Xcode.
+
+These steps were last updated for Xcode 10.3:
+
+#. Open Xcode, and create a new Project with File > New Project. Select
+   the "Cross-platform" tab then under the "Other" template group select
+   the "Empty" project type. the click Next. Name the project and click
+   Next. Create/select a temporary directory to contain the project and
+   then click Create.
+#. Before going any further, close the project (File > Close Project)
+   and open Finder. Find the \*.xcodejproj directory in the temporary
+   directory, move it into your Mozilla source tree, and then
+   double-click on it to reopen it.
+#. In the left-hand pane in Xcode you should see a tree item where the
+   root item has the project name. If the temporary directory that you
+   originally created the Xcode project in is under that, right click it
+   and delete it. Now, right click on the root item, select 'Add files
+   to "<project-name>"', select all the files and directories in your
+   source directory, untick "Copy items if needed", then click Add.
+   (These will then be progressively added under the root item
+   <project-name> in the left-hand pane. Note that subdirectories may
+   initially appear to be empty, but they too will progressively be
+   populated as Xcode processes the sourse files. Once done, you should
+   be able to open any file quickly by hitting Cmd-Shift-O and typing in
+   the name of a file.)
+#. In the Product menu, select Scheme > New Scheme and name your scheme
+   (for example, "Debug"). After you click OK, Xcode should open the
+   settings window for the new scheme. (If not, then open its settings
+   from the Product > Edit Scheme menu.)
+#. Select "Run" on the left-hand side of the settings window, then
+   select the "Info" tab. Set the Executable by clicking on "None" and
+   selecting "Other...". A new dialog titled "Choose an executable to
+   launch" will pop up. Browse to the ``.app`` file that you want to
+   debug (``Firefox.app``, ``Nightly``\ ``Debug.app`` etc). The ``.app``
+   file is typically found inside the ``dist`` folder in your build
+   directory.
+#. If you are debugging Firefox, Thunderbird, or some other application
+   that supports multiple profiles, using a separate profile for
+   debugging purposes is recommended. See "Having a profile for
+   debugging purposes" below. Select the "Arguments" tab in the scheme
+   editor, and click the '+' below the "Arguments passed on launch"
+   field. Add "-P *profilename*", where *profilename* is the name of a
+   profile you created previously. Repeat that to also add the argument
+   "-no-remote".
+#. Also in the "Arguments" panel, you may want to add an environment
+   variable MOZ_DEBUG_CHILD_PROCESS set to the value 1 to help with
+   debugging e10s.
+#. Select "Build" from the left of the scheme editor window, and check
+   that there is nothing listed under Targets (otherwise it may cause
+   problems when you try to run the executable for debugging since you
+   will get build errors).
+#. Click "Close" to close the scheme editor.
+
+At this point you can run the application from Xcode, and when you pause
+or hit breakpoints it should show open the correct source file at the
+correct line.
+
+Setting up lldb
+---------------
+
+``lldb`` is the debugger XCode provides/uses.
+
+.. warning::
+
+   One important issue that the Mozilla .lldbinit file fixes is that by
+   default some breakpoints will be listed as "pending", and XCode will
+   not stop at them. If you don't include the Mozilla's .lldbinit, you
+   must at least put
+   ``settings set target.inline-breakpoint-strategy always`` in your
+   ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with
+   lldb <Debugging Firefox with LLDB>`.
+
+The
+`.lldbinit <http://dxr.mozilla.org/mozilla-central/source/.lldbinit>`__
+file in the source tree imports many useful `Mozilla specific lldb
+settings, commands and
+formatters <https://dxr.mozilla.org/mozilla-central/source/python/lldbutils/README.txt>`__
+into ``lldb``, but you may need to take one of the following steps to
+make sure this file is used.
+
+If you are using ``lldb`` on the command line (independently of Xcode)
+and you will always run it from either the top source directory, the
+object directory or else the dist/bin subdirectory of the object
+directory, then adding the following setting to your ``$HOME/.lldbinit``
+is sufficient:
+
+::
+
+   settings set target.load-cwd-lldbinit true
+
+*However*, if you will run lldb from a different directory, or if you
+will be running it indirectly by debugging in Xcode (Xcode always runs
+lldb from "/"), then this setting will not help you. Instead, add the
+following to your ``$HOME/.lldbinit``:
+
+::
+
+   # This automatically sources the Mozilla project's .lldbinit as soon as lldb
+   # starts or attaches to a Mozilla app (that's in an object directory).
+   #
+   # This is mainly a workaround for Xcode not providing a way to specify that
+   # lldb should be run from a given directory.  (Xcode always runs lldb from "/",
+   # regardless of what directory Xcode was started from, and regardless of the
+   # value of the "Custom working directory" field in the Scheme's Run options.
+   # Therefore setting `settings set target.load-cwd-lldbinit true` can't help us
+   # without Xcode providing that functionality.)
+   #
+   # The following works by setting a one-shot breakpoint to break on a function
+   # that we know will both run early (which we want when we start first start the
+   # app) and run frequently (which we want so that it will trigger ASAP if we
+   # attach to an already running app).  The breakpoint runs some commands to
+   # figure out the object directory path from the attached target and then
+   # sources the .lldbinit from there.
+   #
+   # NOTE: This scripts actions take a few seconds to complete, so the custom
+   # formatters, commands etc. that are added may not be immediately available.
+   #
+   breakpoint set --name nsThread::ProcessNextEvent --thread-index 1 --auto-continue true --one-shot true
+   breakpoint command add -s python
+       # This script that we run does not work if we try to use the global 'lldb'
+       # object, since it is out of date at the time that the script runs (for
+       # example, `lldb.target.executable.fullpath` is empty).  Therefore we must
+       # get the following objects from the 'frame' object.
+       target = frame.GetThread().GetProcess().GetTarget()
+       debugger = target.GetDebugger()
+
+       # Delete our breakpoint (not actually necessary with `--one-shot true`):
+       target.BreakpointDelete(bp_loc.GetBreakpoint().GetID())
+
+       # For completeness, find and delete the dummy breakpoint (the breakpoint
+       # lldb creates when it can't initially find the method to set the
+       # breakpoint on):
+       # BUG WORKAROUND! GetID() on the *dummy* breakpoint appears to be returning
+       # the breakpoint index instead of its ID.  We have to add 1 to correct for
+       # that! :-(
+       dummy_bp_list = lldb.SBBreakpointList(target)
+       debugger.GetDummyTarget().FindBreakpointsByName("nsThread::ProcessNextEvent", dummy_bp_list)
+       dummy_bp_id = dummy_bp_list.GetBreakpointAtIndex(0).GetID() + 1
+       debugger.GetDummyTarget().BreakpointDelete(dummy_bp_id)
+
+       # "source" the Mozilla project .lldbinit:
+       os.chdir(target.executable.fullpath.split("/dist/")[0])
+       debugger.HandleCommand("command source -s true " + os.path.join(os.getcwd(), ".lldbinit"))
+   DONE
+
+see :ref:`Debugging Mozilla with
+lldb <Debugging Firefox with LLDB>`. for more information.
+
+Having a profile for debugging purposes
+---------------------------------------
+
+It is recommended to create a separate profile to debug with, whatever
+your task, so that you don't lose precious data like Bookmarks, saved
+passwords, etc. So that you're not bothered with the profile manager
+every time you start to debug, expand the "Executables" branch of the
+"Groups & Files" list and double click on the Executable you added for
+Mozilla. Click the plus icon under the "Arguments" list and type "-P
+<profile name>" (e.g. "-P MozillaDebug"). Close the window when you're
+done.
+
+Running a debug session
+-----------------------
+
+Make sure breakpoints are active (which implies running under the
+debugger) by opening the Product menu and selecting "Debug / Activate
+Breakpoints" (also shown by the "Breakpoints" button in the top right
+section of the main window). Then click the "Run" button or select "Run"
+from the Product menu.
+
+Setting breakpoints
+~~~~~~~~~~~~~~~~~~~
+
+Setting a breakpoint is easy. Just open the source file you want to
+debug in Xcode, and click in the margin to the left of the line of code
+where you want to break.
+
+During the debugging session, each time that line is executed, the
+debugger will break there, and you will be able to debug it.
+
+.. warning::
+
+   Note that with the default configuration, some breakpoints will be
+   listed as "pending", and XCode will not stop at them. If you don't
+   include the Mozilla's .lldbinit, you must at least put
+   ``settings set target.inline-breakpoint-strategy always`` in your
+   ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with
+   lldb <Debugging Firefox with LLDB>`.
+
+Using Mozilla-specific lldb commands
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you included the .lldbinit when `Setting up
+lldb <#setting-up-lldb>`__, you can use Mozilla-specific lldb commands
+in the console, located in the Debug area of XCode. For example, type
+``js`` to see the JavaScript stack. For more information, see :ref:`Debugging
+Mozilla with lldb <Debugging Firefox with LLDB>`.
+
+Debugging e10s child processes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Using XCode to debug child processes created by an e10s-enabled browser
+is a little trickier than debugging a single-process browser, but it can
+be done. These directions were written using XCode 6.3.1
+
+#. Complete all the steps above under "Creating the Project"
+#. From the "Product" menu, ensure the scheme you created is selected
+   under "Scheme", then choose "Scheme > Edit Scheme"
+#. In the resulting popup, click "Duplicate Scheme"
+#. Give the resulting scheme a more descriptive name than "Copy of
+   Scheme"
+#. Select "Run" on the left-hand side of the settings window, then
+   select the "Info" tab. Set the Executable by clicking on the
+   "Executable" drop-down, and selecting the ``plugin-container.app``
+   that is inside the app bundle of the copy of Firefox you want to
+   debug.
+#. On the same tab, under "Launch" select "Wait for executable to be
+   launched"
+#. On the "Arguments" tab, remove all arguments passed on launch.
+
+Now you're ready to start debugging:
+
+#. From the "Product" menu, ensure the scheme you created above is
+   selected under "Scheme"
+#. Click the "Run" button. The information area at the top of the window
+   will show "Waiting for plugin-container to launch"
+#. From a command line, run your build of Firefox. When that launches a
+   child process (for example, when you start to load a webpage), XCode
+   will notice and attach to that child process. You can then debug the
+   child process like you would any other process.
+#. When you are done debugging, click the "Stop" button and quit the
+   instance of Firefox that you were debugging in the normal way.
+
+For some help on using lldb see :ref:`Debugging Mozilla with
+lldb <Debugging Firefox with LLDB>`.
+
+Other resources
+---------------
+
+Apple has an extensive list of `debugging tips and
+techniques <https://developer.apple.com/library/mac/#technotes/tn2124/_index.html>`__.
+
+Questions? Problems?
+~~~~~~~~~~~~~~~~~~~~
+
+Try asking in our IRC channels
+`#developers <irc://irc.mozilla.org/developers>`__ or
+`#macdev <irc://irc.mozilla.org/macdev>`__.
new file mode 100644
--- /dev/null
+++ b/docs/contributing/debugging/debugging_on_windows.rst
@@ -0,0 +1,438 @@
+Debugging On Windows
+====================
+
+This document explains how to debug Mozilla-derived applications such as
+Firefox, Thunderbird, and SeaMonkey on Windows using the Visual C++ IDE.
+
+If VC++ and your Mozilla application hang shortly after you launch the
+application under the debugger, see `Problems Loading Debug
+Symbols <#problems-loading-debug-symbols>`__.
+
+Ways to start the debugger
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First of all, it's necessary to install a Visual Studio extension to be
+able to follow child processes as they are created. Firefox, in general,
+and even in non-e10s mode, does not start the main process directly, it
+starts it via a Launcher Process. This means that Visual Studio will
+only attach to the first process it finds, and will not hit any
+break-point (and even notifies you that it cannot find their location).
+`Microsoft Child Process Debugging Power
+Tool <https://marketplace.visualstudio.com/items?itemName=vsdbgplat.MicrosoftChildProcessDebuggingPowerTool>`__
+allows automatically attaching to child processes, such as Web Content
+process, GPU process, etc. Enable it by going its configuration menu in
+"Debug > Other debugging targets > Child process debugging settings",
+and ticking the box.
+
+If you have followed the steps in :ref:`Building Firefox for
+Windows <Building Firefox On Windows>`
+and have a local debug build, you can **execute ``./mach run --debug``**
+from the same command line. It would open Visual Studio with Firefox's
+run options configured. You can **click "Start" button** to run Firefox
+then, already attached in the debugger.
+
+Alternatively, if you have generated the Visual Studio solution, via
+``./mach build-backend -b VisualStudio``, opening this solution allows
+you to run ``firefox.exe`` directly in the debugger. Making it the
+startup project, by right clicking on it (it appears bold when it's the
+case) can be useful. Breakpoints are kept across runs, this can be a
+good way to debug startup issues.
+
+**Run the program until you hit an assertion.** You will get a dialog
+box asking if you would like to debug. Hit "Cancel". The MSDEV IDE will
+launch and load the file where the assertion happened. This will also
+create a Visual C++ Mozilla project in the directory of the executable
+by default.
+
+**Attach the debugger to an existing Mozilla process**.  In the Visual
+Studio, select Debug > Attach to Process. If you want to debug a content
+process, you can **hover on the tab** of page you want to debug, which
+would show the pid. You can then select the process from dialog opened
+from "Attach to Process". For more information, see `Attach to Running
+Processes with the Visual Studio
+Debugger <http://msdn.microsoft.com/en-us/library/vstudio/3s68z0b3.aspx>`__.
+
+Debugging Release and Nightly Builds
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Refer to the steps to `use the Mozilla symbol
+server <https://developer.mozilla.org/en-US/docs/Mozilla/Using_the_Mozilla_symbol_server>`__ and `source
+server <https://developer.mozilla.org/en-US/docs/Mozilla/Using_the_Mozilla_source_server>`__
+
+Creating a Visual C++ project for Firefox
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Please refer to the documentation on
+:ref:`firefox-source-docs <Visual Studio Projects>`.
+
+Changing/setting the executable to debug
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+VC++ 6.0: To change or set the executable to debug, go to Project >
+Settings..., Debug tab and select General from the drop down list.
+"Executable for debug session:" should show the executable you are
+debugging. If it is empty or incorrect, use the arrow button and select
+Browse... to locate the executable.
+
+Command line parameters and environment variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+VC++ 6.0: To change or set the command line options, go to Project >
+Settings..., Debug tab and select General from the drop down list.
+"Program arguments:" should show the options.
+
+Some common options would be the URL of the file you want the browser to
+open as soon as it starts, starting the Profile Manager, or selecting a
+profile. You can also redirect the console output to a file (by adding
+"``> filename.txt``" for example, without the quotes).
+
+In VC 7 and 8 this option is called Project > Properties > Debugging >
+Command Arguments. VC 8 also allows you to set environment variables
+there.
+
+Setting breakpoints in DLLs which are not yet loaded in memory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+VC++ 6.0: Go to Project > Settings..., Debug tab and select "Additional
+DLLs" from the drop down list. Check "Locate Additional DLLs" option.
+For each DLL, click the "New" button which creates a new entry and then
+hit the "..." buttons which lets you browse to the DLL. You will only be
+able to add one DLL at a time.
+
+VC++ 7.0 automatically finds additional DLLs.
+
+Customizing the debugger's variable value view
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can customize how Visual C++ displays classes in the variable view.
+By default VC++ displays "{...}" and you need to click the small + icon
+to expand the members. You can change this behaviour, and make Visual
+C++ display whatever data member you want in whatever order, formatter
+however you like instead of just "{...}".
+
+You need to locate a file called "AUTOEXP.DAT" in your Visual C++
+installation. By default it will be:
+
+VC++ 6.0:
+
+.. code::
+
+   C:\Program Files\Microsoft Visual Studio\Common\MSDev98\Bin\AUTOEXP.DAT
+
+VC++ 7.0:
+
+.. code::
+
+   C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Packages\Debugger\AUTOEXP.DAT
+
+The file has information about the format in the beginning, and after a
+little practice you should be well on your way. Here are some entries
+that will make your life easier:
+
+::
+
+   ;; Mozilla (1.7beta and later)
+   nsAutoString=<mData,su>
+   nsString=<mData,su>
+   nsCString=<mData,s>
+   nsCAutoString=<mData,s>
+   nsRect=x=<x,d> y=<y,d> width=<width,d>; height=<height,d>
+   nsStaticAtomWrapper=<mStaticAtom->mString,s>
+   nsIAtom=<mString,su>
+   ; the following are not necessary in vc8
+   nsCOMPtr<*>=<mRawPtr,x>
+   nsRefPtr=<mRawPtr,x>
+   nsAutoPtr=<mRawPtr,x>
+
+After you have made the changes and saved the file, you will need to
+restart Visual C++ for the changes to take effect.
+
+For XPCOM Strings (the "external" string API) you can use the following
+values:
+
+::
+
+   ;; Mozilla (1.9)
+   ; Internal Strings
+   nsAString_internal=<mData,su>, length=<mLength,u>
+   nsACString_internal=<mData,s>, length=<mLength,u>
+   ; XPCOM Strings
+   nsAString=<nsStringContainer.v,su>, length=<nsStringContainer.d1,u>
+   nsACString=<nsCStringContainer.v,s>, length=<nsCStringContainer.d1,u>
+   nsStringContainer=<v,su>, length=<d1,u>
+   nsCStringContainer=<v,s>, length=<d1,u>
+
+There is a more extensive version of this file in progress in
+`AutoExpForVC8. <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/AutoExpForVC8>`__
+
+Avoiding stepping into certain functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can avoid stepping into certain functions, such as nsCOMPtr methods,
+using an undocumented feature of VC. See the blog post `How to Not Step
+Into Functions using the Visual C++
+Debugger <http://blogs.msdn.com/andypennell/archive/2004/02/06/69004.aspx>`__
+for details.
+
+Here are some wildcards you can use (tested with VC 8):
+
+.. code::
+
+   nsCOMPtr.*\:\:.*=NoStepInto
+   (nsG|g)etter_*AddRefs.*=NoStepInto
+   NS_ConvertUTF.*
+   ; Might be too broad:
+   (ns|Promise)[^\:]*[sS]tring.*
+   ...add common functions to this list
+
+should probably make a .reg file for easy importing
+
+Obtaining ``stdout`` and other ``FILE`` handles
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Running the following command in the Command Window in Visual Studio
+returns the value of ``stdout``, which can be used with various
+debugging methods (such as ``nsGenericElement::List``) that take a
+``FILE*`` param:
+
+.. code::
+
+   Debug.EvaluateStatement {,,msvcr80d}(&__iob_func()[1])
+
+(Alternatively you can evaluate ``{,,msvcr80d}(&__iob_func()[1])`` in
+the QuickWatch window)
+
+Similarly, you can open a file on the disk using ``fopen``:
+
+.. code::
+
+   >Debug.EvaluateStatement {,,msvcr80d}fopen("c:\\123", "w")
+   0x10311dc0 { ..snip.. }
+   >Debug.EvaluateStatement ((nsGenericElement*)0x03f0e710)->List((FILE*)0x10311dc0, 1)
+   <void>
+   >Debug.EvaluateStatement {,,msvcr80d}fclose((FILE*)0x10311dc0)
+   0x00000000
+
+Note that you may not see the debugging output until you flush or close
+the file handle.
+
+Disabling ASSERTIONS
+~~~~~~~~~~~~~~~~~~~~
+
+There are basically two ways to disable assertions. One requires setting
+an environment variable, while the other affects only the currently
+running program instance in memory.
+
+Environment variable
+^^^^^^^^^^^^^^^^^^^^
+
+There is an environment variable that can disable breaking for
+assertions. This is how you would normally set it:
+
+.. code::
+
+   set XPCOM_DEBUG_BREAK=warn
+
+The environment variable takes also other values besides ``warn``, see
+``XPCOM_DEBUG_BREAK`` for more details.
+
+Note that unlike Unix, the default for Windows is not warn, it's to pop
+up a dialog. To set the environment variable for Visual Studio, use
+Project > Properties > Debugging > Environment and click the little box.
+Then use
+
+.. code::
+
+   XPCOM_DEBUG_BREAK=warn
+
+Changing running code
+^^^^^^^^^^^^^^^^^^^^^
+
+You normally shouldn't need to do this (just quit the application, set
+the environment variable described above, and run it again). And this
+can be **dangerous** (like **trashing your hard disc and corrupting your
+system**). So unless you feel comfortable with this, don't do it. **You
+have been warned!**
+
+It is possible to change the interrupt code in memory (which causes you
+to break into debugger) to be a NOP (no operation).
+
+You do this by running the program in the debugger until you hit an
+assertion. You should see some assembly code. One assemly code
+instruction reads "int 3". Check the memory address for that line. Now
+open memory view. Type/copy/drag the memory address of "int 3" into the
+memory view to get it to update on that part of the memory. Change the
+value of the memory to "90", close the memory view and hit "F5" to
+continue.
+
+| Confused? See the screenshot below:
+| |Screenshot of disabling assertions|
+
+VC++ 7.0?
+
+Automatically handling ASSERTIONS without a debugger attached
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When an assertion happens and there is not a debugger attached, a small
+helper application
+(```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__)
+is run. That application can automatically select a response to the "Do
+you want to debug" dialog instead of prompting if you configure it, for
+more info, see
+```windbgdlg.exe`` </En/Automatically_Handle_Failed_Asserts_in_Debug_Builds>`__.
+
+Debugging optimized builds
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To effectively debug optimized builds, you should enable debugging
+information which effectively leaves the debug symbols in optimized code
+so you can still set breakpoints etc. Because the code is optimized,
+stepping through the code may occasionally provide small surprises when
+the debugger jumps over something.
+
+You need to make sure this configure parameter is set:
+
+.. code::
+
+   --enable-debugger-info-modules=yes
+
+You can also choose to include or exclude specific modules. This is
+particularly useful to avoid linking layout with debugging information.
+
+Console debugging
+~~~~~~~~~~~~~~~~~
+
+When printing to STDOUT from a content process, the console message will
+not appear on Windows. One way to view it is simply to disable e10s
+(``./mach run --disable-e10s``) but in order to debug with e10s enabled
+one can run
+
+::
+
+   ./mach run ... 2>&1 | tee
+
+It may also be necessary to disable the content sandbox
+(``MOZ_DISABLE_CONTENT_SANDBOX=1 ./mach run ...``).
+
+Running two instances of Mozilla simultaneously
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can run two instances of Mozilla (e.g. debug and optimized)
+simultaneously by setting the environment variable ``MOZ_NO_REMOTE``:
+
+.. code::
+
+   set MOZ_NO_REMOTE=1
+
+Or, starting with Firefox 2 and other Gecko 1.8.1-based applications,
+you can use the ``-no-remote`` command-line switch instead (implemented
+in
+`bug 325509 <https://bugzilla.mozilla.org/show_bug.cgi?id=325509>`__).
+
+You can also specify the profile to use with the ``-P profile_name``
+command-line argument.
+
+Debugging JavaScript
+~~~~~~~~~~~~~~~~~~~~
+
+Use `Venkman <https://developer.mozilla.org/en-US/docs/Archive/Mozilla/Venkman>`__, the JavaScript Debugger for Mozilla.
+
+You can use helper functions from
+`nsXPConnect.cpp <https://dxr.mozilla.org/mozilla-central/source/js/src/xpconnect/src/nsXPConnect.cpp#1395>`__
+to inspect and modify the state of JavaScript code from the MSVS
+debugger.
+
+For example, to print current JavaScript stack to stdout, evaluate this
+in QuickWatch window:
+
+.. code::
+
+   {,,xul}DumpJSStack()
+
+Visual C++ will show you something in the quick watch window, but
+not the stack, you have to look in the OS console for the output.
+
+Also this magical command only works when the VC++ stack is in certain
+states. It works when you have js_Interpret() in the newest stackframe
+
+Debugging minidumps
+~~~~~~~~~~~~~~~~~~~
+
+See :ref:`debugging a minidump <Debugging A Minidump>`.
+
+Debugging tinderbox builds
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See `Running Windows Debug Builds <https://developer.mozilla.org/en-US/docs/Archive/Mozilla/Running_Windows_Debug_Builds>`__
+
+Problems Loading Debug Symbols
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If both your application and Visual C++ hang shortly after launching the
+application under the debugger, you may be hitting a known deadlock in
+the way Visual Studio downloads debug symbols for the system libraries;
+see
+https://connect.microsoft.com/VisualStudio/feedback/details/422970/hang-loading-rasapi32-pdb-when-using-symbol-server.
+
+There are two ways to work around this problem:
+
+#. Turn off automatic symbol downloading for system libraries: in Tools
+   > Options > Debugging > Symbols, uncheck the Microsoft symbol server.
+#. Pre-load all the Windows debug symbols. These instructions apply to
+   Visual Studio 10 on Windows 7; other software versions likely need to
+   have file paths adjusted.
+
+   #. Locate the Microsoft utility "SymChk.exe" on your system (it will
+      likely be in the installation directory of your Windows Debugging
+      Tools).
+
+   #. Find the directory where Visual Studio caches downloaded symbols;
+      in VC++ 10 open the menu to Tools > Options > Debugging  > Symbols
+      and copy the field "Cache symbols in this directory".
+
+   #. In a command window, run
+
+      ::
+
+         symchk.exe /r C:\windows\SysWOW64\ /s "SRV*<your cache symbols directory>\MicrosoftPublicSymbols*http://msdl.microsoft.com/download/symbols"
+
+      | 
+      | Note the "``\MicrosoftPublicSymbols``" appended to the cache
+        directory configured in Visual Studio.
+
+Downloading all symbols can take a long time; you can replace
+C:\windows\SysWOW64\\ with the name of a single .DLL to download symbols
+only for the specific libraries you are trying to debug. Unfortunately,
+it's hard to know which symbols to download without having VS hang and
+seeing the "Downloading symbols for <library>" status at the bottom left
+of the main window.
+
+Problems post-mortem debugging on Windows 7 SP1 x64?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you attempt to use ``NS_DebugBreak`` etc to perform post-mortem
+debugging on a 64bit Windows 7, but as soon as you try and continue
+debugging the program crashes with an Access Violation, you may be
+hitting a Windows bug relating to AVX support.  For more details,
+including a work-around see `this blog
+post <http://www.os2museum.com/wp/?p=960>`__ or `this social.msdn
+thread <http://social.msdn.microsoft.com/Forums/vstudio/en-US/392ca62c-e502-42d9-adbc-b4e22d5da0c3/jit-debugging-32bit-app-crashing-with-access-violation>`__. 
+(And just in-case those links die, the work-around is to execute
+
+::
+
+   bcdedit /set xsavedisable 1
+
+from an elevated command-prompt to disable AVX support.)
+
+Got a tip?
+~~~~~~~~~~
+
+If you think you know a cool Mozilla debugging trick, feel free to
+discuss it with `#developers <irc://irc.mozilla.org/developers>`__ and
+then post it here.
+
+Originally by Heikki Toivonen.
+
+.. |Screenshot of disabling assertions| image:: /@api/deki/files/420/=Win32-debug-nop.png
+   :class: internal
--- a/taskcluster/docs/index.rst
+++ b/taskcluster/docs/index.rst
@@ -29,9 +29,10 @@ check out the :doc:`how-to section <how-
     docker-images
     cron
     try
     actions
     release-promotion
     versioncontrol
     config
     how-tos
+    task-graph
     reference
--- a/taskcluster/docs/release-promotion.rst
+++ b/taskcluster/docs/release-promotion.rst
@@ -42,12 +42,13 @@ fully ship.
 
 In-depth relpro guide
 ---------------------
 
 .. toctree::
 
     release-promotion-action
     balrog
+    setting-up-an-update-server
     partials
     signing
     partner-repacks
     partner-attribution
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/setting-up-an-update-server.rst
@@ -0,0 +1,218 @@
+Setting Up An Update Server
+===========================
+
+The goal of this document is to provide instructions for installing a
+locally-served Firefox update.
+
+Obtaining an update MAR
+-----------------------
+
+Updates are served as MAR files. There are two common ways to obtain a
+MAR to use: download a prebuilt one, or build one yourself.
+
+Downloading a MAR
+~~~~~~~~~~~~~~~~~
+
+Prebuilt Nightly MARs can be found
+`here <https://archive.mozilla.org/pub/firefox/nightly/>`__ on
+archive.mozilla.org. Be sure that you use the one that matches your
+machine's configuration. For example, if you want the Nightly MAR from
+2019-09-17 for a 64 bit Windows machine, you probably want the MAR
+located at
+https://archive.mozilla.org/pub/firefox/nightly/2019/09/2019-09-17-09-36-29-mozilla-central/firefox-71.0a1.en-US.win64.complete.mar.
+
+Prebuilt MARs for release and beta can be found
+`here <https://archive.mozilla.org/pub/firefox/releases/>`__. Beta
+builds are those with a ``b`` in the version string. After locating the
+desired version, the MARs will be in the ``update`` directory. You want
+to use the MAR labelled ``complete``, not a partial MAR. Here is an
+example of an appropriate MAR file to use:
+https://archive.mozilla.org/pub/firefox/releases/69.0b9/update/win64/en-US/firefox-69.0b9.complete.mar.
+
+Building a MAR
+~~~~~~~~~~~~~~
+
+Building a MAR locally is more complicated. Part of the problem is that
+MARs are signed by Mozilla and so you cannot really build an "official"
+MAR yourself. This is a security measure designed to prevent anyone from
+serving malicious updates. If you want to use a locally-built MAR, the
+copy of Firefox being updated will need to be built to allow un-signed
+MARs. See :ref:`Building Firefox <Firefox Contributors' Quick Reference>`
+for more information on building Firefox locally. These changes will
+need to be made in order to use the locally built MAR:
+
+-  Put this line in the mozconfig file in root of the build directory
+   (create it if it does not exist):
+   ``ac_add_options --disable-verify-mar``
+-  Several files contain a line that must be uncommented. Open them and
+   find this line:
+   ``#DEFINES['DISABLE_UPDATER_AUTHENTICODE_CHECK'] = True``. Delete the
+   ``#`` at the beginning of the line to uncomment it. These are the
+   files that must be changed:
+
+   -  toolkit/components/maintenanceservice/moz.build
+   -  toolkit/mozapps/update/tests/moz.build
+
+Firefox should otherwise be built normally. After building, you may want
+to copy the installation of Firefox elsewhere. If you update the
+installation without moving it, attempts at further incremental builds
+will not work properly, and a clobber will be needed when building next.
+To move the installation, first call ``./mach package``, then copy
+``<obj dir>/dist/firefox`` elsewhere. The copied directory will be your
+install directory.
+
+If you are running Windows and want the `Mozilla Maintenance
+Service <https://support.mozilla.org/en-US/kb/what-mozilla-maintenance-service>`__
+to be used, there are a few additional steps to be taken here. First,
+the maintenance service needs to be "installed". Most likely, a
+different maintenance service is already installed, probably at
+``C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice.exe``.
+Backup that file to another location and replace it with
+``<obj dir>/dist/bin/maintenanceservice.exe``. Don't forget to restore
+the backup when you are done. Next, you will need to change the
+permissions on the Firefox install directory that you created. Both that
+directory and its parent directory should have permissions preventing
+the current user from writing to it.
+
+Now that you have a build of Firefox capable of using a locally-built
+MAR, it's time to build the MAR. First, build Firefox the way you want
+it to be after updating. If you want it to be the same before and after
+updating, this step is unnecessary and you can use the same build that
+you used to create the installation. Then run these commands,
+substituting ``<obj dir>``, ``<MAR output path>``, ``<version>`` and
+``<channel>`` appropriately:
+
+.. code:: bash
+
+   $ ./mach package
+   $ touch "<obj dir>/dist/firefox/precomplete"
+   $ MAR="<obj dir>/dist/host/bin/mar.exe" MOZ_PRODUCT_VERSION=<version> MAR_CHANNEL_ID=<channel> ./tools/update-packaging/make_full_update.sh <MAR output path> "<obj dir>/dist/firefox"
+
+For a local build, ``<channel>`` can be ``default``, and ``<version>``
+can be the value from ``browser/config/version.txt`` (or something
+arbitrarily large like ``2000.0a1``).
+
+.. container:: blockIndicator note
+
+   Note: It can be a bit tricky to get the ``make_full_update.sh``
+   script to accept paths with spaces.
+
+Serving the update
+------------------
+
+Preparing the update files
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, create the directory that updates will be served from and put the
+MAR file in it. Then, create a file within called ``update.xml`` with
+these contents, replacing ``<mar name>``, ``<hash>`` and ``<size>`` with
+the MAR's filename, its sha512 hash, and its file size in bytes.
+
+::
+
+   <?xml version="1.0" encoding="UTF-8"?>
+   <updates>
+       <update type="minor" displayVersion="2000.0a1" appVersion="2000.0a1" platformVersion="2000.0a1" buildID="21181002100236">
+           <patch type="complete" URL="http://127.0.0.1:8000/<mar name>" hashFunction="sha512" hashValue="<hash>" size="<size>"/>
+       </update>
+   </updates>
+
+If you've downloaded the MAR you're using, you'll find the sha512 value
+in a file called SHA512SUMS in the root of the release directory on
+archive.mozilla.org for a release or beta build (you'll have to search
+it for the file name of your MAR, since it includes the sha512 for every
+file that's part of that release), and for a nightly build you'll find a
+file with a .checksums extension adjacent to your MAR that contains that
+information (for instance, for the MAR file at
+https://archive.mozilla.org/pub/firefox/nightly/2019/09/2019-09-17-09-36-29-mozilla-central/firefox-71.0a1.en-US.win64.complete.mar,
+the file
+https://archive.mozilla.org/pub/firefox/nightly/2019/09/2019-09-17-09-36-29-mozilla-central/firefox-71.0a1.en-US.win64.checksums
+contains the sha512 for that file as well as for all the other win64
+files that are part of that nightly release).
+
+If you've built your own MAR, you can obtain its sha512 checksum by
+running the following command, which should work in Linux, macOS, or
+Windows in the MozillaBuild environment:
+
+.. code::
+
+   shasum --algorithm 512 <filename>
+
+On Windows, you can get the exact file size in bytes for your MAR by
+right clicking on it in the file explorer and selecting Properties.
+You'll find the correct size in bytes at the end of the line that begins
+"Size", **not** the one that begins "Size on disk". Be sure to remove
+the commas when you paste this number into the XML file.
+
+On macOS, you can get the exact size of your MAR by running the command:
+
+.. code::
+
+   stat -f%z <filename>
+
+Or on Linux, the same command would be:
+
+.. code::
+
+   stat --format "%s" <filename>
+
+Starting your update server
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now, start an update server to serve the update files on port 8000. An
+easy way to do this is with Python. Remember to navigate to the correct
+directory before starting the server. This is the Python2 command:
+
+.. code:: bash
+
+   $ python -m SimpleHTTPServer 8000
+
+or, this is the Python3 command:
+
+.. code:: bash
+
+   $ python3 -m http.server 8000
+
+.. container:: blockIndicator note
+
+   If you aren't sure that you started the server correctly, try using a
+   web browser to navigate to ``http://127.0.0.1:8000/update.xml`` and
+   make sure that you get the XML file you created earlier.
+
+Installing the update
+---------------------
+
+You may want to start by deleting any pending updates to ensure that no
+previously found updates interfere with installing the desired update.
+You can use this command with Firefox's browser console to determine the
+update directory:
+
+.. code::
+
+   const {FileUtils} = ChromeUtils.import("resource://gre/modules/FileUtils.jsm");
+   FileUtils.getDir("UpdRootD", [], false).path
+
+Once you have determined the update directory, close Firefox, browse to
+the directory and remove the subdirectory called ``updates``.
+
+| Next, you need to change the update URL to point to the local XML
+  file. This can be done most reliably with an enterprise policy. The
+  policy file location depends on the operating system you are using.
+| Windows/Linux: ``<install dir>/distribution/policies.json``
+| macOS: ``<install dir>/Contents/Resources/distribution/policies.json``
+| Create the ``distribution`` directory, if necessary, and put this in
+  ``policies.json``:
+
+::
+
+   {
+     "policies": {
+       "AppUpdateURL": "http://127.0.0.1:8000/update.xml"
+     }
+   }
+
+Now you are ready to update! Launch Firefox out of its installation
+directory and navigate to the Update section ``about:preferences``. You
+should see it downloading the update to the update directory. Since the
+transfer is entirely local this should finish quickly, and a "Restart to
+Update" button should appear. Click it to restart and apply the update.
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/task-graph.rst
@@ -0,0 +1,33 @@
+Task Graph
+==========
+
+After a change to the Gecko source code is pushed to version-control,
+jobs for that change appear
+on `Treeherder <https://treeherder.mozilla.org/>`__. How does this
+work?
+
+-  A "decision task" is created to decide what to do with the push.
+-  The decision task creates a lot of additional tasks. These tasks
+   include build and test tasks, along with lots of other kinds of tasks
+   to build docker images, build toolchains, perform analyses, check
+   syntax, and so on.
+-  These tasks are arranged in a "task graph", with some tasks (e.g.,
+   tests) depending on others (builds). Once its prerequisite tasks
+   complete, a dependent task begins.
+-  The result of each task is sent to
+   `TreeHerder <https://treeherder.mozilla.org>`__ where developers and
+   sheriffs can track the status of the push.
+-  The outputs from each task, log files, Firefox installers, and so on,
+   appear attached to each task when it completes. These are viewable in
+   the `Task
+   Inspector <https://tools.taskcluster.net/task-inspector/>`__.
+
+All of this is controlled from within the Gecko source code, through a
+process called *task-graph generation*.  This means it's easy to add a
+new job or tweak the parameters of a job in a `try
+push <https://wiki.mozilla.org/Build:TryServer>`__, eventually landing
+that change on an integration branch.
+
+The details of task-graph generation are documented :ref:`in the source
+code itself <TaskCluster Task-Graph Generation>`,
+including a some :ref:`quick recipes for common changes <How Tos>`.