Bug 1599045, part 2 - merge compare-locales docs into build docs, r=flod
authorAxel Hecht <axel@pike.org>
Tue, 03 Dec 2019 15:43:10 +0000
changeset 505083 a4cd8dfba3b6a3fe82c256e2634434d7d71390cd
parent 505082 8ea8346e9edb68cd462bf400b7b90f467d75b350
child 505084 c566f52a502b1269e066cf09219455be3599de30
push id36879
push userncsoregi@mozilla.com
push dateTue, 03 Dec 2019 21:53:45 +0000
treeherdermozilla-central@3c08edf74d03 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersflod
bugs1599045
milestone73.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1599045, part 2 - merge compare-locales docs into build docs, r=flod The "Localization" docs in tools/compare-locales are really mostly build, so merging that content into the builds doc. Removing Android parts, add Fluent there, noting DTD deprecation. Moving the glossary to the actual l10n docs. With proper top-level structure, show 2 levels of l10n and intl on the front-matter page. Differential Revision: https://phabricator.services.mozilla.com/D55126
build/docs/locales.rst
intl/l10n/docs/glossary.rst
intl/l10n/docs/index.rst
tools/compare-locales/docs/glossary.rst
tools/compare-locales/docs/index.rst
tools/docs/config.yml
tools/docs/index.rst
tools/moz.build
--- a/build/docs/locales.rst
+++ b/build/docs/locales.rst
@@ -1,13 +1,13 @@
 .. _localization:
 
-===================
-Localization (l10n)
-===================
+================
+Localized Builds
+================
 
 Single-locale language repacks
 ==============================
 
 To save on build time, the build system and automation collaborate to allow
 downloading a packaged en-US Firefox, performing some locale-specific
 post-processing, and re-packaging a locale-specific Firefox.  Such artifacts
 are termed "single-locale language repacks".  There is another concept of a
@@ -72,8 +72,185 @@ If you want to create a single build wit
 #. Create the multilingual package:
 
    .. code-block:: shell
 
       AB_CD=multi ./mach package
 
 This `currently <https://bugzilla.mozilla.org/show_bug.cgi?id=1362496>`_ only
 works for Firefox for Android.
+
+Exposing strings
+================
+
+Localizers only handle a few file formats in well-known locations in the
+source tree.
+
+The locations are specified by TOML files. They're part of the bigger
+localization ecosystem at Mozilla, and `the documentation about the
+file format <http://moz-l10n-config.readthedocs.io/en/latest/fileformat.html>`_
+explains how to set them up, and what the entries mean. In short, you find
+
+.. code-block:: toml
+
+    [[paths]]
+        reference = browser/locales/en-US/**
+        l10n = {l}browser/**
+
+to add a directory for all localizations. Changes to these files are best
+submitted for review by :Pike or :flod.
+
+These configuration files are the future, and right now, we still have
+support for the previous way to configuring l10n, which is described below.
+
+The locations are commonly in directories like
+
+    :file:`browser/`\ ``locales/en-US/``\ :file:`subdir/file.ext`
+
+The first thing to note is that only files beneath :file:`locales/en-US` are
+exposed to localizers. The second thing to note is that only a few directories
+are exposed. Which directories are exposed is defined in files called
+``l10n.ini``, which are at a
+`few places <https://dxr.mozilla.org/mozilla-central/search?q=path%3Al10n.ini&redirect=true>`_
+in the source code.
+
+An example looks like this
+
+.. code-block:: ini
+
+    [general]
+    depth = ../..
+
+    [compare]
+    dirs = browser
+        browser/branding/official
+
+    [includes]
+    toolkit = toolkit/locales/l10n.ini
+
+This tells the l10n infrastructure three things: Resolve the paths against the
+directory two levels up, include files in :file:`browser/locales/en-US` and
+:file:`browser/branding/official/locales/en-US`, and load more data from
+:file:`toolkit/locales/l10n.ini`.
+
+For projects like Thunderbird and SeaMonkey in ``comm-central``, additional
+data needs to be provided when including an ``l10n.ini`` from a different
+repository:
+
+.. code-block:: ini
+
+    [include_toolkit]
+    type = hg
+    mozilla = mozilla-central
+    repo = https://hg.mozilla.org/
+    l10n.ini = toolkit/locales/l10n.ini
+
+This tells the l10n pieces where to find the repository, and where inside
+that repository the ``l10n.ini`` file is. This is needed because for local
+builds, :file:`mail/locales/l10n.ini` references
+:file:`mozilla/toolkit/locales/l10n.ini`, which is where the comm-central
+build setup expects toolkit to be.
+
+Now that the directories exposed to l10n are known, we can talk about the
+supported file formats.
+
+File formats
+------------
+
+The following file formats are known to the l10n tool chains:
+
+Fluent
+    Used in Firefox UI, both declarative and programmatically.
+DTD
+    Deprecated. Used in XUL and XHTML.
+Properties
+    Used from JavaScript and C++. When used from js, also comes with
+    `plural support <https://developer.mozilla.org/docs/Mozilla/Localization/Localization_and_Plurals>`_.
+ini
+    Used by the crashreporter and updater, avoid if possible.
+foo.defines
+    Used during builds, for example to create :file:`install.rdf` for
+    language packs.
+
+Adding new formats involves changing various different tools, and is strongly
+discouraged.
+
+Exceptions
+----------
+Generally, anything that exists in ``en-US`` needs a one-to-one mapping in
+all localizations. There are a few cases where that's not wanted, notably
+around search settings and spell-checking dictionaries.
+
+To enable tools to adjust to those exceptions, there's a python-coded
+:py:mod:`filter.py`, implementing :py:func:`test`, with the following
+signature
+
+.. code-block:: python
+
+    def test(mod, path, entity = None):
+        if does_not_matter:
+            return "ignore"
+        if show_but_do_not_merge:
+            return "report"
+        # default behavior, localizer or build need to do something
+        return "error"
+
+For any missing file, this function is called with ``mod`` being
+the *module*, and ``path`` being the relative path inside
+:file:`locales/en-US`. The module is the top-level dir as referenced in
+:file:`l10n.ini`.
+
+For missing strings, the :py:data:`entity` parameter is the key of the string
+in the en-US file.
+
+l10n-merge
+==========
+
+Gecko doesn't support fallback from a localization to ``en-US`` at runtime.
+Thus, the build needs to ensure that the localization as it's built into
+the package has all required strings, and that the strings don't contain
+errors. To ensure that, we're *merging* the localization and ``en-US``
+at build time, nick-named :term:`l10n-merge`.
+
+The process can be manually triggered via
+
+.. code-block:: bash
+
+    $> ./mach build merge-de
+
+It creates another directory in the object dir, :file:`merge-dir/ab-CD`, in
+which the modified files are stored. The actual repackaging process looks for
+the localized files in the merge dir first, then the localized file, and then
+in ``en-US``. Thus, for the ``de`` localization of
+:file:`browser/locales/en-US/chrome/browser/browser.dtd`, it checks
+
+1. :file:`$objdir/browser/locales/merge-de/browser/chrome/browser/browser.dtd`
+2. :file:`$(LOCALE_BASEDIR)/de/browser/chrome/browser/browser.dtd`
+3. :file:`browser/locales/en-US/chrome/browser/browser.dtd`
+
+and will include the first of those files it finds.
+
+l10n-merge modifies a file if it supports the particular file type, and there
+are missing strings which are not filtered out, or if an existing string
+shows an error. See the Checks section below for details.
+
+Checks
+------
+
+As part of the build and other localization tool chains, we run a variety
+of source-based checks. Think of them as linters.
+
+The suite of checks is usually determined by file type, i.e., there's a
+suite of checks for DTD files and one for properties files, etc. An exception
+are Android-specific checks.
+
+Localizations
+-------------
+
+Now that we talked in-depth about how to expose content to localizers,
+where are the localizations?
+
+We host a mercurial repository per locale and per branch. All of our
+localizations can be found on https://hg.mozilla.org/l10n-central/.
+
+You can search inside our localized files on
+`Transvision <https://transvision.mozfr.org/>`_ and
+https://dxr.mozilla.org/l10n-central/source/.
rename from tools/compare-locales/docs/glossary.rst
rename to intl/l10n/docs/glossary.rst
--- a/intl/l10n/docs/index.rst
+++ b/intl/l10n/docs/index.rst
@@ -17,8 +17,9 @@ and messages, while internationalization
   cultural adaptation of icons, communication styles, colors, and UX.
 
 .. toctree::
    :maxdepth: 2
 
    overview
    fluent/index
    migrations/index
+   glossary
deleted file mode 100644
--- a/tools/compare-locales/docs/index.rst
+++ /dev/null
@@ -1,205 +0,0 @@
-============
-Localization
-============
-
-.. toctree::
-   :maxdepth: 1
-
-   glossary
-
-The documentation here is targeted at developers, writing localizable code
-for Firefox and Firefox for Android, as well as Thunderbird and SeaMonkey.
-
-If you haven't dealt with localization in gecko code before, it's a good
-idea to check the :doc:`./glossary` for what localization is, and which terms
-we use for what.
-
-Exposing strings
-----------------
-
-Localizers only handle a few file formats in well-known locations in the
-source tree.
-
-The locations are specified by TOML files. They're part of the bigger
-localization ecosystem at Mozilla, and `the documentation about the
-file format <http://moz-l10n-config.readthedocs.io/en/latest/fileformat.html>`_
-explains how to set them up, and what the entries mean. In short, you find
-
-.. code-block:: toml
-
-    [[paths]]
-        reference = browser/locales/en-US/**
-        l10n = {l}browser/**
-
-to add a directory for all localizations. Changes to these files are best
-submitted for review by :Pike or :flod.
-
-These configuration files are the future, and right now, we still have
-support for the previous way to configuring l10n, which is described below.
-
-The locations are commonly in directories like
-
-    :file:`browser/`\ ``locales/en-US/``\ :file:`subdir/file.ext`
-
-The first thing to note is that only files beneath :file:`locales/en-US` are
-exposed to localizers. The second thing to note is that only a few directories
-are exposed. Which directories are exposed is defined in files called
-``l10n.ini``, which are at a
-`few places <https://dxr.mozilla.org/mozilla-central/search?q=path%3Al10n.ini&redirect=true>`_
-in the source code.
-
-An example looks like this
-
-.. code-block:: ini
-
-    [general]
-    depth = ../..
-
-    [compare]
-    dirs = browser
-        browser/branding/official
-
-    [includes]
-    toolkit = toolkit/locales/l10n.ini
-
-This tells the l10n infrastructure three things: Resolve the paths against the
-directory two levels up, include files in :file:`browser/locales/en-US` and
-:file:`browser/branding/official/locales/en-US`, and load more data from
-:file:`toolkit/locales/l10n.ini`.
-
-For projects like Thunderbird and SeaMonkey in ``comm-central``, additional
-data needs to be provided when including an ``l10n.ini`` from a different
-repository:
-
-.. code-block:: ini
-
-    [include_toolkit]
-    type = hg
-    mozilla = mozilla-central
-    repo = https://hg.mozilla.org/
-    l10n.ini = toolkit/locales/l10n.ini
-
-This tells the l10n pieces where to find the repository, and where inside
-that repository the ``l10n.ini`` file is. This is needed because for local
-builds, :file:`mail/locales/l10n.ini` references
-:file:`mozilla/toolkit/locales/l10n.ini`, which is where the comm-central
-build setup expects toolkit to be.
-
-Now that the directories exposed to l10n are known, we can talk about the
-supported file formats.
-
-File formats
-------------
-
-This is just a quick overview, please check the
-`XUL Tutorial <https://developer.mozilla.org/docs/Mozilla/Tech/XUL/Tutorial/Localization>`_
-for an in-depth tour.
-
-The following file formats are known to the l10n tool chains:
-
-DTD
-    Used in XUL and XHTML. Also for Android native strings.
-Properties
-    Used from JavaScript and C++. When used from js, also comes with
-    `plural support <https://developer.mozilla.org/docs/Mozilla/Localization/Localization_and_Plurals>`_.
-ini
-    Used by the crashreporter and updater, avoid if possible.
-foo.defines
-    Used during builds, for example to create :file:`install.rdf` for
-    language packs.
-
-Adding new formats involves changing various different tools, and is strongly
-discouraged.
-
-Exceptions
-----------
-Generally, anything that exists in ``en-US`` needs a one-to-one mapping in
-all localizations. There are a few cases where that's not wanted, notably
-around search settings and spell-checking dictionaries.
-
-To enable tools to adjust to those exceptions, there's a python-coded
-:py:mod:`filter.py`, implementing :py:func:`test`, with the following
-signature
-
-.. code-block:: python
-
-    def test(mod, path, entity = None):
-        if does_not_matter:
-            return "ignore"
-        if show_but_do_not_merge:
-            return "report"
-        # default behavior, localizer or build need to do something
-        return "error"
-
-For any missing file, this function is called with ``mod`` being
-the *module*, and ``path`` being the relative path inside
-:file:`locales/en-US`. The module is the top-level dir as referenced in
-:file:`l10n.ini`.
-
-For missing strings, the :py:data:`entity` parameter is the key of the string
-in the en-US file.
-
-l10n-merge
-----------
-
-Gecko doesn't support fallback from a localization to ``en-US`` at runtime.
-Thus, the build needs to ensure that the localization as it's built into
-the package has all required strings, and that the strings don't contain
-errors. To ensure that, we're *merging* the localization and ``en-US``
-at build time, nick-named :term:`l10n-merge`.
-
-The process can be manually triggered via
-
-.. code-block:: bash
-
-    $> ./mach build merge-de
-
-It creates another directory in the object dir, :file:`merge-dir/ab-CD`, in
-which the modified files are stored. The actual repackaging process looks for
-the localized files in the merge dir first, then the localized file, and then
-in ``en-US``. Thus, for the ``de`` localization of
-:file:`browser/locales/en-US/chrome/browser/browser.dtd`, it checks
-
-1. :file:`$objdir/browser/locales/merge-de/browser/chrome/browser/browser.dtd`
-2. :file:`$(LOCALE_BASEDIR)/de/browser/chrome/browser/browser.dtd`
-3. :file:`browser/locales/en-US/chrome/browser/browser.dtd`
-
-and will include the first of those files it finds.
-
-l10n-merge modifies a file if it supports the particular file type, and there
-are missing strings which are not filtered out, or if an existing string
-shows an error. See the Checks section below for details.
-
-Checks
-------
-
-As part of the build and other localization tool chains, we run a variety
-of source-based checks. Think of them as linters.
-
-The suite of checks is usually determined by file type, i.e., there's a
-suite of checks for DTD files and one for properties files, etc. An exception
-are Android-specific checks.
-
-Android
-^^^^^^^
-
-For Android, we need to localize :file:`strings.xml`. We're doing so via DTD
-files, which is mostly OK. But the strings inside the XML file have to
-satisfy additional constraints about quotes etc, that are not part of XML.
-There's probably some historic background on why things are the way they are.
-
-The Android-specific checks are enabled for DTD files that are in
-:file:`mobile/android/base/locales/en-US/`.
-
-Localizations
--------------
-
-Now that we talked in-depth about how to expose content to localizers,
-where are the localizations?
-
-We host a mercurial repository per locale and per branch. All of our
-localizations can be found on https://hg.mozilla.org/l10n-central/.
-
-You can search inside our localized files on
-`Transvision <https://transvision.mozfr.org/>`_ and
-https://dxr.mozilla.org/l10n-central/source/.
--- a/tools/docs/config.yml
+++ b/tools/docs/config.yml
@@ -27,17 +27,16 @@ categories:
         - testing/geckodriver
         - web-platform
     code_quality_doc:
         - tools/lint
         - tools/static-analysis
     l10n_doc:
         - intl
         - l10n
-        - tools/compare-locales
     python_doc:
         - mozbase
         - python
     fennec_doc:
         - mobile/android
 
 redirects:
     browser/browser: browser
--- a/tools/docs/index.rst
+++ b/tools/docs/index.rst
@@ -18,17 +18,17 @@ Mozilla Source Tree Documentation
 .. toctree::
    :caption: Testing
    :maxdepth: 1
 
    {testing_doc}
 
 .. toctree::
    :caption: Localization
-   :maxdepth: 1
+   :maxdepth: 2
 
    {l10n_doc}
 
 .. toctree::
    :caption: Python
    :maxdepth: 1
 
    {python_doc}
--- a/tools/moz.build
+++ b/tools/moz.build
@@ -5,19 +5,16 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 with Files("**"):
     BUG_COMPONENT = ("Core", "General")
 
 with Files("code-coverage/**"):
     BUG_COMPONENT = ("Testing", "Code Coverage")
 
-with Files("compare-locales/docs/**"):
-    BUG_COMPONENT = ('Mozilla Localizations', 'Documentation')
-
 with Files("compare-locales/mach_commands.py"):
     BUG_COMPONENT = ("Localization Infrastructure and Tools", "compare-locales")
 
 with Files("coverity/**"):
     BUG_COMPONENT = ("Firefox Build System", "Source Code Analysis")
 
 with Files("docs/**"):
     BUG_COMPONENT = ("Firefox Build System", "Generated Documentation")
@@ -53,21 +50,16 @@ with Files("update-packaging/**"):
 with Files("update-verify/**"):
     BUG_COMPONENT = ("Release Engineering", "Release Automation: Updates")
 
 SPHINX_TREES['lint'] = 'lint/docs'
 
 with Files('lint/docs/**'):
     SCHEDULES.exclusive = ['docs']
 
-SPHINX_TREES['compare-locales'] = 'compare-locales/docs'
-
-with Files('compare-locales/docs/**'):
-    SCHEDULES.exclusive = ['docs']
-
 SPHINX_TREES['try'] = 'tryselect/docs'
 
 SPHINX_TREES['docs'] = 'docs/docs'
 
 SPHINX_TREES['static-analysis'] = 'clang-tidy/docs'
 
 with Files('tryselect/docs/**'):
     SCHEDULES.exclusive = ['docs']