Bug 1636158 - Document how to use attachment dumps in RemoteSettings r=leplatrem
authorRob Wu <rob@robwu.nl>
Fri, 22 May 2020 13:09:50 +0000
changeset 531643 049bdaa3e410558ddcb1a71bbc26f3a16b13d0ed
parent 531642 d3770d94e7c0dc8a5b06b526a01cde67de83104b
child 531644 d5255379819c0c4577eb4cbe46c8bb141a3d21d2
push id37441
push userapavel@mozilla.com
push dateFri, 22 May 2020 21:38:53 +0000
treeherdermozilla-central@d6abd35b54ad [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersleplatrem
bugs1636158
milestone78.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1636158 - Document how to use attachment dumps in RemoteSettings r=leplatrem Differential Revision: https://phabricator.services.mozilla.com/D76159
services/common/docs/RemoteSettings.rst
--- a/services/common/docs/RemoteSettings.rst
+++ b/services/common/docs/RemoteSettings.rst
@@ -159,16 +159,17 @@ The provided helper will:
     Pass the ``useCache`` option to use an IndexedDB-based cache, and unlock the following features:
 
     The ``fallbackToCache`` option allows callers to fall back to the cached file and record, if the requested record's attachment fails to download.
     This enables callers to always have a valid pair of attachment and record,
     provided that the attachment has been retrieved at least once.
 
     The ``fallbackToDump`` option activates a fallback to a dump that has been
     packaged with the client, when other ways to load the attachment have failed.
+    See :ref:`_services/packaging-attachments` for more information.
 
 .. note::
 
     A ``downloadAsBytes()`` method returning an ``ArrayBuffer`` is also available, if writing the attachment into the user profile is not necessary.
 
 
 
 .. _services/initial-data:
@@ -177,22 +178,70 @@ Initial data
 ------------
 
 It is possible to package a dump of the server records that will be loaded into the local database when no synchronization has happened yet.
 
 The JSON dump will serve as the default dataset for ``.get()``, instead of doing a round-trip to pull the latest data. It will also reduce the amount of data to be downloaded on the first synchronization.
 
 #. Place the JSON dump of the server records in the ``services/settings/dumps/main/`` folder
 #. Add the filename to the ``FINAL_TARGET_FILES`` list in ``services/settings/dumps/main/moz.build``
+#. Add the filename to the ``[browser]`` section of ``mobile/android/installer/package-manifest.in`` IF the file should be bundled with Android.
 
 Now, when ``RemoteSettings("some-key").get()`` is called from an empty profile, the ``some-key.json`` file is going to be loaded before the results are returned.
 
+JSON dumps in the tree are periodically updated by ``taskcluster/docker/periodic-updates/scripts/periodic_file_updates.sh``.
+
 .. note::
 
-    JSON dumps are not shipped on Android to minimize the installer size.
+   The example above uses "main" because that's the default bucket name.
+   If you have customized the bucket name, use the actual bucket name instead of "main".
+
+.. _services/packaging-attachments:
+
+Packaging attachments
+~~~~~~~~~~~~~~~~~~~~~
+
+Attachments are not included in the JSON dumps by default. You may choose to package the attachment
+with the client, for example if it is important to have the data available at the first startup
+without requiring network activity. Or if most users would download the attachment anyway.
+Only package attachments if needed, since they increase the file size of the Firefox installer.
+
+To package an attachment for consumers of the `download()` method:
+
+#. Select the desired attachment record from the JSON dump of the server records, and place it at
+   ``services/settings/dumps/<bucket name>/<collection name>/<attachment id>.meta.json``.
+   The ``<attachment id>`` defaults to the ``id`` field of the record. If this ``id`` is not fixed,
+   you must choose a custom ID that can be relied upon as a long-term attachment identifier. See
+   the notes below for more details.
+#. Download the attachment associated with the record, and place it at
+   ``services/settings/dumps/<bucket name>/<collection name>/<attachment id>``.
+#. Update ``taskcluster/docker/periodic-updates/scripts/periodic_file_updates.sh`` and add the attachment,
+   by editing the ``compare_remote_settings_files`` function and describing the attachment.
+   Unlike JSON dumps, attachments must explicitly be listed in that update script, because the
+   attachment selection logic needs to be codified in a ``jq`` filter in the script.
+   For an example, see `Bug 1636158 <https://bugzilla.mozilla.org/show_bug.cgi?id=1636158>`_.
+#. Register the location of the ``<attachment id>.meta.json`` and ``<attachment id>`` in the
+   ``moz.build`` file of the collection folder, and possibly ``package-manifest.in``,
+   as described in `the previous section about registering JSON dumps <services/initial-data>`.
+
+.. note::
+
+   ``<attachment id>`` is used to derive the file names of the packaged attachment dump, and as the
+   key for the (optional) cache where attachment updates from the network are saved. If the cache
+   is enabled, the attachment identifier is expected to be fixed across client application updates.
+   If that expectation cannot be met, the ``attachmentId`` option of the ``download`` method of the
+   attachment downloader should be used to override the attachment ID with a custom (stable) value.
+
+.. note::
+
+   The contents of the ``.meta.json`` file is already contained within the records, but separated
+   from the main set of records to ensure the availability of the original record with the data,
+   independently of the packaged or downloaded records.
+   This file may become optional in a future update, see `Bug 1640059 <https://bugzilla.mozilla.org/show_bug.cgi?id=1640059>`_.
+
 
 Targets and A/B testing
 =======================
 
 In order to deliver settings to subsets of the population, you can set targets on entries (platform, language, channel, version range, preferences values, samples, etc.) when editing records on the server.
 
 From the client API standpoint, this is completely transparent: the ``.get()`` method — as well as the event data — will always filter the entries on which the target matches.