Bug 1542246 - Add initial documentation for the new Search Engine configuration format. r=daleharvey
authorMark Banner <standard8@mozilla.com>
Fri, 28 Jun 2019 15:23:00 +0000
changeset 480573 f004ce63fd7134ab1559b409853d0c6e892cb2fd
parent 480572 8b1bf6faf9ff58017055c8bd0ec5da2fb0f7dbe3
child 480574 27557022172f30a1949312157fafe84497e6e9b4
push id88777
push usermbanner@mozilla.com
push dateFri, 28 Jun 2019 16:07:23 +0000
treeherderautoland@f004ce63fd71 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersdaleharvey
bugs1542246
milestone69.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1542246 - Add initial documentation for the new Search Engine configuration format. r=daleharvey Differential Revision: https://phabricator.services.mozilla.com/D34144
toolkit/components/search/docs/SearchEngineConfiguration.rst
toolkit/components/search/docs/index.rst
toolkit/components/search/moz.build
toolkit/components/search/schema/Readme.txt
toolkit/components/search/schema/search-engine-config-schema.json
toolkit/components/search/schema/search-engine-config-ui-schema.json
toolkit/docs/index.rst
tools/lint/codespell.yml
new file mode 100644
--- /dev/null
+++ b/toolkit/components/search/docs/SearchEngineConfiguration.rst
@@ -0,0 +1,217 @@
+===========================
+Search Engine Configuration
+===========================
+
+.. note::
+    This configuration is currently being implemented in `Bug 1542235`_.
+
+Configuration Management
+========================
+
+The application stores a dump of the configuration that is used for first
+initialisation. Subsequent updates to the configuration are either updates to the
+static dump, or they may be served via remote servers.
+
+The mechanism of delivering the dumps and settings to the Search Service is
+`Remote Settings`_
+
+Remote settings
+---------------
+
+The remote settings bucket for the search engine configuration list is
+``search-config``. The version that is currently being delivered
+to clients can be `viewed live`_.
+
+Configuration Schema
+====================
+
+The configuration format is defined via a `JSON schema`_. The search engine
+configuration schema is `stored in mozilla-central`_ and is uploaded to the
+Remote Settings server at convenient times after it changes.
+
+This document outlines the details of the schema and how the various sub-parts
+interact. For the current fields and descriptions, please see the `schema itself`_.
+
+Configuration Structure
+=======================
+
+.. note::
+    In the examples, only relevant properties are displayed.
+
+Overview
+--------
+
+The configuration is a JSON blob which has a list of keys which are the engine name:
+
+.. code-block:: js
+
+    {
+      "engine1": {
+        ...
+      },
+      "engine2": {
+        ...
+      }
+    }
+
+.. note::
+    The engine name is intended only as a key/identifier.
+
+Engine Objects
+--------------
+
+An engine's details are located in the properties of the object associated with it.
+An engine that is deployed globally could be listed simply as:
+
+.. code-block:: js
+
+    "engine1": {
+      "default": "no",
+      "telemetryId": "engine1",
+      "webExtensionId": "webext",
+      "webExtensionVersion": "1.0",
+      "appliesTo": [{
+        "included": {
+          "everywhere": true
+        }
+      }]
+    }
+
+The ``appliesTo`` section is an array of objects. At least one object is required
+to specify which regions/locales the engine is included within. If an
+``appliesTo`` object lists additional attributes then these will override any
+attributes at the top-level.
+
+For example, a more complex engine definition may be available only to users
+located specific regions or with certain locales. For example:
+
+.. code-block:: js
+
+    "engine1": {
+      "webExtensionId": "webext",
+      "webExtensionVersion": "1.0",
+      "appliesTo": [{
+        "included": {
+          "region": "us"
+        },
+        "webExtensionId": "webext-us",
+        "webExtensionVersion": "1.1"
+      }, {
+        "included": {
+          "region": "gb"
+        },
+        "webExtensionId": "webext-gb",
+        "webExtensionVersion": "1.2"
+      }]
+    }
+
+In this case users identified as being in the US region would use the WebExtension
+with identifier ``webext-engine1``, version 1.1. GB region users would get
+``webext-gb`` version 1.2, and all other users would get ``webext`` version 1.0.
+
+Engine Defaults
+---------------
+
+An engine may or may not be default for particular regions/locales. The ``default``
+property is a tri-state value with states of ``yes``, ``yes-if-no-other`` and
+``no``. Here's an example of how they apply:
+
+.. code-block:: js
+
+    "engine1": {
+      "appliesTo": [{
+        "included": {
+          "region": "us"
+        },
+        "default": "yes"
+      }, {
+        "excluded": {
+          "region": "us"
+        },
+        "default": "yes-if-no-other"
+      }]
+    },
+    "engine2": {
+      "appliesTo": [{
+        "included": {
+          "region": "gb"
+        },
+        "default": "yes"
+      }]
+    },
+    "engine3": {
+      "default": "no"
+      "appliesTo": [{
+        "included": {
+          "everywhere": true
+        },
+      }]
+    }
+
+In this example:
+
+    - engine1 is default in the US region, and all other regions except for GB.
+    - engine2 is default in only the GB region
+    - engine3 is never default anywhere
+
+Engine Ordering
+---------------
+
+The ``orderHint`` field indicates the suggested ordering of an engine relative to
+other engines when displayed to the user, unless the user has customized their
+ordering.
+
+If the engine is default, then it will be displayed as the first engine
+regardless of the ``orderHint`` field, unless the user has customized their
+ordering.
+
+For the ``orderHint`` field, a higher number indicates a higher rank.
+
+Example:
+
+.. code-block:: js
+
+    "engine1": {
+      "orderHint": 2000,
+      "default": "no",
+    },
+    "engine2": {
+      "orderHint": 1000,
+      "default": "yes"
+    },
+    "engine3": {
+      "orderHint": 500,
+      "default": "no"
+    }
+
+This would result in the order: ``engine2, engine1, engine3``.
+
+Engine Updates
+--------------
+
+Within each engine definition is the extension id and version, for example:
+
+.. code-block:: js
+
+    "engine": {
+      "webExtensionId": "webext",
+      "webExtensionVersion": "1.0",
+    }
+
+To locate an engine to use, the Search Service will look in the following locations (in order):
+
+#. within the user's install of the application.
+#. in the configuration to see if there is an ``attachment`` field.
+
+If the WebExtension is listed in the ``attachment``, then the app will download
+to the user's profile, if it is not already there.
+
+If an application is downloading the WebExtension, or it is not available, then
+it may use an earlier version of the WebExtension until a new one becomes available.
+
+.. _Bug 1542235: https://bugzilla.mozilla.org/show_bug.cgi?id=1542235
+.. _Remote Settings: /services/common/services/RemoteSettings
+.. _JSON schema: https://json-schema.org/
+.. _stored in mozilla-central:
+.. _schema itself: https://searchfox.org/mozilla-central/source/toolkit/components/search/schema/
+.. _viewed live: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-engine-configuration/records
new file mode 100644
--- /dev/null
+++ b/toolkit/components/search/docs/index.rst
@@ -0,0 +1,10 @@
+==============
+Search Service
+==============
+
+This is documentation for the Search Service.
+
+.. toctree::
+   :maxdepth: 2
+
+   SearchEngineConfiguration
--- a/toolkit/components/search/moz.build
+++ b/toolkit/components/search/moz.build
@@ -1,14 +1,16 @@
 # -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
 # vim: set filetype=python:
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
+SPHINX_TREES['/toolkit/search'] = 'docs'
+
 XPCSHELL_TESTS_MANIFESTS += [
     'tests/xpcshell/searchconfigs/xpcshell-1.ini',
     'tests/xpcshell/searchconfigs/xpcshell-2.ini',
     'tests/xpcshell/searchconfigs/xpcshell-3.ini',
     'tests/xpcshell/searchconfigs/xpcshell-4.ini',
     'tests/xpcshell/xpcshell.ini',
 ]
 
new file mode 100644
--- /dev/null
+++ b/toolkit/components/search/schema/Readme.txt
@@ -0,0 +1,7 @@
+The schemas in this directory are the primary source for the schemas they represent.
+
+They are uploaded to the RemoteSettings server to validate new configurations.
+
+Any changes should be validated by the Search team.
+
+See the documentation for more information: https://firefox-source-docs.mozilla.org/
new file mode 100644
--- /dev/null
+++ b/toolkit/components/search/schema/search-engine-config-schema.json
@@ -0,0 +1,291 @@
+{
+  "type": "object",
+  "required": [
+    "engineName"
+  ],
+  "properties": {
+    "engineName": {
+      "type": "string",
+      "title": "Engine Name",
+      "description": "General short name for this engine, only used as an identifier",
+      "pattern": "^[a-zA-Z-]+$"
+    },
+    "default": {
+      "$ref": "#/definitions/default"
+    },
+    "application": {
+      "$ref": "#/definitions/application"
+    },
+    "orderHint": {
+      "$ref": "#/definitions/orderHint"
+    },
+    "appliesTo": {
+      "type": "array",
+      "title": "Applies To",
+      "description": "This section defines the region/locales/application information for where a search engine is available, and any specifics for that region/locale/application. If there are no entries in the list, it is considered to be included everywhere",
+      "items": {
+        "$ref": "#/definitions/appliesToSection"
+      }
+    },
+    "telemetryId": {
+      "type": "string",
+      "title": "Telemetry Id",
+      "description": "The telemetry Id as used for some of SEARCH_COUNTS telemetry."
+    },
+    "webExtension": {
+      "$ref": "#/definitions/webExtension"
+    },
+    "codes": {
+      "$ref": "#/definitions/codes"
+    }
+  },
+  "definitions": {
+    "application": {
+      "type": "object",
+      "title": "Application Details",
+      "properties": {
+        "name": {
+          "type": "string",
+          "title": "Name",
+          "description": "The application this applies to (default/not specified is everywhere)",
+          "enum": [
+            "",
+            "desktop",
+            "android"
+          ]
+        },
+        "branches": {
+          "type": "array",
+          "title": "Branches",
+          "description": "Which branches this belongs to (not set = everywhere)",
+          "items": {
+            "type": "string",
+            "enum": [
+              "default",
+              "nightly",
+              "aurora",
+              "beta",
+              "release",
+              "esr"
+            ]
+          },
+          "uniqueItems": true
+        },
+        "minversion": {
+          "type": "string",
+          "title": "Minimum Version",
+          "description": "The minimum version this applies to"
+        },
+        "maxversion": {
+          "type": "string",
+          "title": "Maxium Version",
+          "description": "The maximum version this applies to"
+        }
+      }
+    },
+    "default": {
+      "type": "string",
+      "title": "Default Status",
+      "description": "Whether or not this engine should be default.",
+      "enum": [
+        "yes",
+        "yes-if-no-other",
+        "no"
+      ]
+    },
+    "orderHint": {
+      "type": "number",
+      "title": "Order Hint",
+      "description": "A hint to the display order (higher is a higer rank)"
+    },
+    "sapCodes": {
+      "type": "object",
+      "title": "Search Access Point Codes",
+      "description": "Codes for the search access points.",
+      "properties": {
+        "any": {
+          "type": "string",
+          "title": "Any",
+          "description": "SAP code that is used for all access points (overrides the others)."
+        },
+        "contextMenu": {
+          "type": "string",
+          "title": "Context Menu",
+          "description": "SAP code for searches from the context menu."
+        },
+        "homePage": {
+          "type": "string",
+          "title": "Home page",
+          "description": "SAP code for searches from the home page."
+        },
+        "keyword": {
+          "type": "string",
+          "title": "Keyword",
+          "description": "SAP code for searches via keywords."
+        },
+        "newTab": {
+          "type": "string",
+          "title": "New Tab",
+          "description": "SAP code for searches from the new tab page."
+        },
+        "searchBar": {
+          "type": "string",
+          "title": "Search Bar",
+          "description": "SAP code for searches from the search bar."
+        }
+      }
+    },
+    "codes": {
+      "type": "object",
+      "title": "Codes",
+      "description": "Various codes for the search engines",
+      "properties": {
+        "searchUrlGetExtraCodes": {
+          "type": "string",
+          "title": "Search URL Get Extra codes",
+          "description": "Extra params (for SAP codes) for the search URL (e.g. 'pc=foo')."
+        },
+        "searchFormExtraCodes": {
+          "type": "string",
+          "title": "Search Form Extra codes",
+          "description": "Extra params (for SAP codes) for the search form URL (e.g. 'pc=foo')."
+        },
+        "sapCodes": {
+          "$ref": "#/definitions/sapCodes"
+        }
+      }
+    },
+    "telemetryId": {
+      "type": "string",
+      "title": "Telemetry Id",
+      "description": "The telemetry Id as used for some of SEARCH_COUNTS telemetry."
+    },
+    "webExtension": {
+      "type": "object",
+      "title": "WebExtension",
+      "dependencies": {
+        "webExtensionId": [
+          "webExtensionVersion"
+        ],
+        "webExtensionVersion": [
+          "webExtensionId"
+        ]
+      },
+      "properties": {
+        "webExtensionId": {
+          "type": "string",
+          "title": "WebExtension Id",
+          "description": "The identifier (local part) of the associated WebExtension"
+        },
+        "webExtensionVersion": {
+          "type": "string",
+          "title": "WebExtension Version",
+          "description": "The version of the WebExtension to use"
+        },
+        "webExtensionLocale": {
+          "type": "string",
+          "title": "WebExtension Locale",
+          "description": "Overrides the WebExtension locales and specifies to use a particular one. Ideally this should only be used when really necessary, otherwise considered deprecated."
+        }
+      }
+    },
+    "regionDetails": {
+      "type": "array",
+      "title": "Regions",
+      "description": "Two-letter region codes.",
+      "items": {
+        "type": "string",
+        "pattern": "^[a-z][a-z]$",
+        "minLength": 2,
+        "maxLength": 2
+      }
+    },
+    "localeDetails": {
+      "type": "object",
+      "title": "Locales",
+      "description": "Locale codes.",
+      "properties": {
+        "matches": {
+          "type": "array",
+          "title": "Matches exactly the codes",
+          "items": {
+            "type": "string",
+            "pattern": "^[a-z][a-z](\\-[A-Z][A-Z])?$",
+            "minLength": 2,
+            "maxLength": 5
+          }
+        },
+        "startsWith": {
+          "type": "array",
+          "title": "Matches any code starting with",
+          "items": {
+            "type": "string",
+            "pattern": "^[a-z][a-z]$",
+            "minLength": 2,
+            "maxLength": 2
+          }
+        }
+      }
+    },
+    "included": {
+      "type": "object",
+      "title": "Included Locations",
+      "description": "The locations to which this section applies. Note: Regions and Locales are 'and'ed together.",
+      "properties": {
+        "everywhere": {
+          "type": "boolean",
+          "title": "Everywhere",
+          "description": "Set to true to signify that this is included everywhere."
+        },
+        "regions": {
+          "$ref" : "#/definitions/regionDetails"
+        },
+        "locales": {
+          "$ref": "#/definitions/localeDetails"
+        }
+      }
+    },
+    "excluded": {
+      "type": "object",
+      "title": "Excluded Locations",
+      "description": "The locations to which this section applies. Note: Regions and Locales are 'and'ed together.",
+      "properties": {
+        "regions": {
+          "$ref" : "#/definitions/regionDetails"
+        },
+        "locales": {
+          "$ref": "#/definitions/localeDetails"
+        }
+      }
+    },
+    "appliesToSection": {
+      "type": "object",
+      "properties": {
+        "default": {
+          "$ref": "#/definitions/default"
+        },
+        "orderHint": {
+          "$ref": "#/definitions/orderHint"
+        },
+        "included": {
+          "$ref": "#/definitions/included"
+        },
+        "excluded": {
+          "$ref": "#/definitions/excluded"
+        },
+        "application": {
+          "$ref": "#/definitions/application"
+        },
+        "webextension": {
+          "$ref": "#/definitions/webExtension"
+        },
+        "telemetryId": {
+          "$ref": "#/definitions/telemetryId"
+        },
+        "codes": {
+          "$ref": "#/definitions/codes"
+        }
+      }
+    }
+  }
+}
new file mode 100644
--- /dev/null
+++ b/toolkit/components/search/schema/search-engine-config-ui-schema.json
@@ -0,0 +1,15 @@
+{
+  "engineName": {
+    "ui:autofocus": true
+  },
+  "ui:order": [
+    "engineName",
+    "default",
+    "application",
+    "orderHint",
+    "telemetryId",
+    "webExtension",
+    "codes",
+    "appliesTo"
+  ]
+}
--- a/toolkit/docs/index.rst
+++ b/toolkit/docs/index.rst
@@ -6,15 +6,16 @@ This is the nascent documentation of the
 
 .. toctree::
    :maxdepth: 1
 
    mozapps/extensions/addon-manager/index
    components/crashes/crash-manager/index
    crashreporter/crashreporter/index
    components/featuregates/featuregates/index
+   search/index
    components/normandy/normandy/index
    modules/subprocess/toolkit_modules/subprocess/index
    components/telemetry/telemetry/index
    modules/toolkit_modules/index
    content/toolkit_widgets/index
    components/url-classifier/url-classifier/index
    components/extensions/webextensions/index
--- a/tools/lint/codespell.yml
+++ b/tools/lint/codespell.yml
@@ -34,17 +34,19 @@ codespell:
         - security/manager/locales/en-US/
         - services/sync/locales/en-US/
         - taskcluster/docs/
         - testing/mozbase/docs/
         - toolkit/components/extensions/docs/
         - toolkit/components/normandy/docs/
         - toolkit/components/telemetry/docs/
         - toolkit/crashreporter/docs/
+        - toolkit/docs/
         - toolkit/locales/en-US/
+        - toolkit/components/search/docs/
         - tools/lint/
         - tools/tryselect/
     # List of extensions coming from:
     # tools/lint/{flake8,eslint}.yml
     # tools/mach_commands.py (clang-format)
     # + documentation
     # + localization files
     extensions: