Bug 1124212 - Create telemetry documentation in-tree. r=vladan
authorGeorg Fritzsche <georg.fritzsche@googlemail.com>
Tue, 10 Feb 2015 16:59:01 +0100
changeset 228890 5688e50ade0a50a7b184fe8d0eac5c8d91da968b
parent 228889 3e17fb71357cb09d9adfab48d53a55aba3167e2e
child 228891 2b94fcb274a1f93d0f65645f7fff212e00136bab
push id13888
push userryanvm@gmail.com
push dateFri, 13 Feb 2015 17:27:21 +0000
treeherderb2g-inbound@1cd0ddc21eec [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersvladan
bugs1124212
milestone38.0a1
Bug 1124212 - Create telemetry documentation in-tree. r=vladan
toolkit/components/telemetry/docs/common-ping.rst
toolkit/components/telemetry/docs/environment.rst
toolkit/components/telemetry/docs/index.rst
toolkit/components/telemetry/docs/main-ping.rst
toolkit/components/telemetry/docs/pings.rst
toolkit/components/telemetry/moz.build
new file mode 100644
--- /dev/null
+++ b/toolkit/components/telemetry/docs/common-ping.rst
@@ -0,0 +1,39 @@
+
+Common ping format
+==================
+
+This defines the top-level structure of a Telemetry ping.
+It contains basic information shared between different ping types, which enables proper storage and processing of the raw pings server-side.
+
+It also contains optional further information:
+
+* the :doc:`environment data <environment>`, which contains important info to correlate the measurements against
+* the ``clientId``, a UUID identifying a profile and allowing user-oriented correlation of data
+
+*Note:* Both are not submitted with all ping types due to privacy concerns. This and the data it that can be correlated against is inspected under the `data collection policy <https://wiki.mozilla.org/Firefox/Data_Collection>`_.
+
+Finally, the structure also contains the `payload`, which is the specific data submitted for the respective *ping type*.
+
+Structure::
+
+    {
+      type: <string>, // "main", "activation", "deletion", ...
+      id: <UUID>, // a UUID that identifies this ping
+      creationDate: <ISO date>, // the date the ping was generated
+      version: <number>, // the version of the ping format, currently 2
+
+      application: {
+        architecture: <string>, // build architecture, e.g. x86
+        buildId: <string>, // "20141126041045"
+        name: <string>, // "Firefox"
+        version: <string>, // "35.0"
+        vendor: <string>, // "Mozilla"
+        platformVersion: <string>, // "35.0"
+        xpcomAbi: <string>, // e.g. "x86-msvc"
+        channel: <string>, // "beta"
+      },
+
+      clientId: <UUID>, // optional
+      environment: { ... }, // optional, not all pings contain the environment
+      payload: { ... }, // the actual payload data for this ping type
+    }
new file mode 100644
--- /dev/null
+++ b/toolkit/components/telemetry/docs/environment.rst
@@ -0,0 +1,180 @@
+
+Environment
+===========
+
+The environment consists of data that is expected to be characteristic for performance and other behavior and not expected to change too often.
+
+Changes to most of these data points are detected (where possible and sensible) and will lead to a session split in the :doc:`main-ping`.
+The environment data may also be submitted by other ping types.
+
+*Note:* This is not submitted with all ping types due to privacy concerns. This and other data is inspected under the `data collection policy <https://wiki.mozilla.org/Firefox/Data_Collection>`_.
+
+Structure::
+
+    {
+      build: {
+        applicationId: <string>, // nsIXULAppInfo.ID
+        architecture: <string>, // e.g. "x86", build architecture for the active build
+        architecturesInBinary: <string>, // e.g. "i386-x86_64", from nsIMacUtils.architecturesInBinary, only present for mac universal builds
+        buildId: <string>, // e.g. "20141126041045"
+        version: <string>, // e.g. "35.0"
+        vendor: <string>, // e.g. "Mozilla"
+        platformVersion: <string>, // e.g. "35.0"
+        xpcomAbi: <string>, // e.g. "x86-msvc"
+        hotfixVersion: <string>, // e.g. "20141211.01"
+      },
+      settings: {
+        blocklistEnabled: <bool>, // false on failure
+        isDefaultBrowser: <bool>, // null on failure
+        e10sEnabled: <bool>, // false on failure
+        telemetryEnabled: <bool>, // false on failure
+        locale: <string>, // e.g. "it", null on failure
+        update: {
+          channel: <string>, // e.g. "release", null on failure
+          enabled: <bool>, // false on failure
+          autoDownload: <bool>, // false on failure
+        },
+        userPrefs: {
+          // Two possible behaviours: values of the whitelisted prefs, or for some prefs we
+          // only record they are present with value being set to null.
+        },
+      },
+      profile: {
+        creationDate: <integer>, // integer days since UNIX epoch, e.g. 16446
+        resetDate: <integer>, // integer days since UNIX epoch, e.g. 16446 - optional
+      },
+      partner: {
+        distributionId: <string>, // pref "distribution.id", null on failure
+        distributionVersion: <string>, // pref "distribution.version", null on failure
+        partnerId: <string>, // pref mozilla.partner.id, null on failure
+        distributor: <string>, // pref app.distributor, null on failure
+        distributorChannel: <string>, // pref app.distributor.channel, null on failure
+        partnerNames: [
+          // list from prefs app.partner.<name>=<name>
+        ],
+      },
+      system: {
+        memoryMB: <number>,
+        isWow64: <bool>, // windows-only
+        cpu: {
+            count: <number>,  // e.g. 8, or null on failure
+            vendor: <string>, // e.g. "GenuineIntel", or null on failure
+            family: <string>, // null on failure
+            model: <string>, // null on failure
+            stepping: <string>, // null on failure
+            extensions: [
+              <string>,
+              ...
+              // as applicable:
+              // "MMX", "SSE", "SSE2", "SSE3", "SSSE3", "SSE4A", "SSE4_1",
+              // "SSE4_2", "EDSP", "ARMv6", "ARMv7", "NEON"
+            ],
+        },
+        device: { // This section is only available on mobile devices.
+          model: <string>, // the "device" from FHR, null on failure
+          manufacturer: <string>, // null on failure
+          hardware: <string>, // null on failure
+          isTablet: <bool>, // null on failure
+        },
+        os: {
+            name: <string>, // "Windows_NT" or null on failure
+            version: <string>, // e.g. "6.1", null on failure
+            kernelVersion: <string>, // android/b2g only or null on failure
+            servicePackMajor: <number>, // windows only or null on failure
+            servicePackMinor: <number>, // windows only or null on failure
+            locale: <string>, // "en" or null on failure
+        },
+        hdd: {
+          profile: { // hdd where the profile folder is located
+              model: <string>, // null on failure
+              revision: <string>, // null on failure
+          },
+          binary:  { // hdd where the application binary is located
+              model: <string>, // null on failure
+              revision: <string>, // null on failure
+          },
+          system:  { // hdd where the system files are located
+              model: <string>, // null on failure
+              revision: <string>, // null on failure
+          },
+        },
+        gfx: {
+            D2DEnabled: <bool>, // null on failure
+            DWriteEnabled: <bool>, // null on failure
+            DWriteVersion: <string>, // null on failure
+            adapters: [
+              {
+                description: <string>, // e.g. "Intel(R) HD Graphics 4600", null on failure
+                vendorID: <string>, // null on failure
+                deviceID: <string>, // null on failure
+                subsysID: <string>, // null on failure
+                RAM: <number>, // in MB, null on failure
+                driver: <string>, // null on failure
+                driverVersion: <string>, // null on failure
+                driverDate: <string>, // null on failure
+                GPUActive: <bool>, // currently always true for the first adapter
+              },
+              ...
+            ],
+          },
+      },
+      addons: {
+        activeAddons: { // the currently enabled addons
+          <addon id>: {
+            blocklisted: <bool>,
+            description: <string>,
+            name: <string>,
+            userDisabled: <bool>,
+            appDisabled: <bool>,
+            version: <string>,
+            scope: <integer>,
+            type: <string>, // "extension", "service", ...
+            foreignInstall: <bool>,
+            hasBinaryComponents: <bool>
+            installDay: <number>, // days since UNIX epoch
+            updateDay: <number>, // days since UNIX epoch
+          },
+          ...
+        },
+        theme: { // the active theme
+          id: <string>,
+          blocklisted: <bool>,
+          description: <string>,
+          name: <string>,
+          userDisabled: <bool>,
+          appDisabled: <bool>,
+          version: <string>,
+          scope: <integer>,
+          foreignInstall: <bool>,
+          hasBinaryComponents: <bool>
+          installDay: <number>, // days since UNIX epoch
+          updateDay: <number>, // days since UNIX epoch
+        },
+        activePlugins: [
+          {
+            name: <string>,
+            version: <string>,
+            description: <string>,
+            blocklisted: <bool>,
+            disabled: <bool>,
+            clicktoplay: <bool>,
+            mimeTypes: [<string>, ...],
+            updateDay: <number>, // days since UNIX epoch
+          },
+          ...
+        ],
+        activeGMPlugins: {
+            <gmp id>: {
+                version: <string>,
+                userDisabled: <bool>,
+                applyBackgroundUpdates: <bool>,
+            },
+            ...
+        ],
+        activeExperiment: { // section is empty if there's no active experiment
+            id: <string>, // id
+            branch: <string>, // branch name
+        },
+        persona: <string>, // id of the current persona, null on GONK
+      },
+    }
new file mode 100644
--- /dev/null
+++ b/toolkit/components/telemetry/docs/index.rst
@@ -0,0 +1,21 @@
+=========
+Telemetry
+=========
+
+Telemetry is a feature that allows data collection. This is being used to collect performance metrics and other information about how Firefox performs in the wild.
+
+Client-side, this consists of:
+
+* data collection in `Histograms <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe>`_ and other data structures
+* assembling :doc:`pings` with the general information and the data payload
+* sending them to the server and local ping retention
+
+*Note:* the `data collection policy <https://wiki.mozilla.org/Firefox/Data_Collection>`_ documents the process and requirements that are applied here.
+
+.. toctree::
+   :maxdepth: 2
+
+   pings
+   common-ping
+   environment
+   main-ping
new file mode 100644
--- /dev/null
+++ b/toolkit/components/telemetry/docs/main-ping.rst
@@ -0,0 +1,56 @@
+
+"main" ping
+===========
+
+This is the "main" Telemetry ping type, whose payload contains most of the measurements that are used to track the performance and health of Firefox in the wild.
+It includes the histograms and other performance and diagnostic data.
+
+This ping is triggered by different scenarios, which is documented by the ``reason`` field:
+
+* ``environment-change`` - the :doc:`environment` changed, so the session measurements got reset and a new subsession starts
+* ``shutdown`` - triggered when the browser session ends
+* ``daily`` - a session split triggered in 24h hour intervals at local midnight
+* ``saved-session`` - the *"classic"* Telemetry payload with measurements covering the whole browser session (only submitted for a transition period)
+
+Most reasons lead to a session split, initiating a new *subsession*. We reset important measurements for those subsessions.
+
+Structure::
+
+    {
+      version: 4,
+
+      info: {
+        reason: <string>, // what triggered this ping: "saved-session", "environment-change", "shutdown", ...
+        revision: <string>, // the Histograms.json revision
+        timezoneOffset: <number>, // time-zone offset from UTC, in minutes, for the current locale
+        previousBuildId: <string>,
+
+        sessionId: <uuid>,  // random session id, shared by subsessions
+        subsessionId: <uuid>,  // random subsession id
+        previousSubsessionId: <uuid>, // subsession id of the previous subsession (even if it was in a different session),
+                                      // null on first run.
+
+        subsessionCounter: <number>, // the running no. of this subsession since the start of the browser session
+        profileSubsessionCounter: <number>, // the running no. of all subsessions for the whole profile life time
+
+        sessionStartDate: <ISO date>, // daily precision
+        subsessionStartDate: <ISO date>, // daily precision
+        subsessionLength: <number>, // the subsession length in seconds
+      },
+
+      childPayloads: {...}, // only present with e10s; a reduced payload from content processes
+
+      simpleMeasurements: { ... },
+      histograms: {},
+      keyedHistograms: {},
+      chromeHangs: {},
+      threadHangStats: {},
+      log: [],
+      fileIOReports: {...},
+      lateWrites: {...},
+      addonDetails: { ... },
+      addonHistograms: {...},
+      UIMeasurements: {...},
+      slowSQL: {...},
+      slowSQLstartup: {...},
+    }
new file mode 100644
--- /dev/null
+++ b/toolkit/components/telemetry/docs/pings.rst
@@ -0,0 +1,28 @@
+.. _telemetry_pings:
+
+=====================
+Telemetry pings
+=====================
+
+A *Telemetry ping* is the data that we send to Mozillas Telemetry servers.
+
+That data is stored as a JSON object client-side and contains common information to all pings and a payload specific to a certain *ping types*.
+
+The top-level structure is defined by the :doc:`common-ping` format.
+It contains some basic information shared between different ping types, the :doc:`environment` data (optional) and the data specific to the *ping type*, the *payload*.
+
+Submission
+==========
+
+Pings are submitted via a common API on ``TelemetryPing``. It allows callers to choose a custom retention period that determines how long pings are kept on disk if submission wasn't successful.
+If a ping failed to submit (e.g. because of missing internet connection), Telemetry will retry to submit it until its retention period is up.
+
+*Note:* the :doc:`main pings <main-ping>` are kept locally even after successful submission to enable the HealthReport and SelfSupport features. They will be deleted after their retention period of 180 days.
+
+Ping types
+==========
+
+* :doc:`main <main-ping>` - contains the information collected by Telemetry (Histograms, hang stacks, ...)
+* ``activation`` - *planned* - sent right after installation or profile creation
+* ``upgrade`` - *planned* - sent right after an upgrade
+* ``deletion`` - *planned* - on opt-out we may have to tell the server to delete user data
--- a/toolkit/components/telemetry/moz.build
+++ b/toolkit/components/telemetry/moz.build
@@ -55,8 +55,9 @@ GENERATED_FILES = [
 if CONFIG['MOZILLA_OFFICIAL']:
     DEFINES['MOZILLA_OFFICIAL'] = True
 
 LOCAL_INCLUDES += [
     '/xpcom/build',
     '/xpcom/threads',
 ]
 
+SPHINX_TREES['telemetry'] = 'docs'