Bug 939866 - Add Services docs to Sphinx; r=rnewman
authorGregory Szorc <gps@mozilla.com>
Sun, 17 Nov 2013 19:39:55 -0600
changeset 160091 95c9865ac36688fb99013c8b0adf5d38e3d8612e
parent 160090 0eec37596a6d1c0a0ff1077d518cd1df3e0e0c26
child 160092 9b0d76fafd52668fd5f58ebf784eda656081eb26
push id25821
push usercbook@mozilla.com
push dateThu, 12 Dec 2013 11:53:39 +0000
treeherdermozilla-central@d15ed5648a5f [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersrnewman
bugs939866
milestone29.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 939866 - Add Services docs to Sphinx; r=rnewman DONTBUILD (NPOTB)
services/datareporting/README.rst
services/docs/datareporting.rst
services/docs/healthreport.rst
services/docs/index.rst
services/docs/metrics.rst
services/healthreport/README.rst
services/metrics/README.rst
services/moz.build
deleted file mode 100644
--- a/services/datareporting/README.rst
+++ /dev/null
@@ -1,28 +0,0 @@
-======================
-Data Reporting Service
-======================
-
-This directory contains files related to data collection and reporting
-within Gecko applications.
-
-The important files in this directory are:
-
-DataReportingService.js
-  An XPCOM service that coordinates collection and reporting of data.
-
-policy.jsm
-  A module containing the logic for coordinating and driving collection
-  and upload of data.
-
-sessions.jsm
-  Records Gecko application session history. This is loaded as part of
-  the XPCOM service because it needs to capture state from very early in
-  the application lifecycle. Bug 841561 tracks implementing this in C++.
-
-
-There is other code in the tree that collects and uploads data. The
-original intent of this directory and XPCOM service was to serve as a
-focal point for the coordination of all this activity so that it could
-all be done consistently and properly. This vision may or may not be fully
-realized.
-
new file mode 100644
--- /dev/null
+++ b/services/docs/datareporting.rst
@@ -0,0 +1,28 @@
+.. _data_reporting_service:
+
+======================
+Data Reporting Service
+======================
+
+``/services/datareporting`` contains files related to an XPCOM service
+that collects and reports data within Gecko applications.
+
+The important files in this directory are:
+
+DataReportingService.js
+   An XPCOM service that coordinates collection and reporting of data.
+
+policy.jsm
+   A module containing the logic for coordinating and driving collection
+   and upload of data.
+
+sessions.jsm
+   Records Gecko application session history. This is loaded as part of
+   the XPCOM service because it needs to capture state from very early in
+   the application lifecycle. Bug 841561 tracks implementing this in C++.
+
+There is other code in the tree that collects and uploads data. The
+original intent of this directory and XPCOM service was to serve as a
+focal point for the coordination of all this activity so that it could
+all be done consistently and properly. This vision may or may not be fully
+realized.
new file mode 100644
--- /dev/null
+++ b/services/docs/healthreport.rst
@@ -0,0 +1,250 @@
+.. _firefox_health_report:
+
+=====================
+Firefox Health Report
+=====================
+
+``/services/healthreport`` contains the implementation of the
+``Firefox Health Report`` (FHR).
+
+Firefox Health Report is a background service that collects application
+metrics and periodically submits them to a central server. The core
+parts of the service are implemented in this directory. However, the
+actual XPCOM service is implemented in the
+:ref:`data_reporting_service`.
+
+The core types can actually be instantiated multiple times and used to
+power multiple data submission services within a single Gecko
+application. In other words, everything in this directory is effectively
+a reusable library. However, the terminology and some of the features
+are very specific to what the Firefox Health Report feature requires.
+
+Architecture
+============
+
+``healthreporter.jsm`` contains the main interface for FHR, the
+``HealthReporter`` type. An instance of this is created by the
+:ref:`data_reporting_service`.
+
+``providers.jsm`` contains numerous ``Metrics.Provider`` and
+``Metrics.Measurement`` used for collecting application metrics. If you
+are looking for the FHR probes, this is where they are.
+
+Storage
+-------
+
+Firefox Health Report stores data in 3 locations:
+
+* Metrics measurements and provider state is stored in a SQLite database
+  (via ``Metrics.Storage``).
+* Service state (such as the IDs of documents uploaded) is stored in a
+  JSON file on disk (via OS.File).
+* Lesser state and run-time options are stored in preferences.
+
+Preferences
+-----------
+
+Preferences controlling behavior of Firefox Health Report live in the
+``datareporting.healthreport.*`` branch.
+
+Service and Data Control
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The follow preferences control behavior of the service and data upload.
+
+service.enabled
+   Controls whether the entire health report service runs. The overall
+   service performs data collection, storing, and submission.
+
+   This is the primary kill switch for Firefox Health Report
+   outside of the build system variable. i.e. if you are using an
+   official Firefox build and wish to disable FHR, this is what you
+   should set to false to prevent FHR from not only submitting but
+   also collecting data.
+
+uploadEnabled
+   Whether uploading of data is enabled. This is the preference the
+   checkbox in the preferences UI reflects. If this is
+   disabled, FHR still collects data - it just doesn't upload it.
+
+service.loadDelayMsec
+   How long (in milliseconds) after initial application start should FHR
+   wait before initializing.
+
+   FHR may initialize sooner than this if the FHR service is requested.
+   This will happen if e.g. the user goes to ``about:healthreport``.
+
+service.loadDelayFirstRunMsec
+   How long (in milliseconds) FHR should wait to initialize on first
+   application run.
+
+   FHR waits longer than normal to initialize on first application run
+   because first-time initialization can use a lot of I/O to initialize
+   the SQLite database and this I/O should not interfere with the
+   first-run user experience.
+
+documentServerURI
+   The URI of a Bagheera server that FHR should interface with for
+   submitting documents.
+
+   You typically do not need to change this.
+
+documentServerNamespace
+   The namespace on the document server FHR should upload documents to.
+
+   You typically do not need to change this.
+
+infoURL
+   The URL of a page containing more info about FHR, it's privacy
+   policy, etc.
+
+about.reportUrl
+   The URL to load in ``about:healthreport``.
+
+service.providerCategories
+   A comma-delimited list of category manager categories that contain
+   registered ``Metrics.Provider`` records. Read below for how provider
+   registration works.
+
+If the entire service is disabled, you lose data collection. This means
+that **local** data analysis won't be available because there is no data
+to analyze! Keep in mind that Firefox Health Report can be useful even
+if it's not submitting data to remote servers!
+
+Logging
+^^^^^^^
+
+The following preferences allow you to control the logging behavior of
+Firefox Health Report.
+
+logging.consoleEnabled
+   Whether to write log messages to the web console. This is true by
+   default.
+
+logging.consoleLevel
+   The minimum log level FHR messages must have to be written to the
+   web console. By default, only FHR warnings or errors will be written
+   to the web console. During normal/expected operation, no messages of
+   this type should be produced.
+
+logging.dumpEnabled
+   Whether to write log messages via ``dump()``. If true, FHR will write
+   messages to stdout/stderr.
+
+   This is typically only enabled when developing FHR.
+
+logging.dumpLevel
+   The minimum log level messages must have to be written via
+   ``dump()``.
+
+State
+^^^^^
+
+currentDaySubmissionFailureCount
+   How many submission failures the client has encountered while
+   attempting to upload the most recent document.
+
+lastDataSubmissionFailureTime
+   The time of the last failed document upload.
+
+lastDataSubmissionRequestedTime
+   The time of the last document upload attempt.
+
+lastDataSubmissionSuccessfulTime
+   The time of the last successful document upload.
+
+nextDataSubmissionTime
+   The time the next data submission is scheduled for. FHR will not
+   attempt to upload a new document before this time.
+
+pendingDeleteRemoteData
+   Whether the client currently has a pending request to delete remote
+   data. If true, the client will attempt to delete all remote data
+   before an upload is performed.
+
+FHR stores various state in preferences.
+
+Registering Providers
+=====================
+
+Firefox Health Report providers are registered via the category manager.
+See ``HealthReportComponents.manifest`` for providers defined in this
+directory.
+
+Essentially, the category manager receives the name of a JS type and the
+URI of a JSM to import that exports this symbol. At run-time, the
+providers registered in the category manager are instantiated.
+
+Providers are registered via the category manager to make registration
+simple and less prone to errors. Any XPCOM component can create a
+category manager entry. Therefore, new data providers can be added
+without having to touch core Firefox Health Report code. Additionally,
+category manager registration means providers are more likely to be
+registered on FHR's terms, when it wants. If providers were registered
+in code at application run-time, there would be the risk of other
+components prematurely instantiating FHR (causing a performance hit if
+performed at an inopportune time) or semi-complicated code around
+observers or listeners. Category manager entries are only 1 line per
+provider and leave FHR in control: they are simple and safe.
+
+Document Generation and Lifecycle
+=================================
+
+FHR will attempt to submit a JSON document containing data every 24 wall
+clock hours.
+
+At upload time, FHR will query the database for **all** information from
+the last 180 days and assemble this data into a JSON document. We
+attempt to upload this JSON document with a client-generated UUID to the
+configured server.
+
+Before we attempt upload, the generated UUID is stored in the JSON state
+file on local disk. At this point, the client assumes the document with
+that UUID has been successfully stored on the server.
+
+If the client is aware of other document UUIDs that presumably exist on
+the server, those UUIDs are sent with the upload request so the client
+can request those UUIDs be deleted. This helps ensure that each client
+only has 1 document/UUID on the server at any one time.
+
+Importance of Persisting UUIDs
+------------------------------
+
+The choices of how, where, and when document UUIDs are stored and updated
+are very important. One should not attempt to change things unless she
+has a very detailed understanding of why things are the way they are.
+
+The client is purposefully very conservative about forgetting about
+generated UUIDs. In other words, once a UUID is generated, the client
+deliberately holds on to that UUID until it's very confident that UUID
+is no longer stored on the server. The reason we do this is because
+*orphaned* documents/UUIDs on the server can lead to faulty analysis,
+such as over-reporting the number of Firefox installs that stop being
+used.
+
+When uploading a new UUID, we update the state and save the state file
+to disk *before* an upload attempt because if the upload succeeds but
+the response never makes it back to the client, we want the client to
+know about the uploaded UUID so it can delete it later to prevent an
+orphan.
+
+We maintain a list of UUIDs locally (not simply the last UUID) because
+multiple upload attempts could fail the same way as the previous
+paragraph describes and we have no way of knowing which (if any)
+actually succeeded. The safest approach is to assume every document
+produced managed to get uploaded some how.
+
+We store the UUIDs on a file on disk and not anywhere else because we
+want storage to be robust. We originally stored UUIDs in preferences,
+which only flush to disk periodically. Writes to preferences were
+apparently getting lost. We switched to writing directly to files to
+eliminate this window.
+
+Legal and Privacy Concerns
+==========================
+
+Because Firefox Health Report collects and submits data to remote
+servers and is an opt-out feature, there are legal and privacy
+concerns over what data may be collected and submitted. **Additions or
+changes to submitted data should be signed off by responsible
+parties.**
new file mode 100644
--- /dev/null
+++ b/services/docs/index.rst
@@ -0,0 +1,18 @@
+=======================
+Firefox Services Module
+=======================
+
+The ``/services`` directory contains code for a variety of application
+features that communicate with external services - hence its name.
+
+It was originally created to hold code for Firefox Sync. Later, it
+became the location for code written by the Mozilla Services Client team
+and thus includes Firefox Health Report. This team no longer exists, but
+the directory remains.
+
+.. toctree::
+   :maxdepth: 1
+
+   metrics
+   healthreport
+   datareporting
new file mode 100644
--- /dev/null
+++ b/services/docs/metrics.rst
@@ -0,0 +1,130 @@
+.. _services_metrics:
+
+============================
+Metrics Collection Framework
+============================
+
+The ``services/metrics`` directory contains a generic data metrics
+collecting and persisting framework for Gecko applications.
+
+Overview
+========
+
+The Metrics framework by itself doesn't do much: it simply provides a
+generic mechanism for collecting and persisting data. It is up to users
+of this framework to drive collection and do something with the obtained
+data. A consumer of this framework is :ref:`firefox_health_report`.
+
+Relationship to Telemetry
+-------------------------
+
+Telemetry provides similar features to code in this directory. The two
+may be unified in the future.
+
+Usage
+=====
+
+To use the code in this directory, import Metrics.jsm. e.g.
+
+   Components.utils.import("resource://gre/modules/Metrics.jsm");
+
+This exports a *Metrics* object which holds references to the main JS
+types and functions provided by this feature. Read below for what those
+types are.
+
+Metrics Types
+=============
+
+``Metrics.jsm`` exports a number of types. They are documented in the
+sections below.
+
+Metrics.Provider
+----------------
+
+``Metrics.Provider`` is an entity that collects and manages data. Providers
+are typically domain-specific: if you need to collect a new type of data,
+you create a ``Metrics.Provider`` type that does this.
+
+Metrics.Measurement
+-------------------
+
+A ``Metrics.Measurement`` represents a collection of related pieces/fields
+of data.
+
+All data recorded by the metrics framework is modeled as
+``Metrics.Measurement`` instances. Instances of ``Metrics.Measurement``
+are essentially data structure descriptors.
+
+Each ``Metrics.Measurement`` consists of a name and version to identify
+itself (and its data) as well as a list of *fields* that this measurement
+holds. A *field* is effectively an entry in a data structure. It consists
+of a name and strongly enumerated type.
+
+Metrics.Storage
+---------------
+
+This entity is responsible for persisting collected data and state.
+
+It currently uses SQLite to store data, but this detail is abstracted away
+in order to facilitate swapping of storage backends.
+
+Metrics.ProviderManager
+-----------------------
+
+High-level entity coordinating activity among several ``Metrics.Provider``
+instances.
+
+Providers and Measurements
+==========================
+
+The most important types in this framework are ``Metrics.Provider`` and
+``Metrics.Measurement``, henceforth known as ``Provider`` and
+``Measurement``, respectively. As you will see, these two types go
+hand in hand.
+
+A ``Provider`` is an entity that *provides* data about a specific subsystem
+or feature. They do this by recording data to specific ``Measurement``
+types. Both ``Provider`` and ``Measurement`` are abstract base types.
+
+A ``Measurement`` implementation defines a name and version. More
+importantly, it also defines its storage requirements and how
+previously-stored values are serialized.
+
+Storage allocation is performed by communicating with the SQLite
+backend. There is a startup function that tells SQLite what fields the
+measurement is recording. The storage backend then registers these in
+the database. Internally, this is creating a new primary key for
+individual fields so later storage operations can directly reference
+these primary keys in order to retrieve data without having to perform
+complicated joins.
+
+A ``Provider`` can be thought of as a collection of ``Measurement``
+implementations. e.g. an Addons provider may consist of a measurement
+for all *current* add-ons as well as a separate measurement for
+historical counts of add-ons. A provider's primary role is to take
+metrics data and write it to various measurements. This effectively
+persists the data to SQLite.
+
+Data is emitted from providers in either a push or pull based mechanism.
+In push-based scenarios, the provider likely subscribes to external
+events (e.g. observer notifications). An event of interest can occur at
+any time. When it does, the provider immediately writes the event of
+interest to storage or buffers it for eventual writing. In pull-based
+scenarios, the provider is periodically queried and asked to populate
+data.
+
+SQLite Storage
+==============
+
+``Metrics.Storage`` provides an interface for persisting metrics data to a
+SQLite database.
+
+The storage API organizes values by fields. A field is a named member of
+a ``Measurement`` that has specific type and retention characteristics.
+Some example field types include:
+
+* Last text value
+* Last numeric value for a given day
+* Discrete text values for a given day
+
+See ``storage.jsm`` for more.
deleted file mode 100644
--- a/services/healthreport/README.rst
+++ /dev/null
@@ -1,63 +0,0 @@
-=====================
-Firefox Health Report
-=====================
-
-This directory contains the implementation of the Firefox Health Report
-(FHR).
-
-Firefox Health Report is a background service that collects application
-metrics and periodically submits them to a central server.
-
-Implementation Notes
-====================
-
-healthreporter.jsm contains the main interface for FHR, the
-*HealthReporter* type. An instance of this is created by the data
-reporting service. See /services/datareporting/.
-
-providers.jsm contains numerous Metrics.Provider and Metrics.Measurement
-used for collecting application metrics. If you are looking for the FHR
-probes, this is where they are.
-
-Preferences
-===========
-
-Preferences controlling behavior of Firefox Health Report live in the
-*datareporting.healthreport.* branch.
-
-Some important preferences are:
-
-* **service.enabled** - Controls whether the entire health report
-  service runs. The overall service performs data collection, storing, and
-  submission. This is the primary kill switch for Firefox Health Report
-  outside of the build system variable. i.e. if you are using an
-  official Firefox build and wish to disable FHR, this is what you
-  should change.
-
-* **uploadEnabled** - Whether uploading of data is enabled. This
-  is the preference the checkbox in the UI reflects. If this is
-  disabled, FHR still collects data - it just doesn't upload it.
-
-If the entire service is disabled, you lose data collection. This means that
-data analysis won't be available because there is no data to analyze!
-
-Registering Providers
-=====================
-
-Firefox Health Report providers are registered via the category manager.
-See HealthReportComponents.manifest for providers defined in this
-directory.
-
-Essentially, the category manager receives the name of a JS type and the
-URI of a JSM to import that exports this symbol. At run-time, the
-providers registered in the category manager are instantiated. This
-allows for a loose coupling of providers which in turns makes managing
-the code behind the providers much simpler.
-
-Other Notes
-===========
-
-There are many legal and privacy concerns with this code, especially
-around the data that is submitted. Changes to submitted data should be
-signed off by responsible parties.
-
deleted file mode 100644
--- a/services/metrics/README.rst
+++ /dev/null
@@ -1,109 +0,0 @@
-============================
-Metrics Collection Framework
-============================
-
-This directory contains generic code for collecting and persisting
-metrics data for Gecko applications.
-
-Overview
-========
-
-Metrics by itself doesn't do much. It simply provides a framework for
-collecting data. It is up to users of this framework to hook up the
-individual components into a system that makes sense for their purposes.
-An example consumer of Metrics is Firefox Health Report (see
-/services/healthreport).
-
-Relationship to Telemetry
--------------------------
-
-Telemetry provides similar features to code in this directory. The two
-may be unified in the future.
-
-Type Overview
-=============
-
-This directory defines a number of JavaScript *types*/*classes*:
-
-Metrics.Provider
-  An entity that collects and manages data. A provider is typically
-  domain-specific. e.g. AddonsProvider, SearchesProvider.
-
-Metrics.Measurement
-  Represents a collection of related pieces/fields of data. Instances of
-  these are essentially data structure descriptors.
-
-Metrics.Storage
-  Persistent SQLite-backed store for collected metrics data and state.
-
-Metrics.ProviderManager
-  High-level entity coordinating activity among several Metrics.Provider
-  instances.
-
-SQLite Storage
-==============
-
-*Metrics.Storage* provides an interface for persisting metrics data to a
-SQLite database.
-
-The storage API organizes values by fields. A field is a named member of
-a Measurement that has specific type and retention characteristics. Some
-example field types include:
-
-* Last text value
-* Last numeric value for a given day
-* Discrete text values for a given day
-
-See storage.jsm for more.
-
-While SQLite is used under the hood, this implementation detail is
-hidden from the consumer.
-
-Providers and Measurements
-==========================
-
-The most important types in this framework are *Metrics.Provider* and
-*Metrics.Measurement*, henceforth known as *Provider* and *Measurement*,
-respectively. As you will see, these two types go hand in hand.
-
-A Provider is an entity that *provides* data about a specific subsystem
-or feature. They do this by recording data to specific Measurement
-types. Both Provider and Measurement are abstract base types.
-
-A Measurement implementation defines a name and version. More
-importantly, it also defines its storage requirements and how
-previously-stored values are serialized.
-
-Storage allocation is performed by communicating with the SQLite
-backend. There is a startup function that tells SQLite what fields the
-measurement is recording. The storage backend then registers these in
-the database. Internally, this is creating a new primary key for
-individual fields so later storage operations can directly reference
-these primary keys in order to retrieve data without having to perform
-complicated joins.
-
-A Provider can be thought of as a collection of Measurement
-implementations. e.g. an Addons provider may consist of a measurement
-for all *current* add-ons as well as a separate measurement for
-historical counts of add-ons. A provider's primary role is to take
-metrics data and write it to various measurements. This effectively
-persists the data to SQLite.
-
-Data is emitted from providers in either a push or pull based mechanism.
-In push-based scenarios, the provider likely subscribes to external
-events (e.g. observer notifications). An event of interest can occur at
-any time. When it does, the provider immediately writes the event of
-interest to storage or buffers it for eventual writing. In pull-based
-scenarios, the provider is periodically queried and asked to populate
-data.
-
-Usage
-=====
-
-To use the code in this directory, import Metrics.jsm. e.g.
-
-    Components.utils.import("resource://gre/modules/Metrics.jsm");
-
-This exports a *Metrics* object which holds references to the main JS
-types and functions provided by this feature.
-
--- a/services/moz.build
+++ b/services/moz.build
@@ -19,8 +19,10 @@ if CONFIG['MOZ_WIDGET_TOOLKIT'] != 'andr
     if CONFIG['MOZ_DATA_REPORTING']:
         PARALLEL_DIRS += ['datareporting']
 
 if CONFIG['MOZ_SERVICES_METRICS']:
     PARALLEL_DIRS += ['metrics']
 
 if CONFIG['MOZ_SERVICES_SYNC']:
     PARALLEL_DIRS += ['sync']
+
+SPHINX_TREES['services'] = 'docs'