taskcluster/taskgraph/parameters.py
author Gregory Szorc <gps@mozilla.com>
Thu, 18 Aug 2016 08:58:59 -0700
changeset 335706 7a3a65ff58653cfa96f39c2ab4e458111602b92c
parent 327283 a1748bc8704ad72414ab3fbb7d93e7d0cf00c042
child 335715 0944984daec9a82ee570b0edf02f940cd7c6a087
permissions -rw-r--r--
Bug 1295486 - Decode YAML files to UTF-8 at read time; r=dustin Before, we'd open files and feed bytes to yaml.load(). When a str is fed to yaml.load(), it attempts to guess the encoding. It defaults to UTF-8 unless somebody set us up the BOM. This is probably OK. Except if the file isn't valid UTF-8, the exception will be raised in the bowels of YAML parsing and it may not be obvious the failure is due to invalid UTF-8 input versus say Python str/unicode coercion foo. We change all call sites that load YAML from a file to use codecs.open() to open the file in UTF-8 and perform UTF-8 decoding/validation at file read time. This should make any UTF-8 failures more obvious. Furthermore, it reinforces that our YAML files are UTF-8 and not some other encoding. I discovered this issue as part of trying to get emoji symbols to render on Treeherder. Unfortunately, it appears pyyaml detects many emoji as unprintable characters and refuses to load them. This makes me sad and makes me want to abandon pyyaml/YAML in favor of something that supports emoji :P MozReview-Commit-ID: AOvAruZFfnK

# -*- coding: utf-8 -*-

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

from __future__ import absolute_import, print_function, unicode_literals

import codecs
import json
import yaml
from mozbuild.util import ReadOnlyDict


class Parameters(ReadOnlyDict):
    """An immutable dictionary with nicer KeyError messages on failure"""
    def __getitem__(self, k):
        try:
            return super(Parameters, self).__getitem__(k)
        except KeyError:
            raise KeyError("taskgraph parameter {!r} not found".format(k))


def load_parameters_file(options):
    """
    Load parameters from the --parameters option
    """
    filename = options['parameters']
    if not filename:
        return Parameters()
    with codecs.open(filename, 'rb', 'utf-8') as f:
        if filename.endswith('.yml'):
            return Parameters(**yaml.safe_load(f))
        elif filename.endswith('.json'):
            return Parameters(**json.load(f))
        else:
            raise TypeError("Parameters file `{}` is not JSON or YAML".format(filename))