taskcluster/taskgraph/parameters.py
author Gregory Szorc <gps@mozilla.com>
Thu, 18 Aug 2016 08:58:59 -0700
changeset 413298 84dfa535f53ae5ec7a03a318e945684a2f8ca519
parent 402867 0944984daec9a82ee570b0edf02f940cd7c6a087
permissions -rw-r--r--
Bug 1295486 - Decode YAML files to UTF-8 at read time; r=dustin Before, we'd open files and feed bytes to yaml.load(). When a str is fed to yaml.load(), it attempts to guess the encoding. It defaults to UTF-8 unless somebody set us up the BOM. This is probably OK. Except if the file isn't valid UTF-8, the exception will be raised in the bowels of YAML parsing and it may not be obvious the failure is due to invalid UTF-8 input versus say Python str/unicode coercion foo. We change all call sites that load YAML from a file to use codecs.open() to open the file in UTF-8 and perform UTF-8 decoding/validation at file read time. This should make any UTF-8 failures more obvious. Furthermore, it reinforces that our YAML files are UTF-8 and not some other encoding. I discovered this issue as part of trying to get emoji symbols to render on Treeherder. Unfortunately, it appears pyyaml detects many emoji as unprintable characters and refuses to load them. This makes me sad and makes me want to abandon pyyaml/YAML in favor of something that supports emoji :P MozReview-Commit-ID: AOvAruZFfnK

# -*- coding: utf-8 -*-

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

from __future__ import absolute_import, print_function, unicode_literals

import codecs
import json
import yaml
from mozbuild.util import ReadOnlyDict


class Parameters(ReadOnlyDict):
    """An immutable dictionary with nicer KeyError messages on failure"""
    def __getitem__(self, k):
        try:
            return super(Parameters, self).__getitem__(k)
        except KeyError:
            raise KeyError("taskgraph parameter {!r} not found".format(k))


def load_parameters_file(options):
    """
    Load parameters from the --parameters option
    """
    filename = options['parameters']
    if not filename:
        return Parameters()
    with codecs.open(filename, 'rb', 'utf-8') as f:
        if filename.endswith('.yml'):
            return Parameters(**yaml.safe_load(f))
        elif filename.endswith('.json'):
            return Parameters(**json.load(f))
        else:
            raise TypeError("Parameters file `{}` is not JSON or YAML".format(filename))