c7df9aec9234933051d83b94840c8ba259e72fb7: Bug 1391476 - Tell run-task about volumes so it can sanitize them; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 16:48:54 -0700 - rev 650873
Push 75523 by gszorc@mozilla.com at Wed, 23 Aug 2017 00:42:07 +0000
Bug 1391476 - Tell run-task about volumes so it can sanitize them; r?dustin We recently introduced support for telling run-task about caches so it could sanitize them automatically. We also recently taught docker-worker and docker-engine how to declare volumes. Building on that work, we now pass a list of paths corresponding to Docker volumes to run-task. run-task now verifies volumes behave as expected. Unless the volume paths correspond to caches, run-task verifies they are empty and chowns them to an appropriate owner. MozReview-Commit-ID: 5lm2uIitrS3
a3010c14c6d45c6bd3765fd574c3900593f160b1: Bug 1391476 - Require that all cache paths be declared as volumes; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 17:40:20 -0700 - rev 650872
Push 75523 by gszorc@mozilla.com at Wed, 23 Aug 2017 00:42:07 +0000
Bug 1391476 - Require that all cache paths be declared as volumes; r?dustin See the inline comment for the rationale here. Depending on the order transforms run in, it is possible to evade this check. But I think it is good enough. Plus subsequent commits will essentially make tasks without properly declared caches and volumes impossible to run without permissions failures. Several Dockerfile have been updated to add missing VOLUME so the check passes. In the case of desktop1604-test, we stopped removing /home/worker/.cache because you can't remove a mount point, which is what volumes are inside Docker containers. MozReview-Commit-ID: GEyNkkX00kN
186637c333da09e2619338d0ad3135418694d50d: Bug 1390203: Only get highlight images on wifi. r=sebastian draft
Michael Comella <michael.l.comella@gmail.com> - Tue, 22 Aug 2017 10:59:12 -0700 - rev 650871
Push 75522 by michael.l.comella@gmail.com at Wed, 23 Aug 2017 00:30:00 +0000
Bug 1390203: Only get highlight images on wifi. r=sebastian MozReview-Commit-ID: FtSDZ6Yu5Ql
4a42a080b37104fb49f1f1cfbec884edbab52ba2: Bug 1390203: Add NetworkUtils.isWifi. r=sebastian draft
Michael Comella <michael.l.comella@gmail.com> - Tue, 22 Aug 2017 17:27:19 -0700 - rev 650870
Push 75522 by michael.l.comella@gmail.com at Wed, 23 Aug 2017 00:30:00 +0000
Bug 1390203: Add NetworkUtils.isWifi. r=sebastian It doesn't appear we simply check for a wifi connection type anywhere so the existing code shouldn't need to be updated. MozReview-Commit-ID: 7MqIbdB7uRI
91e4428a206fed543f629acbf4d08848f7d48d2c: Bug 1390352 - Skip adding/accumulating the values which corresponding rect offset is auto. r?manishearth,birtles draft
Mantaroh Yoshinaga <mantaroh@gmail.com> - Wed, 23 Aug 2017 09:27:42 +0900 - rev 650869
Push 75521 by bmo:mantaroh@gmail.com at Wed, 23 Aug 2017 00:28:11 +0000
Bug 1390352 - Skip adding/accumulating the values which corresponding rect offset is auto. r?manishearth,birtles This patch will skip add_weighted() when self_portion + other_portion not equal to zero. (i.e. adding or accumulating). Gecko does same behavior since we can't add two auto value. MozReview-Commit-ID: 7vzF548nQuD
237aedb4121a78ea379fa3d42427557c3f74357a: Bug 1389968 - Reject sendMessage() promise when response handle gets GCd draft
Tomislav Jovanovic <tomica@gmail.com> - Wed, 23 Aug 2017 00:16:48 +0200 - rev 650868
Push 75520 by bmo:tomica@gmail.com at Wed, 23 Aug 2017 00:20:18 +0000
Bug 1389968 - Reject sendMessage() promise when response handle gets GCd MozReview-Commit-ID: C2g3VSWYKuz
29850668a57d7611542e310bd3d7e7c4a35cb4e3: Bug 1390352 - Skip adding/accumulating the values which corresponding rect offset is auto. r?manishearth,birtles draft
Mantaroh Yoshinaga <mantaroh@gmail.com> - Wed, 23 Aug 2017 09:15:59 +0900 - rev 650867
Push 75519 by bmo:mantaroh@gmail.com at Wed, 23 Aug 2017 00:16:28 +0000
Bug 1390352 - Skip adding/accumulating the values which corresponding rect offset is auto. r?manishearth,birtles This patch will skip add_weighted() when self_portion + other_portion not equal to zero. (i.e. adding or accumulating). Gecko does same behavior since we can't add two auto value. MozReview-Commit-ID: 7vzF548nQuD
e69755fd0eca177380830d193e644a499ba4ea96: Bug 1390352 - Skip adding/accumulating the values which corresponding rect offset is auto. r?manishearth,birtles draft
Mantaroh Yoshinaga <mantaroh@gmail.com> - Wed, 23 Aug 2017 09:08:29 +0900 - rev 650866
Push 75518 by bmo:mantaroh@gmail.com at Wed, 23 Aug 2017 00:09:01 +0000
Bug 1390352 - Skip adding/accumulating the values which corresponding rect offset is auto. r?manishearth,birtles This patch will skip add_weighted() when self_portion + other_portion not equal to zero. (i.e. adding or accumulating). Gecko does same behavior since we can't add two auto value. MozReview-Commit-ID: 7vzF548nQuD
ec808fe61a74599e89ca63bb680cc9a5489b5ee8: Bug 1390352 - Make Servo_AnimationValues_ComputeDistance return negative value instead of 0.0 when the function fails to distinguish its failure. r=hiro draft
Mantaroh Yoshinaga <mantaroh@gmail.com> - Wed, 23 Aug 2017 09:07:07 +0900 - rev 650865
Push 75518 by bmo:mantaroh@gmail.com at Wed, 23 Aug 2017 00:09:01 +0000
Bug 1390352 - Make Servo_AnimationValues_ComputeDistance return negative value instead of 0.0 when the function fails to distinguish its failure. r=hiro We need to check whether the function fails or not in order to check whether we support the specified paced animation values. Current servo returns 0.0 when failed computing distance, so servo doesn't distinguish its failure. This patch makes Servo_AnimationValue_ComputeDistance return a negative value when the function fails. MozReview-Commit-ID: 43Q4gu4xwHc
42610adf13d99029de3a81c54d96e67204c6e871: Bug 1391476 - Add UID and GID to cache parameters; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 16:49:26 -0700 - rev 650864
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Add UID and GID to cache parameters; r?dustin The UID and GID that a task executes under is dynamic. As a result, caches need to be aware of the UID and GID that owns files otherwise subsequent tasks could run into permission denied errors. This is why `run-task --chown-recursive` exists. By recursively changing ownership of persisted files, we ensure the current task is able to read and write all existing files. When you take a step back, you realize that chowning of cached files is an expensive workaround. Yes, this results in cache hits. But the cost is you potentially have to perform hundreds of thousands of I/O system calls to mass chown. The ideal situation is that UID/GID is consistent across tasks on any given cache and potentially expensive permissions setting can be avoided. So, that's what this commit does. We add the task's UID and GID to run-task's requirements. When we first see a cache, we record a UID and GID with it and chown the empty cache directory to that UID and GID. Subsequent tasks using this cache *must* use the same UID and GID or else run-task will fail. Since run-task now guarantees that all cache consumers use the same UID and GID, we can avoid a potentially expensive recursive chown. But there is an exception. In untrusted environments (namely Try), we recursively chown existing caches if there is a uid/gid mismatch. We do this because Try is a sandbox and any random task could experiment with a non-standard uid/gid. That populated cache would "poison" the cache for the next caller. Or vice-versa. It would be annoying if caches were randomly poisoned due to Try pushes that didn't realize there was a UID/GID mismatch. We could outlaw "bad" UID and GIDs. But that makes the barrier to testing things on Try harder. So, we go with the flow and recursively chown caches in this scenario. This change will shine light on all tasks using inconsistent UID and GID values on the same cache. Bustage is anticipated. Unfortunately, we can't easily know what will break. So it will be one of those things where we will have to fix problems as they arise. Fortunately, because caches are now tied to the content of run-task, we only need to back out this change and tasks should revert to caches without UID and GID pinning requirements and everything will work again. MozReview-Commit-ID: 2ka4rOnnXIp
2e0720c9dee6a263d5610a0d88282181047f9dec: Bug 1391476 - Automatically set cache/volume permissions in run-task; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 16:26:31 -0700 - rev 650863
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Automatically set cache/volume permissions in run-task; r?dustin run-task's --chown and --chown-recursive are only used on volumes and caches - the only locations that aren't controlled by the Docker image itself and thus whose permissions could be "undefined." Previous commits have taught run-task about the locations of all caches and volumes. Therefore, we no longer need to manually define paths to chown. Instead, we can chown as a side-effect of the path being a cache or a volume. So, this commit changes run-task to chown caches and volumes automatically. Since we no longer have a use for --chown and --chown-recursive, those arguments are removed. There /could/ be some paths that are caches or volumes but aren't getting defined as such in Taskgraph. I consider this a bug in Taskgraph and the recourse is to properly define a path as a cache or a volume there. MozReview-Commit-ID: 1yqrhjil6gy
fa2ea5cda554288c103fda5f8221b2d2cd9070d2: Bug 1391476 - Tell run-task about volumes so it can sanitize them; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 16:48:54 -0700 - rev 650862
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Tell run-task about volumes so it can sanitize them; r?dustin We recently introduced support for telling run-task about caches so it could sanitize them automatically. We also recently taught docker-worker and docker-engine how to declare volumes. Building on that work, we now pass a list of paths corresponding to Docker volumes to run-task. run-task now verifies volumes behave as expected. Unless the volume paths correspond to caches, run-task verifies they are empty and chowns them to an appropriate owner. MozReview-Commit-ID: 5lm2uIitrS3
992be1ba100bb88d9165387160053a0d3fc2bd34: Bug 1391476 - Require that all cache paths be declared as volumes; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 15:54:27 -0700 - rev 650861
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Require that all cache paths be declared as volumes; r?dustin See the inline comment for the rationale here. Depending on the order transforms run in, it is possible to evade this check. But I think it is comprehensive enough. Several Dockerfile have been updated to add missing VOLUME so the check passes. MozReview-Commit-ID: GEyNkkX00kN
6a1c5d468b86b13d1f286d3263c10849f3da9cd2: Bug 1391476 - Capture Docker volumes in docker-worker config; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Mon, 21 Aug 2017 16:57:39 -0700 - rev 650860
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Capture Docker volumes in docker-worker config; r?dustin Docker volumes are host-mounted filesystems. We typically mount caches at their location. But not always. The reason we define VOLUME in Dockerfiles is we're guaranteed to get a fast host filesystem instead of AUFS when a cache isn't mounted. In this commit, we teach the docker-worker payload builder about the existence of Docker volumes. Docker volumes can be declared inline in the YAML. More conveniently, we automatically parse out VOLUME lines from corresponding in-tree Dockerfile. We'll do useful things with this data in subsequent commits. MozReview-Commit-ID: BNxp8EDEYw
b5e409b963dd3583ddc6850ba2b83f7748dcf160: Bug 1391476 - Track whether caches should be used in untrusted environments; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 15:06:40 -0700 - rev 650859
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Track whether caches should be used in untrusted environments; r?dustin Previously, we conditionally added caches to a task if the current parameters warranted it. In order to audit that all caches fulfill basic requirements, we need to have unconditional knowledge of all caches. This commit introduces an optional key on each cache entry stating whether it should be skipped in "untrusted" environments. When we convert a task definition to a worker payload, we filter out these caches if necessary. This change uncovered an inconsistency with filtering caches. In one location we filtered on the source repo name. In others, we filtered on the SCM level. Setting the caches in the spidermonkey kind also changed slightly to ensure we're not overwriting existing caches. I don't think this has any behavior changes. But the new method is more correct. MozReview-Commit-ID: 1crpdWHqQ68
d8c87b657af6be58f184276ce56f5631c11c31b8: Bug 1391476 - Print cache info; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 16:48:03 -0700 - rev 650858
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Print cache info; r?dustin For forensic purposes. MozReview-Commit-ID: 6pcOm90cPdw
bb0dcc1a4a4e5d774384d1a93c3e6bf931c090be: Bug 1391476 - Don't use ~ in paths; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Mon, 21 Aug 2017 17:11:49 -0700 - rev 650857
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Don't use ~ in paths; r?dustin ~ in paths is mostly a shell-ism. Let's not use it. The real reason for this is it interfes with upcoming commits that audit cache and volume paths. MozReview-Commit-ID: AhjMwg5gexx
562ba4423c782a4794ad304caa7b6406f4a82cab: Bug 1391476 - Dedent block; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Tue, 22 Aug 2017 12:40:44 -0700 - rev 650856
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Dedent block; r?dustin MozReview-Commit-ID: 1l8gWfBNBMs
e584940b5db216b92feb76b32f02ce6a288acfca: Bug 1391476 - Inline list append; r?dustin draft
Gregory Szorc <gps@mozilla.com> - Mon, 21 Aug 2017 10:55:10 -0700 - rev 650855
Push 75517 by gszorc@mozilla.com at Tue, 22 Aug 2017 23:53:07 +0000
Bug 1391476 - Inline list append; r?dustin Static analysis in my editor was complaining about the old pattern. Why not fix it while I'm here. MozReview-Commit-ID: HtrGenolNXb
4afd9c9fe2fa0be3ec7f1e8bf5bb757de7fac7b9: Bug 1379265 - Write C++ bindings to rsdparsa and integrate into existing SDP code draft
Paul Ellenbogen <pe5@cs.princeton.edu> - Fri, 30 Jun 2017 12:54:12 -0700 - rev 650854
Push 75516 by bmo:pe5@cs.princeton.edu at Tue, 22 Aug 2017 23:47:47 +0000
Bug 1379265 - Write C++ bindings to rsdparsa and integrate into existing SDP code MozReview-Commit-ID: FdhpTT5wzwI
(0) -300000 -100000 -30000 -10000 -3000 -1000 -300 -100 -50 -20 +20 +50 +100 +300 +1000 +3000 +10000 +30000 +100000 tip