searching for reviewer(gcp)
76339e786c7cc22d2c21d8f503e3d70df76167f2: Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp
dlee <dlee@mozilla.com> - Tue, 14 May 2019 22:42:28 +0000 - rev 532809
Push 11272 by apavel@mozilla.com at Thu, 16 May 2019 15:28:22 +0000
Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp This patch does the following: 1. Run the same prefixset tests when * browser.safebrowsing.prefixset.max_array_size = 0 * browser.safebrowsing.prefixset.max_array_size = UINT32_MAX This makes sure both of the methods to store prefixset are tested by existing testcases 2. Refine gtest with test fixture 3. Add TinySet and LargeSet testcases Differential Revision: https://phabricator.services.mozilla.com/D30338
e5c6dee921ba9731e39eb6600ac3a46bc9497cf1: Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp
dlee <dlee@mozilla.com> - Wed, 15 May 2019 11:17:43 +0000 - rev 532808
Push 11272 by apavel@mozilla.com at Thu, 16 May 2019 15:28:22 +0000
Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp The goal of this patch is to reduce the number of memory reallocation during |MakePrefixSet|[1]. Here is the number of nsTArray memory reallocation occur during |MakePrefixSet| (test in my local platform): googpub-phish-proto: 58k times goog-malware-proto: 9k times goog-unwanted-proto: 25k times goog-badbinurl-proto: 6k times This patch improves the performance by: 1. For tables whose prefixes are less than 128*1024(malware, unwanted, badinurl). Store prefixes directly without dividing allocation into smaller chunks. Because the maximum size to store all the prefixes in a single array for these tables will be less than 512k, we can avoid Bug 1046038. This simplifies the internal prefixset data structure generation and total memory usage is also saved: goog-malware-proto : 437K -> 163k goog-unwanted-proto : 658k -> 446k goog-badbinurl-proto: 320k -> 110k The single largest allocated continuous memory size is: goog-malware-proto : 86k -> 163k goog-unwanted-proto : 86k -> 446k goog-badbinurl-proto: 77k -> 110k A further improvement can be done for this part is for tables with fewer prefixes, we can use an one-dimension delta array to reduce the size of a single continuous memory allocation. 2. For tables with more prefixes: According to experiment, when prefixes are more than 400k the delta arrays have very high chance that are full, in the case of phishing table, we can estimate the capacity accurately before applying delta algorithm. The shortcoming of this part is when prefixes are between 130k~400k, the capacity estimation is not accurate. [1] https://searchfox.org/mozilla-central/rev/b2015fdd464f598d645342614593d4ebda922d95/toolkit/components/url-classifier/nsUrlClassifierPrefixSet.cpp#99 Differential Revision: https://phabricator.services.mozilla.com/D30046
afbe5300acca5438c03282d21fb6e74088eff4ab: Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp
dlee <dlee@mozilla.com> - Tue, 14 May 2019 22:42:31 +0000 - rev 532807
Push 11272 by apavel@mozilla.com at Thu, 16 May 2019 15:28:22 +0000
Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp The checksum calculating code is used to find the root cause of a crash bug during update(Bug 1362761). Since the algorithm will be update in these series of patches, we don't need to keep it. Differential Revision: https://phabricator.services.mozilla.com/D26667
f94b6f3a7fff9783dded2b404b84c0500c4182f2: Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp
dlee <dlee@mozilla.com> - Fri, 10 May 2019 17:40:11 +0000 - rev 532656
Push 11270 by rgurzau@mozilla.com at Wed, 15 May 2019 15:07:19 +0000
Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp This patch does the following: 1. Run the same prefixset tests when * browser.safebrowsing.prefixset.max_array_size = 0 * browser.safebrowsing.prefixset.max_array_size = UINT32_MAX This makes sure both of the methods to store prefixset are tested by existing testcases 2. Refine gtest with test fixture 3. Add TinySet and LargeSet testcases Differential Revision: https://phabricator.services.mozilla.com/D30338
c51b622bb1fe7e0d21df71a02c7f07f03f1e46fe: Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp
dlee <dlee@mozilla.com> - Tue, 14 May 2019 21:05:41 +0000 - rev 532655
Push 11270 by rgurzau@mozilla.com at Wed, 15 May 2019 15:07:19 +0000
Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp The goal of this patch is to reduce the number of memory reallocation during |MakePrefixSet|[1]. Here is the number of nsTArray memory reallocation occur during |MakePrefixSet| (test in my local platform): googpub-phish-proto: 58k times goog-malware-proto: 9k times goog-unwanted-proto: 25k times goog-badbinurl-proto: 6k times This patch improves the performance by: 1. For tables whose prefixes are less than 128*1024(malware, unwanted, badinurl). Store prefixes directly without dividing allocation into smaller chunks. Because the maximum size to store all the prefixes in a single array for these tables will be less than 512k, we can avoid Bug 1046038. This simplifies the internal prefixset data structure generation and total memory usage is also saved: goog-malware-proto : 437K -> 163k goog-unwanted-proto : 658k -> 446k goog-badbinurl-proto: 320k -> 110k The single largest allocated continuous memory size is: goog-malware-proto : 86k -> 163k goog-unwanted-proto : 86k -> 446k goog-badbinurl-proto: 77k -> 110k A further improvement can be done for this part is for tables with fewer prefixes, we can use an one-dimension delta array to reduce the size of a single continuous memory allocation. 2. For tables with more prefixes: According to experiment, when prefixes are more than 400k the delta arrays have very high chance that are full, in the case of phishing table, we can estimate the capacity accurately before applying delta algorithm. The shortcoming of this part is when prefixes are between 130k~400k, the capacity estimation is not accurate. [1] https://searchfox.org/mozilla-central/rev/b2015fdd464f598d645342614593d4ebda922d95/toolkit/components/url-classifier/nsUrlClassifierPrefixSet.cpp#99 Differential Revision: https://phabricator.services.mozilla.com/D30046
aedbe6cdd06fdda240d262c366c594a463df71d0: Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp
dlee <dlee@mozilla.com> - Wed, 08 May 2019 08:35:06 +0000 - rev 532654
Push 11270 by rgurzau@mozilla.com at Wed, 15 May 2019 15:07:19 +0000
Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp The checksum calculating code is used to find the root cause of a crash bug during update(Bug 1362761). Since the algorithm will be update in these series of patches, we don't need to keep it. Differential Revision: https://phabricator.services.mozilla.com/D26667
3d3527abff681615f5a3dde9bd844b60b80f292d: Bug 1551399 part 2. Stop using [array] in url-classifier's makeFindFullHashRequestV4. r=dimi,gcp
Boris Zbarsky <bzbarsky@mit.edu> - Tue, 14 May 2019 09:57:16 +0000 - rev 532588
Push 11270 by rgurzau@mozilla.com at Wed, 15 May 2019 15:07:19 +0000
Bug 1551399 part 2. Stop using [array] in url-classifier's makeFindFullHashRequestV4. r=dimi,gcp Differential Revision: https://phabricator.services.mozilla.com/D31022
d4ee430e90c2c4b21d13b2f1de063928f1b53d1d: Bug 1551399 part 1. Stop using [array] in url-classifier's makeUpdateRequestV4. r=dimi,gcp
Boris Zbarsky <bzbarsky@mit.edu> - Tue, 14 May 2019 09:50:42 +0000 - rev 532587
Push 11270 by rgurzau@mozilla.com at Wed, 15 May 2019 15:07:19 +0000
Bug 1551399 part 1. Stop using [array] in url-classifier's makeUpdateRequestV4. r=dimi,gcp Differential Revision: https://phabricator.services.mozilla.com/D31020
0c7f3a7403bd17f2d0bb2d17cba47ea02b86c12a: Bug 1547732 - Use LOAD_BYPASS_URL_CLASSIFIER flag for download protection ping. r=gcp
dlee <dlee@mozilla.com> - Mon, 29 Apr 2019 17:36:44 +0000 - rev 530693
Push 11265 by ffxbld-merge at Mon, 13 May 2019 10:53:39 +0000
Bug 1547732 - Use LOAD_BYPASS_URL_CLASSIFIER flag for download protection ping. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D29231
75d2b35c092f4b77c3855569e5dd8b6803e8e914: Bug 1543790 - Fix RDD sandboxing conditions so the parent and child processes agree. r=gcp
Jed Davis <jld@mozilla.com> - Tue, 16 Apr 2019 13:53:20 +0000 - rev 528606
Push 11265 by ffxbld-merge at Mon, 13 May 2019 10:53:39 +0000
Bug 1543790 - Fix RDD sandboxing conditions so the parent and child processes agree. r=gcp If the system doesn't support seccomp-bpf, the parent process won't try to set up sandboxing, but the child process has a separate check that didn't test for this, and ends up failing a release assertion (in SandboxReporterClient, but we also release-assert that installing the seccomp-bpf policy succeeds). This patch just fixes the child-side conditional to match the intended behavior, but in the long term we should consider redesigning SandboxInfo to avoid this. Differential Revision: https://phabricator.services.mozilla.com/D27624
01d9700306a4babdee436d402b4bed5f37b1ec2a: Bug 1543858 - Adjust Linux sandbox policies to tolerate glibc's qsort. r=gcp
Jed Davis <jld@mozilla.com> - Tue, 16 Apr 2019 06:50:50 +0000 - rev 528605
Push 11265 by ffxbld-merge at Mon, 13 May 2019 10:53:39 +0000
Bug 1543858 - Adjust Linux sandbox policies to tolerate glibc's qsort. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D27632
85e806016727c9b063d2c79e0e607301bad352fa: Bug 1543319 - P2. Swap the byte order in-place. r=gcp
dlee <dlee@mozilla.com> - Wed, 10 Apr 2019 20:53:17 +0000 - rev 527711
Push 11265 by ffxbld-merge at Mon, 13 May 2019 10:53:39 +0000
Bug 1543319 - P2. Swap the byte order in-place. r=gcp We don't need an additional array just for byte reordering, replace it with in-place processing. Testcase are modified because the LookupCacheV4::Build API now clears the input parameter. Differential Revision: https://phabricator.services.mozilla.com/D26861
95e20c4ed6c3423ee9c7cdebb52525ca3f3594f5: Bug 1543319 - P1. Free intermediate memory as early as possible during Safe Browsing update. r=gcp
dlee <dlee@mozilla.com> - Wed, 10 Apr 2019 14:32:54 +0000 - rev 527710
Push 11265 by ffxbld-merge at Mon, 13 May 2019 10:53:39 +0000
Bug 1543319 - P1. Free intermediate memory as early as possible during Safe Browsing update. r=gcp Here is the flow how prefixes are handled during an V4 update: 1. Prefixes are received from Safe Browsing update, stored in ProtocolBuffer 2. Copy the prefixes from ProtocolBuffer to TableUpdate structure 3. Prefixes in TableUpdate are merged with local prefixes (stored in LookupCacheV4) 4. Merged prefixes are processes by PrefixSet to generate the in-memory prefix set data structure (MakePrefixSet). In this patch, we free the prefixes stored in TableUpdate right after step3. This reduces the peak memory used during an update (peak happens in step 4). Differential Revision: https://phabricator.services.mozilla.com/D26860
c78da14598ce7727d70e51700bf7afee901aa0e7: Bug 1535304 - Remove provider from about:url-classifier if no table is being used. r=gcp
dlee <dlee@mozilla.com> - Thu, 21 Mar 2019 07:54:20 +0000 - rev 524196
Push 11265 by ffxbld-merge at Mon, 13 May 2019 10:53:39 +0000
Bug 1535304 - Remove provider from about:url-classifier if no table is being used. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D24290
9bac094cec23aa298e04e8326cbd8d9b558575f0: Bug 1353956 - P7. Add gtest to ensure .pset is correctly loaded and removed. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:43:16 +0000 - rev 520894
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P7. Add gtest to ensure .pset is correctly loaded and removed. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D22490
a87bd3f9b87d37bfb3b2ee5137c58bfa669ce1cb: Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:42:31 +0000 - rev 520893
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp To avoid forcing a redownload of SafeBrowsing v4 list. Differential Revision: https://phabricator.services.mozilla.com/D21876
aaba7c25b72b9485fc6c71b879cbc4096f3f4f2b: Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:41:52 +0000 - rev 520892
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp This patch is to cleanup old SafeBrowsing v4 prefix files. Differential Revision: https://phabricator.services.mozilla.com/D21464
3b5da75b9c7b7912e31007b9bac31b4a1ff4a558: Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:41:25 +0000 - rev 520891
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp After this patch, we may have the following files in SafeBrowsing directory: - (v2) .sbstore : Store V2 chunkdata, for update, MD5 integrity check while load - (v2) .pset : Store V2 prefixset, for lookup, load upon startup, no integrity check - (v4) .metadata : Store V4 state, for update, no integrity check - (v4) .vlpset : Store V4 prefixset, for lookup, load upon startup, CRC32 integrity check - (v4) .pset : V4 prefix set before this patch, should be removed The magic string is also added to ".vlpset" header so we can add a telemetry to see if sanity check is good enough for prefix set integrity check (The telemetry is not yet added). If yes, we can remove the CRC32 in the future for even better performance. Differential Revision: https://phabricator.services.mozilla.com/D21463
e083106dc24f6f562eb2d72aa02871a27a681ebe: Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:40:56 +0000 - rev 520890
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp SafeBrowsing prefix files LOAD/SAVE operations are handled in xxxPrefixSet.cpp. It would be more clear if xxxPrefixSet.cpp only processes prefix data, while LookupCacheV2/LookupCacheV4 which use prefix set process file. This patch doesn't change any behavior, testcases need to update because the LookupCache & xxxPrefixSet APIs are changed. Differential Revision: https://phabricator.services.mozilla.com/D21462
c7a253aed4508a8df4f1309d889dbfd669cd1693: Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:40:28 +0000 - rev 520889
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp SHA256 is an expensive operation, we should avoid using them if possible. SafeBrowsing prefix files are loaded during startup and verify integrity with SHA256 which may affect the performance especially on the low-end device. This patch simply removes the SHA256 integrity check. CRC32 version integrity check will be introduced in the other patch. This patch also changes the behavior of recording "Telemetry::URLCLASSIFIER_VLPS_LOAD_CORRUPT" a little bit. It used to records only once per session(during startup, the first time we load prefix set), now it records per update. Differential Revision: https://phabricator.services.mozilla.com/D21461
c2331373e10707aa5259eb3d8436f65c8d59c684: Bug 1353956 - P1. Rename checksum used in SafeBrowsing V4 to SHA256. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:40:14 +0000 - rev 520888
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P1. Rename checksum used in SafeBrowsing V4 to SHA256. r=gcp SafeBrowsing V4 protocol use SHA-256 as the checksum to check integrity of update data and also the integrity of prefix files. SafeBrowsing V2 HashStore use MD5 as the checksum to check integrity of .sbstore Since we are going to use CRC32 as the integrity check of V4 prefix files, I think rename V4 "checksum" to SHA256 can improve readability. Differential Revision: https://phabricator.services.mozilla.com/D21460
71dafccc22ae493b656d9130cec82ba37d4babf7: Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp
dlee <dlee@mozilla.com> - Wed, 06 Mar 2019 09:41:34 +0000 - rev 520664
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp To avoid forcing a redownload of SafeBrowsing v4 list. Differential Revision: https://phabricator.services.mozilla.com/D21876
f1f29fe519cf169d43eeacfe710d0b84e9da55db: Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp
dlee <dlee@mozilla.com> - Tue, 05 Mar 2019 18:32:23 +0000 - rev 520663
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp This patch is to cleanup old SafeBrowsing v4 prefix files. Differential Revision: https://phabricator.services.mozilla.com/D21464
4978556a66f66cd6d214f98aa656ea1e223ead63: Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Wed, 06 Mar 2019 22:57:12 +0000 - rev 520662
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp After this patch, we may have the following files in SafeBrowsing directory: - (v2) .sbstore : Store V2 chunkdata, for update, MD5 integrity check while load - (v2) .pset : Store V2 prefixset, for lookup, load upon startup, no integrity check - (v4) .metadata : Store V4 state, for update, no integrity check - (v4) .vlpset : Store V4 prefixset, for lookup, load upon startup, CRC32 integrity check - (v4) .pset : V4 prefix set before this patch, should be removed The magic string is also added to ".vlpset" header so we can add a telemetry to see if sanity check is good enough for prefix set integrity check (The telemetry is not yet added). If yes, we can remove the CRC32 in the future for even better performance. Differential Revision: https://phabricator.services.mozilla.com/D21463
bc0b91abce9bcb43ef501dfa7bca03ea678f9783: Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp
Dimi Lee <dlee@mozilla.com> - Mon, 04 Mar 2019 21:22:46 +0000 - rev 520661
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp SafeBrowsing prefix files LOAD/SAVE operations are handled in xxxPrefixSet.cpp. It would be more clear if xxxPrefixSet.cpp only processes prefix data, while LookupCacheV2/LookupCacheV4 which use prefix set process file. This patch doesn't change any behavior, testcases need to update because the LookupCache & xxxPrefixSet APIs are changed. Differential Revision: https://phabricator.services.mozilla.com/D21462
6b8412db5a05f12bf44b5abc1bf553ed356da41f: Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Thu, 28 Feb 2019 08:18:46 +0000 - rev 520660
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp SHA256 is an expensive operation, we should avoid using them if possible. SafeBrowsing prefix files are loaded during startup and verify integrity with SHA256 which may affect the performance especially on the low-end device. This patch simply removes the SHA256 integrity check. CRC32 version integrity check will be introduced in the other patch. This patch also changes the behavior of recording "Telemetry::URLCLASSIFIER_VLPS_LOAD_CORRUPT" a little bit. It used to records only once per session(during startup, the first time we load prefix set), now it records per update. Differential Revision: https://phabricator.services.mozilla.com/D21461
3d326cfcd002b1c9598151ae5d39bbc9051524f3: Bug 1353956 - P1. Rename checksum used in SafeBrowsing V4 to SHA256. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 28 Feb 2019 08:12:36 +0000 - rev 520659
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1353956 - P1. Rename checksum used in SafeBrowsing V4 to SHA256. r=gcp SafeBrowsing V4 protocol use SHA-256 as the checksum to check integrity of update data and also the integrity of prefix files. SafeBrowsing V2 HashStore use MD5 as the checksum to check integrity of .sbstore Since we are going to use CRC32 as the integrity check of V4 prefix files, I think rename V4 "checksum" to SHA256 can improve readability. Differential Revision: https://phabricator.services.mozilla.com/D21460
d5f2a2a21c1599d35bffbadae8c9a52f9abc25b7: Bug 1531322 - Add "bin" to the list of executable extensions in download protection. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 28 Feb 2019 13:22:15 +0000 - rev 519596
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1531322 - Add "bin" to the list of executable extensions in download protection. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D21529
493b443954fe15f7b542ba14671f25e5f8531dff: Bug 1506291 - Add Linux sandboxing for the RDD (media decoder) process. r=gcp,mjf,flod
Jed Davis <jld@mozilla.com> - Wed, 27 Feb 2019 20:14:54 +0000 - rev 519405
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1506291 - Add Linux sandboxing for the RDD (media decoder) process. r=gcp,mjf,flod The seccomp-bpf policy is currently just the "common" policy with no additions (but with the fixes in bug 1511560 to enable shared memory creation). The file broker policy allows shared memory creation and nothing else. The namespace setup is the same as for GMP (i.e., as restrictive as we currently can be). The sandbox can be turned off for troubleshooting by setting the environment variable MOZ_DISABLE_RDD_SANDBOX, similarly to the other process types. Tested against https://demo.bitmovin.com/public/firefox/av1/ with the necessary prefs set. Depends on D20895 Differential Revision: https://phabricator.services.mozilla.com/D14525
bf58d8320f5a1de358b930d996615c73ff22cce9: Bug 1506291 - Move the AV1 decoders to a sandbox-friendly CPU counting wrapper. r=gcp,mjf
Jed Davis <jld@mozilla.com> - Mon, 25 Feb 2019 16:20:50 +0000 - rev 519404
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1506291 - Move the AV1 decoders to a sandbox-friendly CPU counting wrapper. r=gcp,mjf Counting CPUs accesses the filesystem (sysfs or procfs), which we'd like to disallow when sandboxed if possible, and fails silently if access is denied. Because the CPU count rarely changes, this patch handles that problem for the RDD process by caching a copy before starting sandboxing. Tested with a local patch to have the sandbox file broker client crash if accessing the sysfs node for the CPU count, to verify that it's not accessed. Depends on D14524 Differential Revision: https://phabricator.services.mozilla.com/D20895
94cb1fe9db5eb0f0aa0634541afb08af17cf5c05: Bug 1511560 - Allow dup and ftruncate (when needed) in SandboxPolicyCommon. r=gcp
Jed Davis <jld@mozilla.com> - Wed, 27 Feb 2019 20:14:52 +0000 - rev 519403
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1511560 - Allow dup and ftruncate (when needed) in SandboxPolicyCommon. r=gcp File descriptors are sometimes dup()ed in the process of communicating them over IPC; some of this may be unnecessary (due to insufficient use of move-only types), but dup() is relatively harmless. It was previously allowed for both content and GMP, so this doesn't change anything. The handling of ftruncate is a little complicated -- it's used for IPC shared memory, but only when creating segments; so GMP doesn't allow it and should continue not allowing it, but content needs it and RDD will as well. As a result, the subclass indicates if it will be needed. Note that even when we have memfd_create support (bug 1440203), ftruncate is still necessary even though brokering may not. Depends on D14523 Differential Revision: https://phabricator.services.mozilla.com/D14524
db2dee78ddb0dd23e29948258abd6c7404555b59: Bug 1511560 - Move the socketpair handling into SandboxPolicyCommon. r=gcp
Jed Davis <jld@mozilla.com> - Sat, 23 Feb 2019 00:44:10 +0000 - rev 519402
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1511560 - Move the socketpair handling into SandboxPolicyCommon. r=gcp The sandbox broker uses socketpair to construct the per-request channels over which responses are sent; thus, if and only if the policy will be using brokering, it will allow socketpair as safely as possible (i.e., denying datagram sockets if possible). Depends on D14522 Differential Revision: https://phabricator.services.mozilla.com/D14523
bab79f85596242146787d6d2a5ad56596cc1343e: Bug 1511560 - Move syscalls for adjusting memory mapping properties into SandboxPolicyCommon. r=gcp
Jed Davis <jld@mozilla.com> - Sat, 23 Feb 2019 00:44:08 +0000 - rev 519401
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1511560 - Move syscalls for adjusting memory mapping properties into SandboxPolicyCommon. r=gcp madvise is used by our malloc (and probably others), and mprotect is used with shared memory, including when created by another process, so the common policy should include those rules. Depends on D14521 Differential Revision: https://phabricator.services.mozilla.com/D14522
48431f63d84227177951f65c9c828548d9a8bbb2: Bug 1511560 - Move the Linux sandbox broker hooks into SandboxPolicyCommon. r=gcp
Jed Davis <jld@mozilla.com> - Sat, 23 Feb 2019 00:44:06 +0000 - rev 519400
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1511560 - Move the Linux sandbox broker hooks into SandboxPolicyCommon. r=gcp This will allow other policies to use brokering if needed (e.g., RDD and similar utility processes may need to access /dev/shm to create shared memory). The concrete policy class can deny filesystem access completely (matching the current behavior of the GMP policy) by passing nullptr to the superclass constructor instead. Depends on D14520 Differential Revision: https://phabricator.services.mozilla.com/D14521
56f39977c72c62e0fdff0e5f68e72d6091b221db: Bug 1500297 - Require a broker client in ContentSandboxPolicy at level > 1. r=gcp
Jed Davis <jld@mozilla.com> - Wed, 27 Feb 2019 15:23:25 +0000 - rev 519399
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1500297 - Require a broker client in ContentSandboxPolicy at level > 1. r=gcp ContentSandboxPolicy currently allows direct filesystem access if it isn't given a broker client; this is a legacy design from the B2G era, before the current idea of "sandbox level". With this patch, it allows filesystem access at level 1, and above that it requires brokering. This is both to reduce the opportunities for accidentally having a too-permissive sandbox and to prepare for refactoring the broker glue in bug 1511560. Depends on D14519 Differential Revision: https://phabricator.services.mozilla.com/D14520
bacaa3d582814d0a1ba3769de92e68a01d16a777: Bug 1500297 - Fix Linux content sandbox level 1. r=gcp
Jed Davis <jld@mozilla.com> - Wed, 27 Feb 2019 15:23:25 +0000 - rev 519398
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1500297 - Fix Linux content sandbox level 1. r=gcp Level 1 is meant to enable some seccomp-bpf filtering, but still allow direct access to the filesystem, and level 2 is where brokering starts. This was accidentally broken in 1365257 (making "level 1" act like level 2); this patch fixes that. This feature obviously isn't used much given how long nobody noticed it was broken, but it's useful to have around for troubleshooting, and it's actually easier to fix it than edit it out of the documentation. Differential Revision: https://phabricator.services.mozilla.com/D14519
e7e5611f9bdc7b8aaff1a72382aa92de73b4ecda: Bug 1526885 - P2. Add MS Office files to the list of executable extensions in download protection. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 14 Feb 2019 13:30:38 +0000 - rev 517025
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1526885 - P2. Add MS Office files to the list of executable extensions in download protection. r=gcp Sync with the list in chrome, chrome adds office docs because they have been abused in the past through embedded executables. Differential Revision: https://phabricator.services.mozilla.com/D19759
b7ccbea196c94034e862feb09cb655700a33fd51: Bug 1526885 - P1. Add extensions to download protectionn executable list. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 14 Feb 2019 13:29:45 +0000 - rev 517024
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1526885 - P1. Add extensions to download protectionn executable list. r=gcp Those extensions are set to SAMPLED_PING by google, which means they are not really dangerous. Add these extensions to sync up with google's list but comment them because we don't need to trigger remote lookup request for them. Differential Revision: https://phabricator.services.mozilla.com/D19758
cd21500d51113327d995e7e5c053d525ba0bbed9: Bug 1525199 - Part 1 - removed size_t from IPDL messages for Cameras; r=gcp
Alex Gaynor <agaynor@mozilla.com> - Sun, 10 Feb 2019 10:24:34 +0000 - rev 516371
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1525199 - Part 1 - removed size_t from IPDL messages for Cameras; r=gcp Differential Revision: https://phabricator.services.mozilla.com/D19191
f86cea87bf827f87f664068b76c01a8944deb22e: Bug 1513535 - Support the ability to separate feature toggle and list update in URL classifier. r=gcp
Dimi Lee <dlee@mozilla.com> - Tue, 22 Jan 2019 16:13:41 +0000 - rev 513756
Push 10862 by ffxbld-merge at Mon, 11 Mar 2019 13:01:11 +0000
Bug 1513535 - Support the ability to separate feature toggle and list update in URL classifier. r=gcp Add preferences "browser.safebrowsing.features.[feature name].update". Normally these preferences won't be set so the SafeBrowsing uses features's enable/disable preferences to decide if it should update the list or not. If an update preference is present, then it has higher priority then the enable/disable one. This provides a way for the SafeBrowsing consumer to be able to separate feature toggle and upodate. Differential Revision: https://phabricator.services.mozilla.com/D17233
ff26ee3e478c19baf73700cbd9467e897711ad66: Bug 1435923 - Fix truncated logging of remote lookup protocol buffer. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 17 Jan 2019 12:38:04 +0000 - rev 511379
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1435923 - Fix truncated logging of remote lookup protocol buffer. r=gcp Fix the truncate issue when output the protocol buffer Differential Revision: https://phabricator.services.mozilla.com/D16675
d29c64bd30daa39854747f41bff56d7fb51ca3d3: Bug 1513490 - Support update cryptomining and fingerprinting list in SafeBrowsing. r=gcp
dlee <dlee@mozilla.com> - Tue, 15 Jan 2019 19:29:49 +0000 - rev 511075
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1513490 - Support update cryptomining and fingerprinting list in SafeBrowsing. r=gcp Add the fingerprinting and cryptomining tables to the SafeBrowsing update list. Leave the preference of blacklist/whitelist tables empty until the shavar server is ready. Differential Revision: https://phabricator.services.mozilla.com/D16533
33011122d2d517bdb61ca149e2f0ce20dae528a2: Bug 1513490 - Refactor SafeBrowsing.jsm with feature list. r=gcp
Dimi Lee <dlee@mozilla.com> - Tue, 15 Jan 2019 20:04:40 +0000 - rev 511074
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1513490 - Refactor SafeBrowsing.jsm with feature list. r=gcp When we add a table to SafeBrowsing.jsm we need to add related code in various places. This patch simplify the work by providing a FEATURE table which defines the data required. Differential Revision: https://phabricator.services.mozilla.com/D16532
ff87d87996fda94c59817084578291d2e3ce16c9: Bug 1519660 - Remove flash info bar from SafeBrowsing list. r=gcp
Dimi Lee <dlee@mozilla.com> - Mon, 14 Jan 2019 15:47:23 +0000 - rev 510852
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1519660 - Remove flash info bar from SafeBrowsing list. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D16397
9a966111a9e0aa62985eb96f8ff43bedd1fc2c11: Bug 1517704 - Add cpi to the list of executable extensions in download protection. r=gcp
Dimi Lee <dlee@mozilla.com> - Mon, 07 Jan 2019 15:22:21 +0000 - rev 509850
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1517704 - Add cpi to the list of executable extensions in download protection. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D15804
454e109602369647079000150d94d7aa2f269427: Bug 1516363: Fix duplicate definition of scoreThreshold causing compile error with `--disable-tests` r=gcp
Carl Corcoran <ccorcoran@mozilla.com> - Wed, 26 Dec 2018 13:05:10 +0000 - rev 509028
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1516363: Fix duplicate definition of scoreThreshold causing compile error with `--disable-tests` r=gcp When compiling without ENABLE_TESTS, the following error occurs: /toolkit/xre/ModuleEvaluator_windows.cpp(221,20): error: redefinition of 'scoreThreshold' with a different type: 'const int' vs 'int' This removes the 2 lines of code that caused the error. Differential Revision: https://phabricator.services.mozilla.com/D15372
a3d8cffac0ab7fde2810fe7cc3a8183afe7462b6: Bug 1510559 - Add .desktop to the list of executable extensions in download protection. r=gcp
Dimi Lee <dlee@mozilla.com> - Wed, 12 Dec 2018 10:57:50 +0000 - rev 507377
Push 10547 by ffxbld-merge at Mon, 21 Jan 2019 13:03:58 +0000
Bug 1510559 - Add .desktop to the list of executable extensions in download protection. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D14238
023c546c01cd2d3c4d382ab5e9245a48f138e934: Bug 1508898 - Prepare the Linux sandbox's socketcall/ipc-call dispatch table for reformatting. r=gcp
Jed Davis <jld@mozilla.com> - Wed, 21 Nov 2018 11:05:31 +0000 - rev 503939
Push 10290 by ffxbld-merge at Mon, 03 Dec 2018 16:23:23 +0000
Bug 1508898 - Prepare the Linux sandbox's socketcall/ipc-call dispatch table for reformatting. r=gcp The tables in SandboxFilterUtil.cpp should remain vertically aligned, but clang-format would disagree. This patch excludes that region from reformatting, and applies the other changes that clang-format would make there. Differential Revision: https://phabricator.services.mozilla.com/D12499
83fc4342c255dff83427b1bc5269ff97a371463e: Bug 1506788 - Add pyd and pyo binary extensions to download protection. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 15 Nov 2018 10:17:35 +0000 - rev 503009
Push 10290 by ffxbld-merge at Mon, 03 Dec 2018 16:23:23 +0000
Bug 1506788 - Add pyd and pyo binary extensions to download protection. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D11750
9b1adbad418a0b4aebfe947efc0a088615688481: Bug 1464220 - Extend SANDBOX_HAS_USER_NAMESPACES telemetry. r=chutten,gcp
Jed Davis <jld@mozilla.com> - Wed, 31 Oct 2018 22:29:39 +0000 - rev 500447
Push 10290 by ffxbld-merge at Mon, 03 Dec 2018 16:23:23 +0000
Bug 1464220 - Extend SANDBOX_HAS_USER_NAMESPACES telemetry. r=chutten,gcp Differential Revision: https://phabricator.services.mozilla.com/D9456