searching for reviewer(gcp)
6d946d2a7a313e4ccb8aadb4b1a86e78d1e68301: Bug 1575842 - Do not use |ResetTables| when detecting Safe Browing database corruption in GetLookupCache. r=gcp
dlee <dlee@mozilla.com> - Fri, 23 Aug 2019 08:19:18 +0000 - rev 553308
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1575842 - Do not use |ResetTables| when detecting Safe Browing database corruption in GetLookupCache. r=gcp This patch replaces |ResetTables|(clear table's in-memory and on-disk data) with |DeleteTables|(clear table's on-disk data) in GetLookupCache to avoid infinite loop. We can just delete on-disk data when file corruption is detected in |GetLookupCache| without clearing the cache's internal data and refreshing current active caches because in that scenario, the lookup cache failing to read database has not yet added to the active caches list. Differential Revision: https://phabricator.services.mozilla.com/D43181
266d6e0597d8666ec55d833064cc0296dc7d1502: Bug 1575564 - avoid non-mainthread use of NS_GetSpecialDirectory in linux sandboxbroker, r=jld,gcp
Gijs Kruitbosch <gijskruitbosch@gmail.com> - Thu, 22 Aug 2019 16:37:18 +0000 - rev 553269
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1575564 - avoid non-mainthread use of NS_GetSpecialDirectory in linux sandboxbroker, r=jld,gcp Differential Revision: https://phabricator.services.mozilla.com/D42951
df6b5b4da8b909d61701356243dd95f20e983bf0: Bug 1562822 - P3. Reset all the tables that fail to apply a Safe Browsing update. r=gcp
dlee <dlee@mozilla.com> - Wed, 21 Aug 2019 12:08:12 +0000 - rev 552931
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1562822 - P3. Reset all the tables that fail to apply a Safe Browsing update. r=gcp Before this patch, when Safe Browsing updating process discovers an error, it quits and resets the table failing to update. After this patch, updating process will continue to run when an error occurs to find all the tables failing to apply an update. Differential Revision: https://phabricator.services.mozilla.com/D42615
d49cd2aa45686212954493e2465fc3258c1796a2: Bug 1562822 - P2. Reset corrupted Safe Browsing database before triggering an update. r=gcp
dlee <dlee@mozilla.com> - Wed, 21 Aug 2019 12:08:03 +0000 - rev 552930
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1562822 - P2. Reset corrupted Safe Browsing database before triggering an update. r=gcp Patch P2 & P3 refine how Safe Browsing handles Safe Browsing database loading failure. Safe Browsing databases are read in 3 scenarios: 1. |GetLookupCache| is called on startup. Safe Browsing reads prefix files in this case. Metadata for updates(.sbstore, .metadata) are not read in this scenario. 2. |TableRequest| is called before applying an update, Safe Browsing reads update metadata to apply a partial update. 3. During an update, Safe Browsing reads both prefix files and metadata in order to merge the update result. For Case 1, we reset a table's database only when it returns FILE_CORRUPTED while loading prefixes from the prefix file(.vlpset). For Case 2, we reset a table's database when the table fails to load its metadata file or prefix file. This is because we need to make sure both files are complete so we can correctly perform a partial update. Note that in this case, we don't just reset the database when "FILE_CORRUPTED" is detected, we reset the database as long as an error occurs while loading the database. For Case 3, For all the tables failing to load their database during an updating process, the databases of those tables will be reset. Case 1 and Case 2 are done in Patch P2; Case 3 is done in Patch P3 Differential Revision: https://phabricator.services.mozilla.com/D42614
7a0ab3a7558b4fc930d1066f3f204ee99ccfc162: Bug 1562822 - P1. Return NS_ERROR_FILE_CORRUPTED when Safe Browsing cannot read the header of its database. r=gcp
dlee <dlee@mozilla.com> - Wed, 21 Aug 2019 12:07:44 +0000 - rev 552929
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1562822 - P1. Return NS_ERROR_FILE_CORRUPTED when Safe Browsing cannot read the header of its database. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D42613
9c6e5858528aca9447967b6b5130cc4a96b9c91a: Bug 1551524 - Report failure in LookupCache::WriteFile() if StoreToFile returns an error. r=gcp
dlee <dlee@mozilla.com> - Tue, 20 Aug 2019 18:07:42 +0000 - rev 552788
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1551524 - Report failure in LookupCache::WriteFile() if StoreToFile returns an error. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D42618
466843804c09151c4f2c7733b9c763f4edc27fc4: Bug 1573666 - Do not show warning message when there is no Safe Browsing V3 .sbstore files. r=gcp
dlee <dlee@mozilla.com> - Mon, 19 Aug 2019 09:34:18 +0000 - rev 552481
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1573666 - Do not show warning message when there is no Safe Browsing V3 .sbstore files. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D41942
b4a0f4026d4db81feab1c8b1c2172a97f2079cbe: Bug 1574450 - Close inputstream before reseting HashStore file. r=gcp
dlee <dlee@mozilla.com> - Mon, 19 Aug 2019 07:45:33 +0000 - rev 552435
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1574450 - Close inputstream before reseting HashStore file. r=gcp mInputStream holds a reference to the current opened HashStore file. While resetting the file, mInputStream should be closed first, otherwise, the file->Remove returns failure code NS_ERROR_FILE_IS_LOCKED. (This only happens in Windows platform) Differential Revision: https://phabricator.services.mozilla.com/D42460
d1316e6def59b8017bb3b48558ce86df329279a4: Bug 1564346 - SafeBrowsing gtest code refactoring. r=gcp
dlee <dlee@mozilla.com> - Thu, 08 Aug 2019 14:29:40 +0000 - rev 550902
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1564346 - SafeBrowsing gtest code refactoring. r=gcp Refactor the gtest code because it confused me while adding new tests. This patch focus on refining utility function but it also contains other minor refinements. Changes includes: 1. Add comments to utility function 2. Move common utility functions to Common.cpp and remove duplicates 3. Header file removal and reorder 4. Unify MPL commnetc 5. Replace anonymouse namespace with static function Differential Revision: https://phabricator.services.mozilla.com/D37532
938ff7ae5eff2739d6999b7d982383c0abc878f4: Bug 1564041 - P1. Add telemetry to measure download protection binary type r=gcp
dimi <dlee@mozilla.com> - Thu, 08 Aug 2019 07:38:19 +0000 - rev 550611
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1564041 - P1. Add telemetry to measure download protection binary type r=gcp This patch adds a telemetry, APPLICATION_REPUTATION_BINARY_TYPE, which records different binary type based on the file extension. 1. BinaryFile, file is considered as a binary file, file is eligible for remote lookup 2. NonBinaryFile, file is not considered as a binary file. 3. MozNonBinaryFile, file is considered as a binary file in Chrome, but we don't send a download protection ping for this file 4. UnknownFile, file is not in any of the above lists. Differential Revision: https://phabricator.services.mozilla.com/D37275
89b42e05fd3125c12acd27c61fd9c24f5423ff6a: Bug 1559368 - When determining sandbox capabilities, check for the specific X11 socket that would be used. r=gcp
Jed Davis <jld@mozilla.com> - Wed, 07 Aug 2019 22:34:50 +0000 - rev 550585
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1559368 - When determining sandbox capabilities, check for the specific X11 socket that would be used. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D40915
6ec4bd94bb511e3f73f6f290c30e463e6ce740ee: Bug 1564346 - SafeBrowsing gtest code refactoring. r=gcp
dlee <dlee@mozilla.com> - Wed, 07 Aug 2019 15:17:49 +0000 - rev 550513
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1564346 - SafeBrowsing gtest code refactoring. r=gcp Refactor the gtest code because it confused me while adding new tests. This patch focus on refining utility function but it also contains other minor refinements. Changes includes: 1. Add comments to utility function 2. Move common utility functions to Common.cpp and remove duplicates 3. Header file removal and reorder 4. Unify MPL commnetc 5. Replace anonymouse namespace with static function Differential Revision: https://phabricator.services.mozilla.com/D37532
3174ab6c79af5d9edeea03645e86ea07ab5415a4: Bug 1562875 - Listmanager kiffoffUpdate doesn't need to obtain on-disk data. r=gcp
dlee <dlee@mozilla.com> - Thu, 11 Jul 2019 12:49:14 +0000 - rev 548862
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1562875 - Listmanager kiffoffUpdate doesn't need to obtain on-disk data. r=gcp While listmanager called |kickoffUpdate|, it used to call |GetTables| to retrieve update information and used the information to distinguish if a table is a "existing" table or not. In Bug 1045163, the "existing table" logic was removed, which also means we don't need to call |GetTabkes| anymore. This patch removes calling Classifier::TableRequest to reduce unnecessary disk IO during startup. Differential Revision: https://phabricator.services.mozilla.com/D37037
5367a238ac11df32db7183ba304f0455827aec90: Bug 1565818 - Disable the safebrowsing tests on MinGW clang builds r=gcp
Tom Ritter <tom@mozilla.com> - Wed, 24 Jul 2019 13:41:19 +0000 - rev 547800
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1565818 - Disable the safebrowsing tests on MinGW clang builds r=gcp Differential Revision: https://phabricator.services.mozilla.com/D37985
eb7f4d56f54b3283fc15983ee859b5e62fcb9f3b: Bug 1567236: Drop Windows content sandbox level back to 5 on Nightly. r=gcp
Bob Owen <bobowencode@gmail.com> - Fri, 19 Jul 2019 11:40:14 +0000 - rev 547230
Push 2165 by ffxbld-merge at Mon, 14 Oct 2019 16:30:58 +0000
Bug 1567236: Drop Windows content sandbox level back to 5 on Nightly. r=gcp This is because of profiling and performance regressions, which we don't want to live with while we investigate. Differential Revision: https://phabricator.services.mozilla.com/D38645
9b2474ad94517039fcc1c23732c829c13b1e5d62: Bug 1574450 - Close inputstream before reseting HashStore file. r=gcp, a=RyanVM
dlee <dlee@mozilla.com> - Mon, 19 Aug 2019 07:45:33 +0000 - rev 545184
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1574450 - Close inputstream before reseting HashStore file. r=gcp, a=RyanVM mInputStream holds a reference to the current opened HashStore file. While resetting the file, mInputStream should be closed first, otherwise, the file->Remove returns failure code NS_ERROR_FILE_IS_LOCKED. (This only happens in Windows platform) Differential Revision: https://phabricator.services.mozilla.com/D42460
93c33782f57deb40cd05a08bf0c72092bddb327e: Bug 1531354 - P6. Remove unused testing files and load old version of prefixes data. r=gcp
dlee <dlee@mozilla.com> - Sat, 29 Jun 2019 19:24:14 +0000 - rev 543540
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1531354 - P6. Remove unused testing files and load old version of prefixes data. r=gcp This patch does the following: 1. Remove testing files from disk because they are no longer required. 2. Load completions from previous version of HashStore until an update is applied. 3. Older version of HashStore(.sbstore) & PrefixSet(.vlpset) will be removed during an update Differential Revision: https://phabricator.services.mozilla.com/D36002
cc7cce83765a90aa1b25b566917b42e9450532fe: Bug 1531354 - P5. Safe Browsing test entries are directly stored in LookupCache. r=gcp
dlee <dlee@mozilla.com> - Sat, 29 Jun 2019 19:05:41 +0000 - rev 543539
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1531354 - P5. Safe Browsing test entries are directly stored in LookupCache. r=gcp Create test entries via update introduces performance overhead. We can store them directly in LookupCache and do not save test entries to disk. Differential Revision: https://phabricator.services.mozilla.com/D34576
7417be59b320f4ba945e3c4f6c0a47b0c19d8df2: Bug 1531354 - P4. Skip reading hashstore in RegenActiveTables. r=gcp
dlee <dlee@mozilla.com> - Fri, 21 Jun 2019 23:11:06 +0000 - rev 543538
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1531354 - P4. Skip reading hashstore in RegenActiveTables. r=gcp For Safe Browsing V2, Data for lookup(LookupCache) and data for update(HashStore) are now separated. |RegenActiveTables| doesn't need to check the chunk number in HashStore. Differential Revision: https://phabricator.services.mozilla.com/D34575
3799349af73f98fb9c10051b536417717c68047d: Bug 1531354 - P3. Do not store completion in HashStore. r=gcp
dlee <dlee@mozilla.com> - Wed, 26 Jun 2019 19:45:08 +0000 - rev 543537
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1531354 - P3. Do not store completion in HashStore. r=gcp Completions are now stored in .vlpset, we can remove it from .sbstore Functions related to optimize reading completions from .sbstore can also be removed because it is no longer HashStore's responsibility Differential Revision: https://phabricator.services.mozilla.com/D34574
df0ff09bdcf9eb397e7b6fdacd46a95047be77a1: Bug 1531354 - P2. Use variable-length prefix set in LookupCacheV2. r=gcp
dlee <dlee@mozilla.com> - Wed, 26 Jun 2019 19:40:45 +0000 - rev 543536
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1531354 - P2. Use variable-length prefix set in LookupCacheV2. r=gcp 1. VariableLengthPrefixSet supports getting/setting prefixes with AddPrefixArray and AddCompletesArray 2. VariableLengthPrefixSet supports passing prefix as an integer in Match API. This is because how V2 and V4 see prefixes as an integer works differently. Differential Revision: https://phabricator.services.mozilla.com/D34547
4ec7371e5a41eb4a6a54a5e1174fa228d912a968: Bug 1531354 - P1. Remove mPrefixSet and mUpdateCompletions from LookupCacheV2 and use mVLPresetSet. r=gcp
dlee <dlee@mozilla.com> - Fri, 21 Jun 2019 23:07:52 +0000 - rev 543535
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1531354 - P1. Remove mPrefixSet and mUpdateCompletions from LookupCacheV2 and use mVLPresetSet. r=gcp The goal of the series of patches is to improve Safe Browsing performance by skipping uncessary file IO. The first two patches is to remove the dependency between LookupCache and HashStore, so HashStore is only responsible for udpates. Before this patch, LookupCacheV2 treats prefixes and completions differently. It uses two data structures to maintain prefixes: 1. mPrefixSet to store prefixes from .pset 2. mUpdateCompletions to store completions from .sbstore After this patch 1. LookupCacheV2 & LookupCacheV4 both use variable-length prefix set. mUpdateCompletions and mPrefixSet are removed and mVLPrefixSet is used to store all prefixes data. 2. Move common function to base class. Note that in this patch, conversion between 4/32 bytes prefixes and mVLPrefixSet is not yet included, it will be handled in next patch. This patch tries not to deal with any logic changes, only focus on refining LookupCacheV2 & LookupCacheV4 class structure to use variable-length prefixset for both classes. Differential Revision: https://phabricator.services.mozilla.com/D34546
db899a74f036940fc80cb3f6cf3452738dbf95d8: Bug 1553963 - Fix Safe Browsing doesn't cache the gethash result for V4 tables. r=gcp
dlee <dlee@mozilla.com> - Fri, 24 May 2019 19:42:00 +0000 - rev 538298
Push 2131 by ffxbld-merge at Mon, 26 Aug 2019 18:30:20 +0000
Bug 1553963 - Fix Safe Browsing doesn't cache the gethash result for V4 tables. r=gcp We use ".pset" to find active tables, but in Bug 1353956, v4 prefix files are renamed to ".vlpset". This patches include both 'pset' and 'vlpset' to ScanStoreDir. Differential Revision: https://phabricator.services.mozilla.com/D32433
bc9fed12c6e8f969337d00af730585d5590ce302: Bug 1553963 - Fix Safe Browsing doesn't cache the gethash result for V4 tables. r=gcp a=jcristau
dlee <dlee@mozilla.com> - Fri, 24 May 2019 19:42:00 +0000 - rev 536512
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1553963 - Fix Safe Browsing doesn't cache the gethash result for V4 tables. r=gcp a=jcristau We use ".pset" to find active tables, but in Bug 1353956, v4 prefix files are renamed to ".vlpset". This patches include both 'pset' and 'vlpset' to ScanStoreDir. Differential Revision: https://phabricator.services.mozilla.com/D32433
76339e786c7cc22d2c21d8f503e3d70df76167f2: Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp
dlee <dlee@mozilla.com> - Tue, 14 May 2019 22:42:28 +0000 - rev 535876
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp This patch does the following: 1. Run the same prefixset tests when * browser.safebrowsing.prefixset.max_array_size = 0 * browser.safebrowsing.prefixset.max_array_size = UINT32_MAX This makes sure both of the methods to store prefixset are tested by existing testcases 2. Refine gtest with test fixture 3. Add TinySet and LargeSet testcases Differential Revision: https://phabricator.services.mozilla.com/D30338
e5c6dee921ba9731e39eb6600ac3a46bc9497cf1: Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp
dlee <dlee@mozilla.com> - Wed, 15 May 2019 11:17:43 +0000 - rev 535875
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp The goal of this patch is to reduce the number of memory reallocation during |MakePrefixSet|[1]. Here is the number of nsTArray memory reallocation occur during |MakePrefixSet| (test in my local platform): googpub-phish-proto: 58k times goog-malware-proto: 9k times goog-unwanted-proto: 25k times goog-badbinurl-proto: 6k times This patch improves the performance by: 1. For tables whose prefixes are less than 128*1024(malware, unwanted, badinurl). Store prefixes directly without dividing allocation into smaller chunks. Because the maximum size to store all the prefixes in a single array for these tables will be less than 512k, we can avoid Bug 1046038. This simplifies the internal prefixset data structure generation and total memory usage is also saved: goog-malware-proto : 437K -> 163k goog-unwanted-proto : 658k -> 446k goog-badbinurl-proto: 320k -> 110k The single largest allocated continuous memory size is: goog-malware-proto : 86k -> 163k goog-unwanted-proto : 86k -> 446k goog-badbinurl-proto: 77k -> 110k A further improvement can be done for this part is for tables with fewer prefixes, we can use an one-dimension delta array to reduce the size of a single continuous memory allocation. 2. For tables with more prefixes: According to experiment, when prefixes are more than 400k the delta arrays have very high chance that are full, in the case of phishing table, we can estimate the capacity accurately before applying delta algorithm. The shortcoming of this part is when prefixes are between 130k~400k, the capacity estimation is not accurate. [1] https://searchfox.org/mozilla-central/rev/b2015fdd464f598d645342614593d4ebda922d95/toolkit/components/url-classifier/nsUrlClassifierPrefixSet.cpp#99 Differential Revision: https://phabricator.services.mozilla.com/D30046
afbe5300acca5438c03282d21fb6e74088eff4ab: Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp
dlee <dlee@mozilla.com> - Tue, 14 May 2019 22:42:31 +0000 - rev 535874
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp The checksum calculating code is used to find the root cause of a crash bug during update(Bug 1362761). Since the algorithm will be update in these series of patches, we don't need to keep it. Differential Revision: https://phabricator.services.mozilla.com/D26667
f94b6f3a7fff9783dded2b404b84c0500c4182f2: Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp
dlee <dlee@mozilla.com> - Fri, 10 May 2019 17:40:11 +0000 - rev 535723
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1542744 - P3. Run the same prefixset testcases for different configuration. r=gcp This patch does the following: 1. Run the same prefixset tests when * browser.safebrowsing.prefixset.max_array_size = 0 * browser.safebrowsing.prefixset.max_array_size = UINT32_MAX This makes sure both of the methods to store prefixset are tested by existing testcases 2. Refine gtest with test fixture 3. Add TinySet and LargeSet testcases Differential Revision: https://phabricator.services.mozilla.com/D30338
c51b622bb1fe7e0d21df71a02c7f07f03f1e46fe: Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp
dlee <dlee@mozilla.com> - Tue, 14 May 2019 21:05:41 +0000 - rev 535722
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1542744 - P2. Improve performance of MakePrefixSet by using different algorithm according to the number of prefixes. r=gcp The goal of this patch is to reduce the number of memory reallocation during |MakePrefixSet|[1]. Here is the number of nsTArray memory reallocation occur during |MakePrefixSet| (test in my local platform): googpub-phish-proto: 58k times goog-malware-proto: 9k times goog-unwanted-proto: 25k times goog-badbinurl-proto: 6k times This patch improves the performance by: 1. For tables whose prefixes are less than 128*1024(malware, unwanted, badinurl). Store prefixes directly without dividing allocation into smaller chunks. Because the maximum size to store all the prefixes in a single array for these tables will be less than 512k, we can avoid Bug 1046038. This simplifies the internal prefixset data structure generation and total memory usage is also saved: goog-malware-proto : 437K -> 163k goog-unwanted-proto : 658k -> 446k goog-badbinurl-proto: 320k -> 110k The single largest allocated continuous memory size is: goog-malware-proto : 86k -> 163k goog-unwanted-proto : 86k -> 446k goog-badbinurl-proto: 77k -> 110k A further improvement can be done for this part is for tables with fewer prefixes, we can use an one-dimension delta array to reduce the size of a single continuous memory allocation. 2. For tables with more prefixes: According to experiment, when prefixes are more than 400k the delta arrays have very high chance that are full, in the case of phishing table, we can estimate the capacity accurately before applying delta algorithm. The shortcoming of this part is when prefixes are between 130k~400k, the capacity estimation is not accurate. [1] https://searchfox.org/mozilla-central/rev/b2015fdd464f598d645342614593d4ebda922d95/toolkit/components/url-classifier/nsUrlClassifierPrefixSet.cpp#99 Differential Revision: https://phabricator.services.mozilla.com/D30046
aedbe6cdd06fdda240d262c366c594a463df71d0: Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp
dlee <dlee@mozilla.com> - Wed, 08 May 2019 08:35:06 +0000 - rev 535721
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1542744 - P1. Remove calculating checksum for mIndexDelta array. r=gcp The checksum calculating code is used to find the root cause of a crash bug during update(Bug 1362761). Since the algorithm will be update in these series of patches, we don't need to keep it. Differential Revision: https://phabricator.services.mozilla.com/D26667
3d3527abff681615f5a3dde9bd844b60b80f292d: Bug 1551399 part 2. Stop using [array] in url-classifier's makeFindFullHashRequestV4. r=dimi,gcp
Boris Zbarsky <bzbarsky@mit.edu> - Tue, 14 May 2019 09:57:16 +0000 - rev 535655
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1551399 part 2. Stop using [array] in url-classifier's makeFindFullHashRequestV4. r=dimi,gcp Differential Revision: https://phabricator.services.mozilla.com/D31022
d4ee430e90c2c4b21d13b2f1de063928f1b53d1d: Bug 1551399 part 1. Stop using [array] in url-classifier's makeUpdateRequestV4. r=dimi,gcp
Boris Zbarsky <bzbarsky@mit.edu> - Tue, 14 May 2019 09:50:42 +0000 - rev 535654
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1551399 part 1. Stop using [array] in url-classifier's makeUpdateRequestV4. r=dimi,gcp Differential Revision: https://phabricator.services.mozilla.com/D31020
0c7f3a7403bd17f2d0bb2d17cba47ea02b86c12a: Bug 1547732 - Use LOAD_BYPASS_URL_CLASSIFIER flag for download protection ping. r=gcp
dlee <dlee@mozilla.com> - Mon, 29 Apr 2019 17:36:44 +0000 - rev 533760
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1547732 - Use LOAD_BYPASS_URL_CLASSIFIER flag for download protection ping. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D29231
75d2b35c092f4b77c3855569e5dd8b6803e8e914: Bug 1543790 - Fix RDD sandboxing conditions so the parent and child processes agree. r=gcp
Jed Davis <jld@mozilla.com> - Tue, 16 Apr 2019 13:53:20 +0000 - rev 531673
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1543790 - Fix RDD sandboxing conditions so the parent and child processes agree. r=gcp If the system doesn't support seccomp-bpf, the parent process won't try to set up sandboxing, but the child process has a separate check that didn't test for this, and ends up failing a release assertion (in SandboxReporterClient, but we also release-assert that installing the seccomp-bpf policy succeeds). This patch just fixes the child-side conditional to match the intended behavior, but in the long term we should consider redesigning SandboxInfo to avoid this. Differential Revision: https://phabricator.services.mozilla.com/D27624
01d9700306a4babdee436d402b4bed5f37b1ec2a: Bug 1543858 - Adjust Linux sandbox policies to tolerate glibc's qsort. r=gcp
Jed Davis <jld@mozilla.com> - Tue, 16 Apr 2019 06:50:50 +0000 - rev 531672
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1543858 - Adjust Linux sandbox policies to tolerate glibc's qsort. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D27632
85e806016727c9b063d2c79e0e607301bad352fa: Bug 1543319 - P2. Swap the byte order in-place. r=gcp
dlee <dlee@mozilla.com> - Wed, 10 Apr 2019 20:53:17 +0000 - rev 530778
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1543319 - P2. Swap the byte order in-place. r=gcp We don't need an additional array just for byte reordering, replace it with in-place processing. Testcase are modified because the LookupCacheV4::Build API now clears the input parameter. Differential Revision: https://phabricator.services.mozilla.com/D26861
95e20c4ed6c3423ee9c7cdebb52525ca3f3594f5: Bug 1543319 - P1. Free intermediate memory as early as possible during Safe Browsing update. r=gcp
dlee <dlee@mozilla.com> - Wed, 10 Apr 2019 14:32:54 +0000 - rev 530777
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1543319 - P1. Free intermediate memory as early as possible during Safe Browsing update. r=gcp Here is the flow how prefixes are handled during an V4 update: 1. Prefixes are received from Safe Browsing update, stored in ProtocolBuffer 2. Copy the prefixes from ProtocolBuffer to TableUpdate structure 3. Prefixes in TableUpdate are merged with local prefixes (stored in LookupCacheV4) 4. Merged prefixes are processes by PrefixSet to generate the in-memory prefix set data structure (MakePrefixSet). In this patch, we free the prefixes stored in TableUpdate right after step3. This reduces the peak memory used during an update (peak happens in step 4). Differential Revision: https://phabricator.services.mozilla.com/D26860
c78da14598ce7727d70e51700bf7afee901aa0e7: Bug 1535304 - Remove provider from about:url-classifier if no table is being used. r=gcp
dlee <dlee@mozilla.com> - Thu, 21 Mar 2019 07:54:20 +0000 - rev 527263
Push 2082 by ffxbld-merge at Mon, 01 Jul 2019 08:34:18 +0000
Bug 1535304 - Remove provider from about:url-classifier if no table is being used. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D24290
9bac094cec23aa298e04e8326cbd8d9b558575f0: Bug 1353956 - P7. Add gtest to ensure .pset is correctly loaded and removed. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:43:16 +0000 - rev 523892
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P7. Add gtest to ensure .pset is correctly loaded and removed. r=gcp Differential Revision: https://phabricator.services.mozilla.com/D22490
a87bd3f9b87d37bfb3b2ee5137c58bfa669ce1cb: Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:42:31 +0000 - rev 523891
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp To avoid forcing a redownload of SafeBrowsing v4 list. Differential Revision: https://phabricator.services.mozilla.com/D21876
aaba7c25b72b9485fc6c71b879cbc4096f3f4f2b: Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:41:52 +0000 - rev 523890
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp This patch is to cleanup old SafeBrowsing v4 prefix files. Differential Revision: https://phabricator.services.mozilla.com/D21464
3b5da75b9c7b7912e31007b9bac31b4a1ff4a558: Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:41:25 +0000 - rev 523889
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp After this patch, we may have the following files in SafeBrowsing directory: - (v2) .sbstore : Store V2 chunkdata, for update, MD5 integrity check while load - (v2) .pset : Store V2 prefixset, for lookup, load upon startup, no integrity check - (v4) .metadata : Store V4 state, for update, no integrity check - (v4) .vlpset : Store V4 prefixset, for lookup, load upon startup, CRC32 integrity check - (v4) .pset : V4 prefix set before this patch, should be removed The magic string is also added to ".vlpset" header so we can add a telemetry to see if sanity check is good enough for prefix set integrity check (The telemetry is not yet added). If yes, we can remove the CRC32 in the future for even better performance. Differential Revision: https://phabricator.services.mozilla.com/D21463
e083106dc24f6f562eb2d72aa02871a27a681ebe: Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:40:56 +0000 - rev 523888
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp SafeBrowsing prefix files LOAD/SAVE operations are handled in xxxPrefixSet.cpp. It would be more clear if xxxPrefixSet.cpp only processes prefix data, while LookupCacheV2/LookupCacheV4 which use prefix set process file. This patch doesn't change any behavior, testcases need to update because the LookupCache & xxxPrefixSet APIs are changed. Differential Revision: https://phabricator.services.mozilla.com/D21462
c7a253aed4508a8df4f1309d889dbfd669cd1693: Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:40:28 +0000 - rev 523887
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp SHA256 is an expensive operation, we should avoid using them if possible. SafeBrowsing prefix files are loaded during startup and verify integrity with SHA256 which may affect the performance especially on the low-end device. This patch simply removes the SHA256 integrity check. CRC32 version integrity check will be introduced in the other patch. This patch also changes the behavior of recording "Telemetry::URLCLASSIFIER_VLPS_LOAD_CORRUPT" a little bit. It used to records only once per session(during startup, the first time we load prefix set), now it records per update. Differential Revision: https://phabricator.services.mozilla.com/D21461
c2331373e10707aa5259eb3d8436f65c8d59c684: Bug 1353956 - P1. Rename checksum used in SafeBrowsing V4 to SHA256. r=gcp
Dimi Lee <dlee@mozilla.com> - Thu, 07 Mar 2019 14:40:14 +0000 - rev 523886
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P1. Rename checksum used in SafeBrowsing V4 to SHA256. r=gcp SafeBrowsing V4 protocol use SHA-256 as the checksum to check integrity of update data and also the integrity of prefix files. SafeBrowsing V2 HashStore use MD5 as the checksum to check integrity of .sbstore Since we are going to use CRC32 as the integrity check of V4 prefix files, I think rename V4 "checksum" to SHA256 can improve readability. Differential Revision: https://phabricator.services.mozilla.com/D21460
71dafccc22ae493b656d9130cec82ba37d4babf7: Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp
dlee <dlee@mozilla.com> - Wed, 06 Mar 2019 09:41:34 +0000 - rev 523662
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P6. Load the old prefixset(.pset) when there is no .vlpset. r=gcp To avoid forcing a redownload of SafeBrowsing v4 list. Differential Revision: https://phabricator.services.mozilla.com/D21876
f1f29fe519cf169d43eeacfe710d0b84e9da55db: Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp
dlee <dlee@mozilla.com> - Tue, 05 Mar 2019 18:32:23 +0000 - rev 523661
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P5. Remove old v4 prefix files after new files are stored. r=gcp This patch is to cleanup old SafeBrowsing v4 prefix files. Differential Revision: https://phabricator.services.mozilla.com/D21464
4978556a66f66cd6d214f98aa656ea1e223ead63: Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Wed, 06 Mar 2019 22:57:12 +0000 - rev 523660
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P4. Add header and CRC32 checksum to SafeBrowsing V4 prefix files. r=gcp After this patch, we may have the following files in SafeBrowsing directory: - (v2) .sbstore : Store V2 chunkdata, for update, MD5 integrity check while load - (v2) .pset : Store V2 prefixset, for lookup, load upon startup, no integrity check - (v4) .metadata : Store V4 state, for update, no integrity check - (v4) .vlpset : Store V4 prefixset, for lookup, load upon startup, CRC32 integrity check - (v4) .pset : V4 prefix set before this patch, should be removed The magic string is also added to ".vlpset" header so we can add a telemetry to see if sanity check is good enough for prefix set integrity check (The telemetry is not yet added). If yes, we can remove the CRC32 in the future for even better performance. Differential Revision: https://phabricator.services.mozilla.com/D21463
bc0b91abce9bcb43ef501dfa7bca03ea678f9783: Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp
Dimi Lee <dlee@mozilla.com> - Mon, 04 Mar 2019 21:22:46 +0000 - rev 523659
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P3. Separate file processing and prefix data processing for SafeBrowsing prefix set. r=gcp SafeBrowsing prefix files LOAD/SAVE operations are handled in xxxPrefixSet.cpp. It would be more clear if xxxPrefixSet.cpp only processes prefix data, while LookupCacheV2/LookupCacheV4 which use prefix set process file. This patch doesn't change any behavior, testcases need to update because the LookupCache & xxxPrefixSet APIs are changed. Differential Revision: https://phabricator.services.mozilla.com/D21462
6b8412db5a05f12bf44b5abc1bf553ed356da41f: Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp
dlee <dlee@mozilla.com> - Thu, 28 Feb 2019 08:18:46 +0000 - rev 523658
Push 2032 by ffxbld-merge at Mon, 13 May 2019 09:36:57 +0000
Bug 1353956 - P2. Do not use SHA-256 while loading the V4 prefix files. r=gcp SHA256 is an expensive operation, we should avoid using them if possible. SafeBrowsing prefix files are loaded during startup and verify integrity with SHA256 which may affect the performance especially on the low-end device. This patch simply removes the SHA256 integrity check. CRC32 version integrity check will be introduced in the other patch. This patch also changes the behavior of recording "Telemetry::URLCLASSIFIER_VLPS_LOAD_CORRUPT" a little bit. It used to records only once per session(during startup, the first time we load prefix set), now it records per update. Differential Revision: https://phabricator.services.mozilla.com/D21461