js/src/frontend/TokenStream.h
1fb7ddfad86d5e085c4f2af23a2519d37e45a3e4
created 2018-08-24 21:25 -0500
pushed 2018-08-29 09:43 +0000
Jeff Walden Jeff Walden - Bug 1486577 - Make all users of the various *CompileOptions classes #include "js/CompileOptions.h" so that nothing but that file has to know about these classes having a JS_PUBLIC_API on them, that would have to be present in forward-declarations. r=jandem
805445fc768f0159334f0e066f62c543be14c2d8
created 2018-07-25 20:11 -0700
pushed 2018-07-31 09:53 +0000
Jeff Walden Jeff Walden - Bug 1478587 - Return the result of CodeUnitValue(), not a CharT, from GeneralTokenStreamChars::getCodeUnit. r=arai
bb0dd8b2d4e2cb93766120a7ef82746892639f26
created 2018-07-25 20:11 -0700
pushed 2018-07-31 09:53 +0000
Jeff Walden Jeff Walden - Bug 1478587 - Invoke TokenStreamChars::getNonAsciiCodePointDontNormalize passing CharT lead (not int32_t). r=arai
fd12c0a357b378feaf48266389bfa0e5c27d766b
created 2018-07-29 13:33 +0300
pushed 2018-07-30 09:52 +0000
shindli shindli - Backed out 11 changesets (bug 1478892, bug 1478587) for build bustages in ../../../dist/bin/gdb-tests on a CLOSED TREE
9546bfc93c7af1391bbcf901ed1ddc1304da8b80
created 2018-07-25 20:11 -0700
pushed 2018-07-30 09:52 +0000
Jeff Walden Jeff Walden - Bug 1478587 - Return the result of CodeUnitValue(), not a CharT, from GeneralTokenStreamChars::getCodeUnit. r=arai
a47cd872a700a331b13c598a26ec956a28d14308
created 2018-07-25 20:11 -0700
pushed 2018-07-30 09:52 +0000
Jeff Walden Jeff Walden - Bug 1478587 - Invoke TokenStreamChars::getNonAsciiCodePointDontNormalize passing CharT lead (not int32_t). r=arai
dad6f1d062719f26f6f934c2e8a5225ab2491351
created 2018-07-17 21:21 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Make atomizeSourceChars take a Span<CharT>, and define a specialization for UTF-8. r=arai
38216fdba3ddea982cc4d3178c3b2c75637f71a1
created 2018-07-17 21:13 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Move TokenStreamChars::getCodePoint to a generalized TokenStreamSpecific::getCodePoint that works for any CharT. r=arai
107211d728e60e9b069c2d0107241f74b575d576
created 2018-06-29 13:46 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Implement TokenStreamChars::getNonAsciiCodePoint for UTF-8. r=arai
46c400e4a018c1cb827266441018a5c43a84f8f5
created 2018-07-09 16:21 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Implement consumeRestOfSingleLine for UTF-8. r=arai
45b34e5af77e953970b0497f1ab20690333457d8
created 2018-07-18 22:46 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Implement getNonAsciiCodePointDontNormalize for UTF-8. r=arai
292bd4af4056fd252939f0b9a31ad71ce9270785
created 2018-07-25 14:51 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Implement SourceUnits::peekCodePoint for UTF-8. r=arai
0ccbc10cd6a1b7ce53f852fb8f7f81cac1ecc31f
created 2018-07-25 14:46 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Replace SpecializedTokenStreamCharsBase::consumeRestOfSingleLineComment with an infallible SourceUnits::consumeRestOfSingleLineComment, in light of the fact that the UTF-8 version can just stop consuming code points when an encoding error is encountered, leaving error-reporting to whatever actually tries to consume the first code point of the "next" line. r=arai
66952e88e68c22576f6ddc0600d0883d0b7dba6a
created 2018-07-25 14:24 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478170 - Make SourceUnits::findWindow{Start,End} const because they can/should be. r=arai
d93c6f631ca76e457c819b7bd74039395c33fded
created 2018-07-25 14:24 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478045 - Remove TokenStreamChars::{ungetCodePointIgnoreEOL,ungetNonAsciiNormalizedCodePoint} as unused now that SourceUnits::{peek,consumeKnown}CodePoint have replaced them. r=arai
b4a63f805c039170375033eff016d850192cf3d6
created 2018-07-25 14:24 -0700
pushed 2018-07-26 21:58 +0000
Jeff Walden Jeff Walden - Bug 1478045 - Implement SourceUnits::{peek,consumeKnown}CodePoint for the uncommon cases where a code point must be gotten, tested against one or more predicates, then sometimes ungotten based on those predicates. (This is unfortunately a bit subtle, but getting and ungetting is arguably worse, because ungetting has to unget a variable number of code units -- whereas peeking can compute that number of code units and then use it directly when the peeked code point is consumed, avoiding double-computation and increased potential for error.) r=arai
735ee4ba2523472561514447de4e7558efae9e1b
created 2018-07-17 21:13 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Move TokenStreamChars::getFullAsciiCodePoint to GeneralTokenStreamChars::getFullAsciiCodePoint, because its algorithm is identical for UTF-8 and UTF-16. r=arai
7da01c72860d3461d177fc3a07cbb8f0b51e6c27
created 2018-07-17 21:00 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Replace SourceUnits<CharT>::isRawEOLChar (where CharT had rather uncertain meaning) with standalone IsLineTerminator of clearer meaning. r=arai
670d911efa814f0b6f042bdf859abf666e4d510a
created 2018-07-17 21:01 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Implement TokenStreamCharsBase::addLineOfContext that does all the work of creating a syntax-error window and works for either CharT. r=arai
30c6932eea32e4b49abb30f20b1e49c804734731
created 2018-07-13 18:59 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Move error-message window-start computation into SourceUnits::findWindowStart so that specialized versions can be provided for UTF-8 and UTF-16. r=arai
ff1852065b5c6a4f2a1829b290f91dae26f7804f
created 2018-07-10 12:06 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Rename SourceUnits::findEOLMax to SourceUnits::findWindowEnd, and make it include only valid UTF-16. r=arai
6ac4c8ad6b9a20ba54462a2e877c594df8767401
created 2018-07-18 18:05 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Move TokenStreamSpecific::computeLineOfContext to GeneralTokenStreamChars::internalComputeLineOfContext, and beef up its doc comment a little. r=arai
96e6f8645b7c00eb4bd9eab952c2c24ab1ed9ec2
created 2018-07-09 16:11 -0700
pushed 2018-07-22 09:46 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Implement a bare-bones TokenStreamChars specialization for UTF-8, into which subsequent UTF-8-centric functions can be added. r=arai
7d07cfa666bf8bb2ec9aef3c7f88654f66e9a433
created 2018-06-29 13:46 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Move fillCharBufferWithTemplateStringContents into TokenStreamCharsBase so that a UTF-8 specialization can eventually be defined for it. r=arai
4bd4c0e74bc6ce7ef610ef902085a3609b9517f8
created 2018-07-17 21:00 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Make matchCodeUnit only accept ASCII, and split matchLineTerminator (for '\r' and '\n') out of it. r=arai
80b3a14e84c23a7215243376d1a3143a985aee8a
created 2018-07-09 16:22 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Remove ungetLineTerminator, used only to unget Unicode separators, and replace it with a SourceUnits::ungetLineOrParagraphSeparator. r=arai
3a4e6ae59b597084afaed1d84d1674ee556406d5
created 2018-07-09 14:38 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Add a getNonAsciiCodePointDontNormalize for use in situations that demand such. r=arai
44b64b5a44fcfdaf086bd32b7c8038efac5bf652
created 2018-06-29 13:46 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Move consumeRestOfSingleLineComment into TokenStreamChars (and the infallible UTF-16 implementation into SpecializedTokenStreamCharsBase) so that it can be easily specialized for UTF-8. r=arai
9ed1d40847424b11e2bc6b5cbc7e497753c81938
created 2018-07-06 17:12 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Add a SpecializedTokenStreamCharsBase class to the hierarchy as a clear collection point for functions entirely specific to UTF-8 or UTF-16 tokenizing only. r=arai
1b604df36960311748f612b3dd01597938977d77
created 2018-07-17 21:00 -0700
pushed 2018-07-20 09:57 +0000
Jeff Walden Jeff Walden - Bug 1476866 - Fix the TokenStreamChars::getNonAsciiCodePoint doc comment to be clearer about |lead| being non-ASCII but still potentially (for UTF-16) a full code point. r=arai
1031a09274e0bc148e11f0dae2426e64e107c69a
created 2018-06-28 20:18 -0700
pushed 2018-07-06 09:47 +0000
Jeff Walden Jeff Walden - Bug 1472066 - Define UTF-8-specific versions of certain tokenizing helper functions, complementing the existing UTF-16 versions. r=arai
25d29385236c0062dc73e2b7199de1dc40d702e0
created 2018-07-02 19:16 -0700
pushed 2018-07-03 21:57 +0000
Jeff Walden Jeff Walden - Bug 1472569 - |this->|-qualify a bunch of function calls on |using Base::foo| members that gcc 8 completely chokes on and grossly misunderstands. r=arai
29dc2c52071353c63e34f90f36fe9aefd4b07c0c
created 2018-06-28 20:17 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472066 - Add some helper functions to enable (once the non-integral Utf8Unit code unit type lands, soon) dealing with Utf8Unit as the type of UTF-8 source code in addition to char16_t for UTF-16 source code. r=arai
234fc6b955c94397b0d17f1b9b273315b91b92bc
created 2018-06-28 20:14 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472066 - Specialize TokenStreamCharsBase::fillCharBufferWithTemplateStringContents for char16_t now that its alternative UTF-8 implementation will have to function a bit differently to write data into a char16_t charBuffer. r=arai
ba24991f7f4d976ee709a23d62b91fb39d4641d6
created 2018-06-28 16:08 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472031 - Add SourceUnits::matchHexDigits and remove SourceUnits::peekCodeUnits used for the same ends (but that also performed a needless copy). r=arai
dad92f9be9b723945131b9065d30333de2f0cd13
created 2018-06-28 15:39 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472031 - Avoid copying potential directives in TokenStreamSpecific::getDirective. r=arai
d857b06f51caedfb9c5844b82f1308effe2a7b4b
created 2018-06-28 15:34 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472031 - Implement a SourceUnits::remaining() function exposing how many code units are left. r=arai
481871dcc4888e3e67aa4bf1864a78dc41996744
created 2018-06-28 15:33 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472031 - Define TokenStreamAnyChars::updateFlagsForEOL inline in the header, for simplicity. r=arai
08cf6e4ccccec3eb39d8d5cb3a2989a547b50cf0
created 2018-06-28 15:32 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472031 - Rename atomizeChars to atomizeSourceChars to clarify it's only for atomizing directly from source text. r=arai
b64aec5cf563ec12e1ec505a769389ed0cf6e936
created 2018-06-28 15:31 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1472031 - Introduce a drainCharBufferIntoAtom function to clear charBuffer and create an atom from the chars that were in it. r=arai
4396a31f09b5e3b49204cd6e9ca14e1d41ca366d
created 2018-06-28 02:30 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Change a bunch of 'character' nomenclature in token stream code to 'code unit', completing the transition from tokenizing by UTF-16 'character' to tokenizinng by UTF-8/16 code unit. (There are straggling places where algorithms will need to be specialized for UTF-8, or functions will need to move within the TokenStream* hierarchy to permit such; but what is in the tree now universally acts on code units first, full code points second.) r=arai
5d9c420a912ba9fbfcb4d3d6c20f9233c69dbb13
created 2018-06-28 02:27 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Remove GeneralTokenStreamChars::{,un}getChar now that they're unused. r=arai
f3b134e1e8f154a9df09490a459b8cf6fc218c34
created 2018-06-27 13:26 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1471464 - Make getNonAsciiCodePoint, getFullAsciiCodePoint, and isAsciiCodePoint all take |int32_t| and not |CharT|, because they're all passed |int32_t| from |getCodeUnit()| or a code point-getting mechanism. r=arai
e20c1d6852e4fb05f4fce71523d698cb9ebf28fe
created 2018-06-27 13:25 -0700
pushed 2018-06-30 21:52 +0000
Jeff Walden Jeff Walden - Bug 1471463 - Move TokenStreamCharsBase::tokenbuf and related functions into a new TokenStreamCharsShared base class, make the character type always char16_t, and rename it/them to 'charBuffer' for clarity. r=arai
67c0442a7a44a0618504911b3239aa0117b3d4ee
created 2018-06-26 12:02 -0400
pushed 2018-06-29 09:42 +0000
Ted Campbell Ted Campbell - Bug 1471272 - Add [SMDOC] tags for in-source documentation. r=jandem,nbp
6a76baf321797f3831aeffaab4eb852e9383f213
created 2018-06-23 16:14 +0300
pushed 2018-06-23 21:45 +0000
Andi-Bogdan Postelnicu Andi-Bogdan Postelnicu - Bug 1453795 - JS - Initialize member fields in classes/ structures. r=waldoo
6e75896cb76c94fa2e8d89d7a7a1ed0237575f52
created 2018-06-21 23:23 -0700
pushed 2018-06-22 21:50 +0000
Jeff Walden Jeff Walden - Backed out changeset efab409f7335. r=bustage in a CLOSED TREE
efab409f73352a925744a2c9be5a61e741788b0d
created 2018-06-18 11:55 -0700
pushed 2018-06-22 21:50 +0000
Jeff Walden Jeff Walden - Bug 1468877 - Remove TokenStreamShared::ntokens as being duplicative in meaning of TokenStreamShared::maxLookahead. r=jorendorff
8949f1ff42ac2e850f61537cd0cb3da3eee11995
created 2018-06-09 19:49 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Move a couple small functions to be defined inline in TokenStream class bodies, not out-of-line in TokenStream.cpp, for readability. r=arai
36f3b5eb7294925bd97e77a9023234210bad928c
created 2018-06-09 03:59 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Make TokenStreamSpecific::getTokenInternal itself deal only in code units. r=arai
57a1edba27d75ed4d748d8b6fae992e82baeaef7
created 2018-06-09 03:36 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Make TokenStreamSpecific::identifierName consume in terms of code units. r=arai
96438f0ed68a8897e2e37c155c8527f98d0c413e
created 2018-06-09 03:36 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Refactor Unicode-escape identifier matching to work in terms of code units. This also eliminates ungets-then-skips that cancel out when a valid escape is matched. r=arai
96bab59f48e0323c926d42e14f271bc666f3b96a
created 2018-06-09 03:36 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Move Unicode escape processing from TokenStreamSpecific to GeneralTokenStreamChars. r=arai
b04275fa34e65d808ebc20790060f942b2606a55
created 2018-06-09 03:36 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Make TokenStreamSpecific::decimalNumber work by consuming code units. r=arai
18c217828e1b5d7ce2b945ac10ba935592f60f7d
created 2018-06-09 03:36 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Convert TokenStreamSpecific::getDirective act on code units, and remove TokenStreamSpecific::peekChar. r=arai
2b5d5815e7ef2cb3fbfb0f62c3175a418dd8ad52
created 2018-06-08 11:48 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Move skipChars to SourceUnits::skipCodeUnits, and make it efficient in the obvious way. r=arai
673d93f7c030c02f2ed3d70d77a7a2ba9e017312
created 2018-06-08 11:47 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Change SourceUnits::atStart to indicate being at the start of the source units of code, not at the start of the *containing* source. r=arai
03e60a8119191f315216a7a5a0b0431264ae5467
created 2018-06-08 11:46 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Invert SourceUnits::hasRawChars's meaning and rename it to SourceUnits::atEnd. r=arai
3d10297c43844a8941879a91b9574000fb3f6f8a
created 2018-06-08 11:35 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Rename TokenStreamSpecific::peekChars to SourceUnits::peekCodeUnits and optimize its code in now-obvious ways. r=arai
82729f89bc6111c8506901c46f8baa33ba1fe364
created 2018-06-07 22:48 -0700
pushed 2018-06-14 09:44 +0000
Jeff Walden Jeff Walden - Bug 1467336 - Make much of TokenStreamSpecific::getStringOrTemplateToken process by code unit, not by code-unit-generally-assumed-to-be-code-point. r=arai
less more (0) -300 -100 -60 tip