summaryrefslogtreecommitdiff
path: root/lib/prism
AgeCommit message (Collapse)Author
42 hours[ruby/prism] Fix `kDO_LAMBDA` token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser::Lexer` ## Summary This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block. ### Parser gem (Expected) Returns `kDO_LAMBDA` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` ### `Prism::Translation::Parser` (Actual) Previously, the parser returned `kDO` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` After the update, the parser now returns `kDO_LAMBDA` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` ## Additional Information Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`. However, since `kDO` is already being returned in this case, there is no change in behavior. ### Parser gem Returns `tLAMBDA` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]], [:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]], [:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 23...25>]], [:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]] ``` ### `Prism::Translation::Parser` Returns `kDO` token: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]], [:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]], [:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO, ["do", #<Parser::Source::Range example.rb 23...25>]], [:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]] ``` As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`. In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR. https://github.com/ruby/prism/commit/2ee480654c
43 hours[ruby/prism] Introduce partial_script optionKevin Newton
https://github.com/ruby/prism/commit/b28877fa4f
6 days[ruby/prism] Do not leak explicit encodingKevin Newton
Fixes [Bug #20744] https://github.com/ruby/prism/commit/f1b8b1b2a2
9 days[ruby/prism] Expose main_script in serialization APIKevin Newton
https://github.com/ruby/prism/commit/0b527ca93f
10 days[ruby/prism] Check errno for parsing directoryKevin Newton
https://github.com/ruby/prism/commit/d68ea29d04 Notes: Merged: https://github.com/ruby/ruby/pull/11497
13 days[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument. ## Parser gem (Expected) Returns `tDSTAR` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tPOW` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` After the update, the parser now returns `tDSTAR` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` With this change, the following code could be removed from test/prism/ruby/parser_test.rb: ```diff - when :tPOW - actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR ``` `tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["a", #<Parser::Source::Range example.rb 0...1>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 2...4>]], [:tIDENTIFIER, ["b", #<Parser::Source::Range example.rb 5...6>]]] ``` https://github.com/ruby/prism/commit/66bde35a44
2024-09-07[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis. ## Parser gem (Expected) Returns `tLPAREN2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 \ -ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tLPAREN` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` After the update, the parser now returns `tLPAREN2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem. The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`. With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`: ```diff - when :tLPAREN - actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2 ``` https://github.com/ruby/prism/commit/04d6f3478d
2024-09-04[ruby/prism] essentialy ==> essentiallyydah
https://github.com/ruby/prism/commit/9c68c01bcd
2024-08-29[ruby/prism] Remove deprecated lib/prism/translation/parser/rubocop.rb fileKoichi ITO
Follow up https://github.com/ruby/prism/pull/2558. This PR removes deprecated lib/prism/translation/parser/rubocop.rb file. The file was a workaround to allow setting `TargetRubyVersion: 80_82_73_83_77.xx` until Prism (`Prism::Translation::Parser`) is integrated into RuboCop. RuboCop already supports Prism (`Prism::Translation::Parser`) as of https://github.com/rubocop/rubocop/pull/12724. It has been several months since the file was deprecated in ruby/prism#2558. And, yesterday, Prism 1.0.0 was released, but perhaps the file should have been removed before then. Although this might be seen as incompatible with semver 1.0.0, I think there is no longer a reason to keep the file. https://github.com/ruby/prism/commit/646a10270e
2024-08-28[PRISM] Sync version updateKevin Newton
2024-08-28[ruby/prism] Reverse-sync ruby/ruby and deprecate old fieldsKevin Newton
https://github.com/ruby/prism/commit/bc21c9f3ee
2024-08-28[PRISM] Field renamingKevin Newton
Rename some fields that do not quite make sense. * CaseMatchNode#consequent -> CaseMatchNode#else_clause * CaseNode#consequent -> CaseNode#else_clause * IfNode#consequent -> IfNode#subsequent * RescueNode#consequent -> RescueNode#subsequent * UnlessNode#consequent -> UnlessNode#else_clause Notes: Merged: https://github.com/ruby/ruby/pull/11480
2024-08-14[ruby/prism] Tweak inspect representation of `Prism::Location`Koichi ITO
This PR tweaks inspect representation of `Prism::Location`. ## Before During debugging, the meaning of `@location=https://github.com/ruby/prism/commit/21474836481` was unclear: ```console $ ruby -Ilib -rprism -e 'p Prism.lex("puts :hi").value.map(&:first)[1]' #<Prism::Token:0x000000010cd74e40 @source=#<Prism::ASCIISource:0x000000010cb5f808 @source="puts :hi", @start_line=1, @offsets=[0]>, @type=:SYMBOL_BEGIN, @value=":", @location=https://github.com/ruby/prism/commit/21474836481> ``` ## After This PR clarifies the contents of the location object, aligning with what I think user expects: ```console $ ruby -Ilib -rprism -e 'p Prism.lex("puts :hi").value.map(&:first)[1]' #<Prism::Token:0x000000010e174d50 @source=#<Prism::ASCIISource:0x000000010df5efe8 @source="puts :hi", @start_line=1, @offsets=[0]>, @type=:SYMBOL_BEGIN, @value=":", @location=#<Prism::Location @start_offset=5 @length=1 start_line=1>> ``` Although it is uncertain whether Prism will accept this change in the inspect representation, it is submitted here as a suggestion. https://github.com/ruby/prism/commit/e7421ce1c5
2024-07-11[ruby/prism] (ruby_parser) Handle bare string in implicit concatKevin Newton
https://github.com/ruby/prism/commit/afc7c9344a
2024-07-11[ruby/prism] (ruby_parser) Match match3 nodes for regular expressionsKevin Newton
https://github.com/ruby/prism/commit/47cb73ce69
2024-07-11[ruby/prism] (ruby_parser) Reverse associativity of and and or nodesKevin Newton
https://github.com/ruby/prism/commit/073e8ba307
2024-07-11[ruby/prism] Various cleanup for initializers and typechecksKevin Newton
https://github.com/ruby/prism/commit/86cf82794a
2024-07-11[ruby/prism] Add node ids to nodesKevin Newton
https://github.com/ruby/prism/commit/bf16ade7f9
2024-07-11[ruby/prism] Move Node#type and Node::type documentationKevin Newton
https://github.com/ruby/prism/commit/08a71f6259
2024-07-11[ruby/prism] Expose flags on every node typeKevin Newton
https://github.com/ruby/prism/commit/9f12a56fd6
2024-07-11[ruby/prism] Move location to second position for node initializersKevin Newton
https://github.com/ruby/prism/commit/4cc0eda4ca
2024-07-11[ruby/prism] Reconfigure error testsKevin Newton
https://github.com/ruby/prism/commit/fb7e1ebb7f
2024-06-18[ruby/prism] (parser) Fix up tokens for empty symbolKevin Newton
https://github.com/ruby/prism/commit/5985ab7687
2024-06-10[ruby/prism] Provide ability to lock encoding while parsingKevin Newton
https://github.com/ruby/prism/commit/f7faedfb3f
2024-06-07Sync prism version to latestKevin Newton
2024-06-07[ruby/prism] Handle chomped bytesize with lines without newlinesKevin Newton
https://github.com/ruby/prism/commit/1528d3c019
2024-06-07[ruby/prism] Ensure inner heredoc nodes have the correct locationKevin Newton
https://github.com/ruby/prism/commit/100340bc6b
2024-06-07[ruby/prism] Use correct newlines for heredoc inner linesKevin Newton
https://github.com/ruby/prism/commit/4a9a7a62af Co-authored-by: Jason Kim <jasonkim@github.com> Co-authored-by: Adam Hess <HParker@github.com>
2024-06-05[ruby/prism] Remove unused string list structKevin Newton
https://github.com/ruby/prism/commit/36c6851c85
2024-06-04[ruby/prism] (parser) split up regexp content by linesKevin Newton
https://github.com/ruby/prism/commit/85b4a5f804
2024-06-04[ruby/prism] (parser) handle quoted symbols in hash patternsKevin Newton
https://github.com/ruby/prism/commit/461aa5e658
2024-06-04[ruby/prism] (parser) fix up srange_find to anchor at the start of the sliceKevin Newton
https://github.com/ruby/prism/commit/aecce571d8
2024-06-04[ruby/prism] (parser) fix up nested multi writeKevin Newton
https://github.com/ruby/prism/commit/12e079c97e
2024-06-04[ruby/prism] Revert "Revert "Properly destructure procarg0 in parser ↵Kevin Newton
translation"" This reverts commit https://github.com/ruby/prism/commit/d8ae19d0334a. https://github.com/ruby/prism/commit/df1eda2811
2024-06-03[ruby/prism] Revert "Properly destructure procarg0 in parser translation"Kevin Newton
This reverts commit https://github.com/ruby/prism/commit/823e931ff230. https://github.com/ruby/prism/commit/d8ae19d033
2024-06-03[ruby/prism] Properly destructure procarg0 in parser translationKevin Newton
https://github.com/ruby/prism/commit/823e931ff2
2024-06-03[ruby/prism] Fix up steep issuesKevin Newton
https://github.com/ruby/prism/commit/21d314eb2e
2024-05-31[ruby/prism] Fix up heredoc location translation for parserKevin Newton
https://github.com/ruby/prism/commit/a4e164e22b
2024-05-31[ruby/prism] Match % strings in parserKevin Newton
https://github.com/ruby/prism/commit/840185110f
2024-05-31[ruby/prism] Use correct opening and closing parenthesis for array pattern ↵Kevin Newton
in parser https://github.com/ruby/prism/commit/beed43922c
2024-05-31[ruby/prism] Match match_hash_var when quotes are usedKevin Newton
https://github.com/ruby/prism/commit/f2a327449a
2024-05-31[ruby/prism] Match parser for match_rest in patternKevin Newton
https://github.com/ruby/prism/commit/785de2c39d
2024-05-30[ruby/prism] Tests overhaulKevin Newton
https://github.com/ruby/prism/commit/6f886be0a4
2024-05-24[ruby/prism] Fix up ruby_parser string concatKevin Newton
https://github.com/ruby/prism/commit/4b06eae0df
2024-05-24[ruby/prism] Remove Debug moduleKevin Newton
https://github.com/ruby/prism/commit/4d8929ff6a
2024-05-24[ruby/prism] Remove dynamic Debug module methodsKevin Newton
https://github.com/ruby/prism/commit/b850794db9
2024-05-22[ruby/prism] Fix support for 'it' implicit local variableKevin Newton
https://github.com/ruby/prism/commit/53bbcfe513
2024-05-21[ruby/prism] Fix up ruby_parser interpolation concatenationKevin Newton
https://github.com/ruby/prism/commit/79cec4be22
2024-05-21[ruby/prism] Provide deprecated numeric methodKevin Newton
https://github.com/ruby/prism/commit/77bf9038a2
2024-05-21[ruby/prism] Reconfigure rationalsKevin Newton
This eliminates the subnode on RationalNode and replaces it with two integer fields, which represent the ratio for the rational. It also reduces those two integers if they both fit into 32 bits. Importantly, this PR does not implement bignum reduction. That's something I'd like to consider for the future, but it's simple enough for now to leave them unreduced, which makes it more useful than it used to be. https://github.com/ruby/prism/commit/86e06c7068
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-22 11:09:03 +0000