summaryrefslogtreecommitdiff
path: root/test/prism/ruby
AgeCommit message (Collapse)Author
2025-12-08Bump Prism to v1.5.2Takashi Kokubun
[Backport #21187]
2025-09-12Bump Prism version to 1.5.0Takashi Kokubun
2024-10-16[ruby/prism] RelocationKevin Newton
https://github.com/ruby/prism/commit/5ea6042408
2024-10-11[ruby/prism] Prism::StringQueryKevin Newton
Introduce StringQuery to provide methods to access some metadata about the Ruby lexer. https://github.com/ruby/prism/commit/d3f55b67b9
2024-10-10[ruby/prism] Prism::CodeUnitsCacheKevin Newton
Calculating code unit offsets for a source can be very expensive, especially when the source is large. This commit introduces a new class that wraps the source and desired encoding into a cache that reuses pre-computed offsets. It performs quite a bit better. There are still some problems with this approach, namely character boundaries and the fact that the cache is unbounded, but both of these may be addressed in subsequent commits. https://github.com/ruby/prism/commit/2e3e1a4d4d
2024-10-09[ruby/prism] Attempt to assume binary is UTF-8Kevin Newton
https://github.com/ruby/prism/commit/343197e4ff
2024-10-09[ruby/prism] Avoid breaking code units offset on binary encodingVinicius Stock
https://github.com/ruby/prism/commit/25a4cf6794 Co-authored-by: Kevin Newton <kddnewton@users.noreply.github.com>
2024-09-20[ruby/prism] Fix `kDO_LAMBDA` token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser::Lexer` ## Summary This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block. ### Parser gem (Expected) Returns `kDO_LAMBDA` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` ### `Prism::Translation::Parser` (Actual) Previously, the parser returned `kDO` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` After the update, the parser now returns `kDO_LAMBDA` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` ## Additional Information Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`. However, since `kDO` is already being returned in this case, there is no change in behavior. ### Parser gem Returns `tLAMBDA` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]], [:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]], [:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 23...25>]], [:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]] ``` ### `Prism::Translation::Parser` Returns `kDO` token: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]], [:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]], [:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO, ["do", #<Parser::Source::Range example.rb 23...25>]], [:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]] ``` As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`. In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR. https://github.com/ruby/prism/commit/2ee480654c
2024-09-09[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument. ## Parser gem (Expected) Returns `tDSTAR` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tPOW` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` After the update, the parser now returns `tDSTAR` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` With this change, the following code could be removed from test/prism/ruby/parser_test.rb: ```diff - when :tPOW - actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR ``` `tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["a", #<Parser::Source::Range example.rb 0...1>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 2...4>]], [:tIDENTIFIER, ["b", #<Parser::Source::Range example.rb 5...6>]]] ``` https://github.com/ruby/prism/commit/66bde35a44
2024-09-07[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis. ## Parser gem (Expected) Returns `tLPAREN2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 \ -ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tLPAREN` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` After the update, the parser now returns `tLPAREN2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem. The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`. With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`: ```diff - when :tLPAREN - actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2 ``` https://github.com/ruby/prism/commit/04d6f3478d
2024-06-18[ruby/prism] (parser) Print when token tests are now passingKevin Newton
https://github.com/ruby/prism/commit/9e4fb665ee
2024-06-18[ruby/prism] (parser) Fix up tokens for empty symbolKevin Newton
https://github.com/ruby/prism/commit/5985ab7687
2024-06-10[ruby/prism] Provide ability to lock encoding while parsingKevin Newton
https://github.com/ruby/prism/commit/f7faedfb3f
2024-06-07[ruby/prism] Handle chomped bytesize with lines without newlinesKevin Newton
https://github.com/ruby/prism/commit/1528d3c019
2024-06-07[ruby/prism] Document that nested heredocs are not properly parsed for parserKevin Newton
https://github.com/ruby/prism/commit/d218e65561
2024-06-07[ruby/prism] Ensure inner heredoc nodes have the correct locationKevin Newton
https://github.com/ruby/prism/commit/100340bc6b
2024-06-07[ruby/prism] Use correct newlines for heredoc inner linesKevin Newton
https://github.com/ruby/prism/commit/4a9a7a62af Co-authored-by: Jason Kim <jasonkim@github.com> Co-authored-by: Adam Hess <HParker@github.com>
2024-06-04[ruby/prism] (parser) mark spanning newlines as incorrectKevin Newton
https://github.com/ruby/prism/commit/09ba678066
2024-06-04[ruby/prism] (parser) split up regexp content by linesKevin Newton
https://github.com/ruby/prism/commit/85b4a5f804
2024-06-04[ruby/prism] (parser) handle quoted symbols in hash patternsKevin Newton
https://github.com/ruby/prism/commit/461aa5e658
2024-06-04[ruby/prism] (parser) mark unary_num_pow_precedence as incorrectKevin Newton
https://github.com/ruby/prism/commit/c49efdf824
2024-06-04[ruby/prism] (parser) fix up nested multi writeKevin Newton
https://github.com/ruby/prism/commit/12e079c97e
2024-06-04[ruby/prism] Revert "Revert "Properly destructure procarg0 in parser ↵Kevin Newton
translation"" This reverts commit https://github.com/ruby/prism/commit/d8ae19d0334a. https://github.com/ruby/prism/commit/df1eda2811
2024-06-03[ruby/prism] Revert "Properly destructure procarg0 in parser translation"Kevin Newton
This reverts commit https://github.com/ruby/prism/commit/823e931ff230. https://github.com/ruby/prism/commit/d8ae19d033
2024-06-03[ruby/prism] Properly destructure procarg0 in parser translationKevin Newton
https://github.com/ruby/prism/commit/823e931ff2
2024-05-31[ruby/prism] Fix up heredoc location translation for parserKevin Newton
https://github.com/ruby/prism/commit/a4e164e22b
2024-05-31[ruby/prism] Match % strings in parserKevin Newton
https://github.com/ruby/prism/commit/840185110f
2024-05-31[ruby/prism] Use correct opening and closing parenthesis for array pattern ↵Kevin Newton
in parser https://github.com/ruby/prism/commit/beed43922c
2024-05-31[ruby/prism] Match match_hash_var when quotes are usedKevin Newton
https://github.com/ruby/prism/commit/f2a327449a
2024-05-31[ruby/prism] Match parser for match_rest in patternKevin Newton
https://github.com/ruby/prism/commit/785de2c39d
2024-05-30[ruby/prism] Tests overhaulKevin Newton
https://github.com/ruby/prism/commit/6f886be0a4
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-17 05:55:40 +0000