summaryrefslogtreecommitdiff
path: root/lib/prism
AgeCommit message (Collapse)Author
2024-12-16[ruby/prism] Fix up regression in ruby parser translationKevin Newton
https://github.com/ruby/prism/commit/b283a72c88 Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-12-16[ruby/prism] Simplify srange_find in parser compilerKevin Newton
https://github.com/ruby/prism/commit/34efacc618 Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-12-16[ruby/prism] Add do keyword tracking for While/UntilKevin Newton
https://github.com/ruby/prism/commit/9686897290 Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-12-16[ruby/prism] Bump to v1.2.0Kevin Newton
https://github.com/ruby/prism/commit/817a8e39d9 Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-12-16[ruby/prism] Bump to v1.1.0Kevin Newton
https://github.com/ruby/prism/commit/f80026883d Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-12-11[ruby/prism] Bump typechecking depsKevin Newton
https://github.com/ruby/prism/commit/230c8b8a48
2024-11-12[ruby/prism] Use RbConfig to locate libprism and headers when it is a ↵Benoit Daloze
default gem * This is notably necessary on TruffleRuby, which is updating to Ruby 3.3 which introduces Prism as a default gem. * Using the existing path is not an option as it would end up in truffleruby/lib/build/libprism.so and "truffleruby/lib/include/#{header}" which are not good places for such files. https://github.com/ruby/prism/commit/5d16473e69
2024-10-16[ruby/prism] RelocationKevin Newton
https://github.com/ruby/prism/commit/5ea6042408
2024-10-11[ruby/prism] Prism::StringQueryKevin Newton
Introduce StringQuery to provide methods to access some metadata about the Ruby lexer. https://github.com/ruby/prism/commit/d3f55b67b9
2024-10-10[ruby/prism] Prism::CodeUnitsCacheKevin Newton
Calculating code unit offsets for a source can be very expensive, especially when the source is large. This commit introduces a new class that wraps the source and desired encoding into a cache that reuses pre-computed offsets. It performs quite a bit better. There are still some problems with this approach, namely character boundaries and the fact that the cache is unbounded, but both of these may be addressed in subsequent commits. https://github.com/ruby/prism/commit/2e3e1a4d4d
2024-10-09[ruby/prism] Attempt to assume binary is UTF-8Kevin Newton
https://github.com/ruby/prism/commit/343197e4ff
2024-10-09[ruby/prism] Avoid breaking code units offset on binary encodingVinicius Stock
https://github.com/ruby/prism/commit/25a4cf6794 Co-authored-by: Kevin Newton <kddnewton@users.noreply.github.com>
2024-10-03[ruby/prism] Use `partial_script` for the parser translatorsEarlopain
Followup to https://github.com/ruby/prism/pull/3079 https://github.com/ruby/prism/commit/68f434e356
2024-09-25[ruby/prism] Fix up lex difference when ~ heredoc with 0 dedent and line ↵Kevin Newton
continuation https://github.com/ruby/prism/commit/84a9251915
2024-09-24[ruby/prism] Accept version shorthand like 3.4Kevin Newton
https://github.com/ruby/prism/commit/098f1c4607
2024-09-24[ruby/prism] Accept all 3.3.x and 3.4.x Ruby versions for Prism.parseBenoit Daloze
https://github.com/ruby/prism/commit/a4fcd5339a
2024-09-20[ruby/prism] Fix `kDO_LAMBDA` token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser::Lexer` ## Summary This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block. ### Parser gem (Expected) Returns `kDO_LAMBDA` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` ### `Prism::Translation::Parser` (Actual) Previously, the parser returned `kDO` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` After the update, the parser now returns `kDO_LAMBDA` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]], [:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]] ``` ## Additional Information Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`. However, since `kDO` is already being returned in this case, there is no change in behavior. ### Parser gem Returns `tLAMBDA` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]], [:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]], [:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 23...25>]], [:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]] ``` ### `Prism::Translation::Parser` Returns `kDO` token: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23] [[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]], [:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]], [:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO, ["do", #<Parser::Source::Range example.rb 23...25>]], [:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]] ``` As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`. In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR. https://github.com/ruby/prism/commit/2ee480654c
2024-09-20[ruby/prism] Introduce partial_script optionKevin Newton
https://github.com/ruby/prism/commit/b28877fa4f
2024-09-16[ruby/prism] Do not leak explicit encodingKevin Newton
Fixes [Bug #20744] https://github.com/ruby/prism/commit/f1b8b1b2a2
2024-09-13[ruby/prism] Expose main_script in serialization APIKevin Newton
https://github.com/ruby/prism/commit/0b527ca93f
2024-09-12[ruby/prism] Check errno for parsing directoryKevin Newton
https://github.com/ruby/prism/commit/d68ea29d04 Notes: Merged: https://github.com/ruby/ruby/pull/11497
2024-09-09[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument. ## Parser gem (Expected) Returns `tDSTAR` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tPOW` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` After the update, the parser now returns `tDSTAR` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` With this change, the following code could be removed from test/prism/ruby/parser_test.rb: ```diff - when :tPOW - actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR ``` `tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["a", #<Parser::Source::Range example.rb 0...1>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 2...4>]], [:tIDENTIFIER, ["b", #<Parser::Source::Range example.rb 5...6>]]] ``` https://github.com/ruby/prism/commit/66bde35a44
2024-09-07[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis. ## Parser gem (Expected) Returns `tLPAREN2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 \ -ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tLPAREN` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` After the update, the parser now returns `tLPAREN2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem. The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`. With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`: ```diff - when :tLPAREN - actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2 ``` https://github.com/ruby/prism/commit/04d6f3478d
2024-09-04[ruby/prism] essentialy ==> essentiallyydah
https://github.com/ruby/prism/commit/9c68c01bcd
2024-08-29[ruby/prism] Remove deprecated lib/prism/translation/parser/rubocop.rb fileKoichi ITO
Follow up https://github.com/ruby/prism/pull/2558. This PR removes deprecated lib/prism/translation/parser/rubocop.rb file. The file was a workaround to allow setting `TargetRubyVersion: 80_82_73_83_77.xx` until Prism (`Prism::Translation::Parser`) is integrated into RuboCop. RuboCop already supports Prism (`Prism::Translation::Parser`) as of https://github.com/rubocop/rubocop/pull/12724. It has been several months since the file was deprecated in ruby/prism#2558. And, yesterday, Prism 1.0.0 was released, but perhaps the file should have been removed before then. Although this might be seen as incompatible with semver 1.0.0, I think there is no longer a reason to keep the file. https://github.com/ruby/prism/commit/646a10270e
2024-08-28[PRISM] Sync version updateKevin Newton
2024-08-28[ruby/prism] Reverse-sync ruby/ruby and deprecate old fieldsKevin Newton
https://github.com/ruby/prism/commit/bc21c9f3ee
2024-08-28[PRISM] Field renamingKevin Newton
Rename some fields that do not quite make sense. * CaseMatchNode#consequent -> CaseMatchNode#else_clause * CaseNode#consequent -> CaseNode#else_clause * IfNode#consequent -> IfNode#subsequent * RescueNode#consequent -> RescueNode#subsequent * UnlessNode#consequent -> UnlessNode#else_clause Notes: Merged: https://github.com/ruby/ruby/pull/11480
2024-08-14[ruby/prism] Tweak inspect representation of `Prism::Location`Koichi ITO
This PR tweaks inspect representation of `Prism::Location`. ## Before During debugging, the meaning of `@location=https://github.com/ruby/prism/commit/21474836481` was unclear: ```console $ ruby -Ilib -rprism -e 'p Prism.lex("puts :hi").value.map(&:first)[1]' #<Prism::Token:0x000000010cd74e40 @source=#<Prism::ASCIISource:0x000000010cb5f808 @source="puts :hi", @start_line=1, @offsets=[0]>, @type=:SYMBOL_BEGIN, @value=":", @location=https://github.com/ruby/prism/commit/21474836481> ``` ## After This PR clarifies the contents of the location object, aligning with what I think user expects: ```console $ ruby -Ilib -rprism -e 'p Prism.lex("puts :hi").value.map(&:first)[1]' #<Prism::Token:0x000000010e174d50 @source=#<Prism::ASCIISource:0x000000010df5efe8 @source="puts :hi", @start_line=1, @offsets=[0]>, @type=:SYMBOL_BEGIN, @value=":", @location=#<Prism::Location @start_offset=5 @length=1 start_line=1>> ``` Although it is uncertain whether Prism will accept this change in the inspect representation, it is submitted here as a suggestion. https://github.com/ruby/prism/commit/e7421ce1c5
2024-07-11[ruby/prism] (ruby_parser) Handle bare string in implicit concatKevin Newton
https://github.com/ruby/prism/commit/afc7c9344a
2024-07-11[ruby/prism] (ruby_parser) Match match3 nodes for regular expressionsKevin Newton
https://github.com/ruby/prism/commit/47cb73ce69
2024-07-11[ruby/prism] (ruby_parser) Reverse associativity of and and or nodesKevin Newton
https://github.com/ruby/prism/commit/073e8ba307
2024-07-11[ruby/prism] Various cleanup for initializers and typechecksKevin Newton
https://github.com/ruby/prism/commit/86cf82794a
2024-07-11[ruby/prism] Add node ids to nodesKevin Newton
https://github.com/ruby/prism/commit/bf16ade7f9
2024-07-11[ruby/prism] Move Node#type and Node::type documentationKevin Newton
https://github.com/ruby/prism/commit/08a71f6259
2024-07-11[ruby/prism] Expose flags on every node typeKevin Newton
https://github.com/ruby/prism/commit/9f12a56fd6
2024-07-11[ruby/prism] Move location to second position for node initializersKevin Newton
https://github.com/ruby/prism/commit/4cc0eda4ca
2024-07-11[ruby/prism] Reconfigure error testsKevin Newton
https://github.com/ruby/prism/commit/fb7e1ebb7f
2024-06-18[ruby/prism] (parser) Fix up tokens for empty symbolKevin Newton
https://github.com/ruby/prism/commit/5985ab7687
2024-06-10[ruby/prism] Provide ability to lock encoding while parsingKevin Newton
https://github.com/ruby/prism/commit/f7faedfb3f
2024-06-07Sync prism version to latestKevin Newton
2024-06-07[ruby/prism] Handle chomped bytesize with lines without newlinesKevin Newton
https://github.com/ruby/prism/commit/1528d3c019
2024-06-07[ruby/prism] Ensure inner heredoc nodes have the correct locationKevin Newton
https://github.com/ruby/prism/commit/100340bc6b
2024-06-07[ruby/prism] Use correct newlines for heredoc inner linesKevin Newton
https://github.com/ruby/prism/commit/4a9a7a62af Co-authored-by: Jason Kim <jasonkim@github.com> Co-authored-by: Adam Hess <HParker@github.com>
2024-06-05[ruby/prism] Remove unused string list structKevin Newton
https://github.com/ruby/prism/commit/36c6851c85
2024-06-04[ruby/prism] (parser) split up regexp content by linesKevin Newton
https://github.com/ruby/prism/commit/85b4a5f804
2024-06-04[ruby/prism] (parser) handle quoted symbols in hash patternsKevin Newton
https://github.com/ruby/prism/commit/461aa5e658
2024-06-04[ruby/prism] (parser) fix up srange_find to anchor at the start of the sliceKevin Newton
https://github.com/ruby/prism/commit/aecce571d8
2024-06-04[ruby/prism] (parser) fix up nested multi writeKevin Newton
https://github.com/ruby/prism/commit/12e079c97e
2024-06-04[ruby/prism] Revert "Revert "Properly destructure procarg0 in parser ↵Kevin Newton
translation"" This reverts commit https://github.com/ruby/prism/commit/d8ae19d0334a. https://github.com/ruby/prism/commit/df1eda2811
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-05 06:43:37 +0000