ruby.git/lib/prism/translation/parser/lexer.rb, branch v4.0.2

[ruby/prism] Reject some cases with `return` and command calls

2025-09-11T15:55:55+00:00

The same also applies to `break`/`next`.

https://bugs.ruby-lang.org/issues/21540

https://github.com/ruby/prism/commit/3a38b192e3

[ruby/prism] Be more defensive in the parser translator lexer

2025-08-14T15:42:33+00:00

Generally I have been good about safely accessing the tokens but failed
to properly guard against no tokens in places
where it could theoretically happen through invalid syntax.

I added a test case for one occurance, other changes are theoretical only.

https://github.com/ruby/prism/commit/4a3866af19

[ruby/prism] Fix parser translator with trailing backslash in `%W` /`%I` array

2025-06-30T12:32:31+00:00

https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-25w+and+-25W-3A+String-Array+Literals
> %W allow escape sequences described in Escape Sequences. However the continuation line  is not usable because it is interpreted as the escaped newline described above.

https://github.com/ruby/prism/commit/f5c7460ad5

[ruby/prism] Fix parser translator during string escaping with invalid utf-8

2025-06-11T18:07:43+00:00

Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues.

Fixes https://github.com/ruby/prism/issues/3582

https://github.com/ruby/prism/commit/7f3008b2b5

[ruby/prism] [DOC] Specify markdown mode to RDoc

2025-05-29T04:45:58+00:00

https://github.com/ruby/prism/commit/12af4e144e

[ruby/prism] Fix parser translator when pinning hash with string keys

2025-03-30T17:24:05+00:00

`StringNode` and `SymbolNode` don't have the same shape
(`content` vs `value`) and that wasn't handled.

I believe the logic for the common case can be reused.
I simply left the special handling for implicit nodes in pattern matching
and fall through otherwise.

NOTE: patterns.txt is not actually tested at the moment,
because it contains syntax that `parser` mistakenly rejects.
But I checked manually that this doesn't introduce other failures.
https://github.com/whitequark/parser/pull/1060

https://github.com/ruby/prism/commit/55adfaa895

Fix up merge conflicts for prism sync

2025-03-18T17:36:53+00:00

[ruby/prism] Further refine string handling in the parser translator

2025-03-18T17:36:53+00:00

Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

https://github.com/ruby/prism/commit/4edfe9d981

[ruby/prism] Use Set.new over to_set

2025-03-18T17:36:53+00:00

https://github.com/ruby/prism/commit/422d5c4c64

[ruby/prism] Optimize array inclusion checks in the parser translator

2025-03-18T17:36:53+00:00

I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but
the others will be faster as well.

Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7

https://github.com/ruby/prism/commit/ca9500a3fc