ruby.git/test/prism/fixtures/strings.txt, branch v4.0.4 The Ruby Programming Language [ruby/prism] Fix parser translator with trailing backslash in `%W` /`%I` array 2025-06-30T12:32:31+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-06-12T12:12:40+00:00 3071c5d04cb4369ee8118ece30fb2887205f6c61 https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-25w+and+-25W-3A+String-Array+Literals > %W allow escape sequences described in Escape Sequences. However the continuation line <newline> is not usable because it is interpreted as the escaped newline described above. https://github.com/ruby/prism/commit/f5c7460ad5 
https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-25w+and+-25W-3A+String-Array+Literals
> %W allow escape sequences described in Escape Sequences. However the continuation line <newline> is not usable because it is interpreted as the escaped newline described above.

https://github.com/ruby/prism/commit/f5c7460ad5
[ruby/prism] Fix parser translator during string escaping with invalid utf-8 2025-06-11T18:07:43+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-06-11T13:28:21+00:00 970813d98285b8f59fe5e4d3c815cc044926cb1b Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues. Fixes https://github.com/ruby/prism/issues/3582 https://github.com/ruby/prism/commit/7f3008b2b5 
Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues.

Fixes https://github.com/ruby/prism/issues/3582

https://github.com/ruby/prism/commit/7f3008b2b5
[ruby/prism] Add a multiple statements flag to parentheses 2025-03-18T17:36:53+00:00 Kevin Newton kddnewton@gmail.com 2025-03-13T17:59:45+00:00 dc48c1aca347f43ef9cb122342c80d699c5860fa This can get triggered even if the list of statements only contains a single statement. This is necessary to properly support compiling ```ruby defined? (;a) defined? (a;) ``` as "expression". Previously these were parsed as statements lists with single statements in them. https://github.com/ruby/prism/commit/b63b5d67a9 
This can get triggered even if the list of statements only contains
a single statement. This is necessary to properly support compiling

```ruby
defined? (;a)
defined? (a;)
```

as "expression". Previously these were parsed as statements lists
with single statements in them.

https://github.com/ruby/prism/commit/b63b5d67a9
[ruby/prism] Further refine string handling in the parser translator 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-15T22:24:05+00:00 a8adf5e006da03b8ccaa2bf900f4f077ca9888cf Mostly around newlines and line continuation. * percent arrays need special backslash handling in the ast * Fix offset issue for heredocs with many line continuations (used wrong variable as index access) * More refined rules on when to simplify string tokens * Handle line continuations in squiggly heredocs * Correctly dedent squiggly heredocs with interpolation * Consider `':foo:` and `%s[foo]` to not be interpolation https://github.com/ruby/prism/commit/4edfe9d981 
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

https://github.com/ruby/prism/commit/4edfe9d981
[ruby/prism] Further refine string handling in the parser translator 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-15T22:24:05+00:00 fd7a10cf4a73f27a0113a6bc2a65c4c274ee11ec Mostly around newlines and line continuation. * percent arrays need special backslash handling in the ast * Fix offset issue for heredocs with many line continuations (used wrong variable as index access) * More refined rules on when to simplify string tokens * Handle line continuations in squiggly heredocs * Correctly dedent squiggly heredocs with interpolation * Consider `':foo:` and `%s[foo]` to not be interpolation https://github.com/ruby/prism/commit/4edfe9d981 
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

https://github.com/ruby/prism/commit/4edfe9d981
[ruby/prism] Fix parser translator tokens for %-arrays with whitespace escapes 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-13T20:17:29+00:00 177adf6fa543663334bfb8918b356b4771e5ff1a Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline https://github.com/ruby/prism/commit/bd3dd2b62a 
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline

https://github.com/ruby/prism/commit/bd3dd2b62a
[ruby/prism] Fix parser translator edge-case when multiline string ends with \n 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-20T14:49:06+00:00 ac728389e219480dd678a01e6f91f96098d3b6d6 When the line contains no real newline but contains unescaped ones, then there will be one less entry https://github.com/ruby/prism/commit/4ef093b600 
When the line contains no real newline but contains unescaped ones, then there will be one less entry

https://github.com/ruby/prism/commit/4ef093b600
[ruby/prism] Better handle all kinds of multiline strings in the parser translator 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-08T20:17:21+00:00 0fcb7fc21d1923a61ffe4d127e6e72cc55560972 This is a followup to #3373, where the implementation was extracted https://github.com/ruby/prism/commit/2637007929 
This is a followup to #3373, where the implementation
was extracted

https://github.com/ruby/prism/commit/2637007929
[ruby/prism] Further refine string handling in the parser translator 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-15T22:24:05+00:00 bc506295a30a5806b3346ed09cd679f3b8ee6f64 Mostly around newlines and line continuation. * percent arrays need special backslash handling in the ast * Fix offset issue for heredocs with many line continuations (used wrong variable as index access) * More refined rules on when to simplify string tokens * Handle line continuations in squiggly heredocs * Correctly dedent squiggly heredocs with interpolation * Consider `':foo:` and `%s[foo]` to not be interpolation https://github.com/ruby/prism/commit/4edfe9d981 
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

https://github.com/ruby/prism/commit/4edfe9d981
[ruby/prism] Fix parser translator when unescaping invalid utf8 2025-03-18T17:36:53+00:00 Earlopain 14981592+Earlopain@users.noreply.github.com 2025-01-12T19:08:28+00:00 705bd6fadbd5a784d2c0fd52138007404de6a5c2 1. The string starts out as binary 2. `ち` is appended, forcing it back into utf-8 3. Some invalid byte sequences are tried to append > incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT) This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new so it needs a fallback https://github.com/ruby/prism/commit/e31e94a775 
1. The string starts out as binary
2. `ち` is appended, forcing it back into utf-8
3. Some invalid byte sequences are tried to append

> incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT)

This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new
so it needs a fallback

https://github.com/ruby/prism/commit/e31e94a775