<feed xmlns='http://www.w3.org/2005/Atom'>
<title>ruby.git/test/prism/parser_test.rb, branch master</title>
<subtitle>The Ruby Programming Language</subtitle>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/'/>
<entry>
<title>[ruby/prism] Tests overhaul</title>
<updated>2024-05-30T19:18:20+00:00</updated>
<author>
<name>Kevin Newton</name>
<email>kddnewton@gmail.com</email>
</author>
<published>2024-05-29T14:12:51+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=72452f43871b8034bfa718ed823bc62b5b81d6f9'/>
<id>72452f43871b8034bfa718ed823bc62b5b81d6f9</id>
<content type='text'>
https://github.com/ruby/prism/commit/6f886be0a4
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
https://github.com/ruby/prism/commit/6f886be0a4
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix parser translation's heredoc whitespace calculation</title>
<updated>2024-04-12T13:55:35+00:00</updated>
<author>
<name>Stan Lo</name>
<email>stan.lo@shopify.com</email>
</author>
<published>2024-04-02T20:41:40+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=0924ff2d390084922fbcebada20b968fa3e8238c'/>
<id>0924ff2d390084922fbcebada20b968fa3e8238c</id>
<content type='text'>
Given this example:

```rb
&lt;&lt;~HEREDOC
  #{x}
HEREDOC
```

Both the parser gem and Prism's translation layer would generate the following AST:

```
s(:dstr,
  s(:begin,
    s(:int, 1)),
  s(:str, " a\n"))
```

However, the parser gem inserts a empty string node into this node's location, like:

```
&lt;Parser::Source::Map::Heredoc:0x0000000104ce73b8
 @expression=#&lt;Parser::Source::Range (string) 0...10&gt;,
 @heredoc_body=#&lt;Parser::Source::Range (string) 11...20&gt;,
 @heredoc_end=#&lt;Parser::Source::Range (string) 20...27&gt;,
 @node=s(:dstr,
  s(:str, ""),
  s(:begin,
    s(:int, 1)),
  s(:str, " a\n"))&gt;
```

This is required to calculate the correct whitespace for the heredoc body.

We need to adjust the translation layer to account for this.

With this fix, we also won't need to ignore the tilde heredoc fixture anymore.

https://github.com/ruby/prism/commit/e7372e3ba5
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Given this example:

```rb
&lt;&lt;~HEREDOC
  #{x}
HEREDOC
```

Both the parser gem and Prism's translation layer would generate the following AST:

```
s(:dstr,
  s(:begin,
    s(:int, 1)),
  s(:str, " a\n"))
```

However, the parser gem inserts a empty string node into this node's location, like:

```
&lt;Parser::Source::Map::Heredoc:0x0000000104ce73b8
 @expression=#&lt;Parser::Source::Range (string) 0...10&gt;,
 @heredoc_body=#&lt;Parser::Source::Range (string) 11...20&gt;,
 @heredoc_end=#&lt;Parser::Source::Range (string) 20...27&gt;,
 @node=s(:dstr,
  s(:str, ""),
  s(:begin,
    s(:int, 1)),
  s(:str, " a\n"))&gt;
```

This is required to calculate the correct whitespace for the heredoc body.

We need to adjust the translation layer to account for this.

With this fix, we also won't need to ignore the tilde heredoc fixture anymore.

https://github.com/ruby/prism/commit/e7372e3ba5
</pre>
</div>
</content>
</entry>
<entry>
<title>Sync latest prism</title>
<updated>2024-04-12T13:52:29+00:00</updated>
<author>
<name>Kevin Newton</name>
<email>kddnewton@gmail.com</email>
</author>
<published>2024-04-12T13:52:29+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=abd05c848f437405e10410ded2a3d666e1b9bba5'/>
<id>abd05c848f437405e10410ded2a3d666e1b9bba5</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-03-15T18:07:59+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-15T15:30:01+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=c9da8d67fdb9fab82f76d583239f5b9761f60350'/>
<id>c9da8d67fdb9fab82f76d583239f5b9761f60350</id>
<content type='text'>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser`
for the heredocs_leading_whitespace.txt test.

https://github.com/ruby/prism/commit/7d45fb1eed
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser`
for the heredocs_leading_whitespace.txt test.

https://github.com/ruby/prism/commit/7d45fb1eed
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Turn on heredocs nested test for `Prism::Translation::Parser`</title>
<updated>2024-03-13T17:35:31+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-13T17:06:29+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=a5ac27e9e9d704af16ca615ee7eeae38d54c11cf'/>
<id>a5ac27e9e9d704af16ca615ee7eeae38d54c11cf</id>
<content type='text'>
This PR enables the heredocs_nested.txt that was previously skipped testing.

https://github.com/ruby/prism/commit/9c54f7f10b
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR enables the heredocs_nested.txt that was previously skipped testing.

https://github.com/ruby/prism/commit/9c54f7f10b
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-03-12T17:33:14+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-12T17:05:57+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=f8cab4ef8e214dc05efe8a0d4178864bce3b790a'/>
<id>f8cab4ef8e214dc05efe8a0d4178864bce3b790a</id>
<content type='text'>
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`.
The former is used in xstrings like `` `foo` ``, while the latter is utilized as
a back reference in contexts like `` A::` ``.
This PR will make corrections to differentiate the use of `BACKTICK`.

This mistake was discovered through the investigation of xstring.txt file.
The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail,
hence it will be separated into xstring_with_backslash.txt file.
This separation will facilitate addressing the correction at a different time.

https://github.com/ruby/prism/commit/49ad8df40a
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`.
The former is used in xstrings like `` `foo` ``, while the latter is utilized as
a back reference in contexts like `` A::` ``.
This PR will make corrections to differentiate the use of `BACKTICK`.

This mistake was discovered through the investigation of xstring.txt file.
The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail,
hence it will be separated into xstring_with_backslash.txt file.
This separation will facilitate addressing the correction at a different time.

https://github.com/ruby/prism/commit/49ad8df40a
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Follow ruby/prism#2581</title>
<updated>2024-03-12T17:20:41+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-12T15:25:10+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=a17a33b38e994564268d0fc0642cd8b569dd65ce'/>
<id>a17a33b38e994564268d0fc0642cd8b569dd65ce</id>
<content type='text'>
Due to ruby/prism#2581, the test case for non_alphanumeric_methods.txt has also been restored.
It also solved issues with `def self.`' not limited to constants for `tBACK_REF2` token.

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def self.`; end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:kSELF, ["self", #&lt;Parser::Source::Range example.rb 4...8&gt;]],
[:tDOT, [".", #&lt;Parser::Source::Range example.rb 8...9&gt;]], [:tBACK_REF2, ["`", #&lt;Parser::Source::Range example.rb 9...10&gt;]],
[:tSEMI, [";", #&lt;Parser::Source::Range example.rb 10...11&gt;]], [:kEND, ["end", #&lt;Parser::Source::Range example.rb 12...15&gt;]]]
```

https://github.com/ruby/prism/commit/f814db8ada
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Due to ruby/prism#2581, the test case for non_alphanumeric_methods.txt has also been restored.
It also solved issues with `def self.`' not limited to constants for `tBACK_REF2` token.

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def self.`; end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:kSELF, ["self", #&lt;Parser::Source::Range example.rb 4...8&gt;]],
[:tDOT, [".", #&lt;Parser::Source::Range example.rb 8...9&gt;]], [:tBACK_REF2, ["`", #&lt;Parser::Source::Range example.rb 9...10&gt;]],
[:tSEMI, [";", #&lt;Parser::Source::Range example.rb 10...11&gt;]], [:kEND, ["end", #&lt;Parser::Source::Range example.rb 12...15&gt;]]]
```

https://github.com/ruby/prism/commit/f814db8ada
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-03-12T15:17:20+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-12T15:00:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=c637f53edd0c3f285d29d8679903d3b8bfcfbe75'/>
<id>c637f53edd0c3f285d29d8679903d3b8bfcfbe75</id>
<content type='text'>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `tBACK_REF2`:

## Parser gem (Expected)

Returns `tBACK_REF2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tCOLON2, ["::", #&lt;Parser::Source::Range example.rb 1...3&gt;]],
[:tBACK_REF2, ["`", #&lt;Parser::Source::Range example.rb 3...4&gt;]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tXSTRING_BEG` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tCOLON2, ["::", #&lt;Parser::Source::Range example.rb 1...3&gt;]],
[:tXSTRING_BEG, ["`", #&lt;Parser::Source::Range example.rb 3...4&gt;]]]
```

After the update, the parser now returns `tBACK_REF2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tCOLON2, ["::", #&lt;Parser::Source::Range example.rb 1...3&gt;]],
[:tBACK_REF2, ["`", #&lt;Parser::Source::Range example.rb 3...4&gt;]]]
```

This correction enables the restoration of `constants.txt` as a test case.

https://github.com/ruby/prism/commit/7f63b28f98
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `tBACK_REF2`:

## Parser gem (Expected)

Returns `tBACK_REF2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tCOLON2, ["::", #&lt;Parser::Source::Range example.rb 1...3&gt;]],
[:tBACK_REF2, ["`", #&lt;Parser::Source::Range example.rb 3...4&gt;]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tXSTRING_BEG` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tCOLON2, ["::", #&lt;Parser::Source::Range example.rb 1...3&gt;]],
[:tXSTRING_BEG, ["`", #&lt;Parser::Source::Range example.rb 3...4&gt;]]]
```

After the update, the parser now returns `tBACK_REF2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tCOLON2, ["::", #&lt;Parser::Source::Range example.rb 1...3&gt;]],
[:tBACK_REF2, ["`", #&lt;Parser::Source::Range example.rb 3...4&gt;]]]
```

This correction enables the restoration of `constants.txt` as a test case.

https://github.com/ruby/prism/commit/7f63b28f98
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Follow #2579</title>
<updated>2024-03-12T14:48:39+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-12T14:35:31+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=a41649bff4611a6004471e89cac31e6535e2cff8'/>
<id>a41649bff4611a6004471e89cac31e6535e2cff8</id>
<content type='text'>
Due to #2579, the test case for range_begin_open_inclusive.txt has also been restored.

https://github.com/ruby/prism/commit/da173526a1
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Due to #2579, the test case for range_begin_open_inclusive.txt has also been restored.

https://github.com/ruby/prism/commit/da173526a1
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Restore some skipped heredoc tests for `Prism::Translation::Parser`</title>
<updated>2024-03-12T14:15:52+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-12T11:57:36+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=1ed6b7b40d7d1f459e37160a7bcd1e79420229c9'/>
<id>1ed6b7b40d7d1f459e37160a7bcd1e79420229c9</id>
<content type='text'>
This PR restores some heredoc tests that have already passed
for `Prism::Translation::Parser`.

https://github.com/ruby/prism/commit/297b9e2380
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR restores some heredoc tests that have already passed
for `Prism::Translation::Parser`.

https://github.com/ruby/prism/commit/297b9e2380
</pre>
</div>
</content>
</entry>
</feed>
