| Age | Commit message (Collapse) | Author |
|---|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Of course, these won't really be fixtures, but it allows to test against whole codebases
without copying them, doing symlinks or something like that.
For example, I can tell that over the whole RuboCop codebase, there are only 8 files that produce mismatched ast.
Telling what the problem is is a different problem. The ast for real files can and will be huge so I haven't checked yet
(maybe parser bug) but it's nice for discoverability regardless
https://github.com/ruby/prism/commit/2184d82ba6
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
https://github.com/ruby/prism/commit/e1c75f304b
|
|
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline
https://github.com/ruby/prism/commit/bd3dd2b62a
|
|
When the line contains no real newline but contains unescaped ones, then there will be one less entry
https://github.com/ruby/prism/commit/4ef093b600
|
|
translator
This is a followup to #3373, where the implementation
was extracted
https://github.com/ruby/prism/commit/2637007929
|
|
The offset cache contains an entry for each byte so it can't be accessed via the string length.
Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```
For some reason, this still has the wrong offset.
https://github.com/ruby/prism/commit/a651126458
|
|
There are a few other locations that should be included in that check.
I think the end location must always be present but I left it in to be safe (maybe implicit begin somehow?)
https://github.com/ruby/prism/commit/545d07ddc3
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
|
|
1. The string starts out as binary
2. `ち` is appended, forcing it back into utf-8
3. Some invalid byte sequences are tried to append
> incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT)
This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new
so it needs a fallback
https://github.com/ruby/prism/commit/e31e94a775
|
|
https://github.com/ruby/prism/commit/f734350499
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12947
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12798
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12889
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12946
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12923
|
|
Enum array may be the list of pairs of key and value. Check if only
key is completable, not pair.
Fix https://github.com/ruby/optparse/pull/93
Fix https://github.com/ruby/optparse/pull/94
https://github.com/ruby/optparse/commit/a8d0ba8dac
|
|
Infinite ranges, i.e. unbounded ranges, should overlap with any other range
which wasn't the case in the following example: (0..3).overlap?(nil..nil)
Notes:
Merged: https://github.com/ruby/ruby/pull/12937
|
|
* Use `assert_raise_kind_of` instead of `rescue` and `flunk`.
* Use `assert_include` for the pattern that may contain regexp meta
characters.
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12934
|
|
[Misc #21143]
Conceptually this makes sense and is more consistent with using
the `Name = Class.new(Superclass)` alternative method.
However the new class is still named before `inherited` is called.
Notes:
Merged: https://github.com/ruby/ruby/pull/12927
|
|
https://github.com/ruby/prism/commit/56eaf53732
|
|
https://github.com/ruby/prism/commit/10e5431b38
|
|
Previously, vm_make_env_each() (used during proc
creation and for the debug inspector C API) picked up the
non-GC-allocated iseq that rb_vm_push_frame_fname() creates,
which led to a SEGV when the GC tried to mark the non GC object.
Put a real iseq imemo instead. Speed should be about the same since
the old code also did a imemo allocation and a malloc allocation.
Real iseq allows ironing out the special-casing of dummy frames in
rb_execution_context_mark() and rb_execution_context_update(). A check
is added to RubyVM::ISeq#eval, though, to stop attempts to run dummy
iseqs.
[Bug #21180]
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Notes:
Merged: https://github.com/ruby/ruby/pull/12898
|
|
Previously, live range of `ast_value` ended on the call right before
rb_ast_dispose(), which led to premature collection and use-after-free.
We observed this crashing on -O3, -DVM_CHECK_MODE, with GCC 11.4.0 on
Ubuntu.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Notes:
Merged: https://github.com/ruby/ruby/pull/12898
|
|
The code between the two ObjectSpace.count_objects could trigger a GC,
which could free string objects causing this test to fail.
We can see this failure on CI http://ci.rvm.jp/results/trunk-random2@ruby-sp2-noble-docker/5651016
TestHashOnly#test_AREF_fstring_key [test/ruby/test_hash.rb:1991]:
<197483> expected but was
<129689>.
Notes:
Merged: https://github.com/ruby/ruby/pull/12916
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12914
|
|
This was the behavior until `2.10.0` unadvertently changed it.
`"\u1"` would raise, but `"\u1zzz"` wouldn't.
https://github.com/ruby/json/commit/7d0637b9e6
|
|
https://github.com/ruby/ruby/pull/12801 changed regexp matches to reuse
the backref, which causes memory to leak if the original registers of the
match is not freed.
For example, the following script leaks memory:
10.times do
1_000_000.times do
"aaaaaaaaaaa".gsub(/a/, "")
end
puts `ps -o rss= -p #{$$}`
end
Before:
774256
1535152
2297360
3059280
3821296
4583552
5160304
5091456
5114256
4980192
After:
12480
11440
11696
11632
11632
11760
11824
11824
11824
11888
Notes:
Merged: https://github.com/ruby/ruby/pull/12905
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12901
|
|
The test could flake because a major GC could be triggered due to allocation
for caches or other things, which would cause the test to fail.
|
|
The test fails sometimes with:
TestGc#test_latest_gc_info_weak_references_count [test/ruby/test_gc.rb:421]:
Expected 2 to be <= 1.
Notes:
Merged: https://github.com/ruby/ruby/pull/12894
|
|
## Summary
`itblock` node is added to support the `it` block parameter syntax introduced in Ruby 3.4.
```console
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:itblock,
s(:send, nil, :proc), :it,
s(:lvar, :it))
```
This node design is similar to the `numblock` node, which was introduced for the numbered parameter syntax in Ruby 2.7.
```
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { _1 }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:numblock,
s(:send, nil, :proc), 1,
s(:lvar, :_1))
```
The difference is that while numbered parameters can have multiple parameters, the `it` block parameter syntax allows only a single parameter.
In Ruby 3.3, the conventional node prior to the `it` block parameter syntax is returned.
```console
$ ruby -Ilib -rprism -rprism/translation/parser33 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser33.new.tokenize(buffer)[0]'
s(:block,
s(:send, nil, :proc),
s(:args),
s(:send, nil, :it))
```
## Development Note
The Parser gem does not yet support the `it` block parameter syntax. This is the first case where Prism's node design precedes that of the Parser gem.
When implementing https://github.com/whitequark/parser/issues/962, this node design will need to be taken into consideration.
https://github.com/ruby/prism/commit/c141e1420a
|
|
In mock testing for stdout, `StringIO.new` is sometimes used to redirect the output.
In such cases, the assignment is done with `$stdout = StringIO.new`, not the constant `STDOUT`.
e.g., https://github.com/rubocop/rubocop/blob/v1.71.1/lib/rubocop/rspec/shared_contexts.rb#L154-L164
After assigning `StringIO.new`, `$stdout.tty?` returns `false`,
allowing the standard output destination to be switched during test execution.
```ruby
STDOUT.tty? # => true
StringIO.new.tty? # => false
```
However, since `STDOUT.tty?` returns `true`, a failure occurred in environments
where the environment variables `RUBY_PAGER` or `PAGER` are set.
e.g., https://github.com/rubocop/rubocop/pull/13784
To address this, `STDOUT` has been updated to `$stdout` so that the result of `tty?` can be flexibly overridden.
A potential concern is that `$stdout`, unlike `STDOUT`,
does not always represent the standard output at the time the Ruby process started.
However, no concrete examples of issues related to this have been identified.
`STDOUT.tty?` is the logic of optparse introduced in https://github.com/ruby/optparse/pull/70.
This PR replaces `STDOUT` with `$stdout` throughout, based on the assumption
that `$stdout` is sufficient for use with optparse.
https://github.com/ruby/optparse/commit/262cf6f9ac
|
|
Fix https://github.com/ruby/optparse/pull/80
https://github.com/ruby/optparse/commit/050a87d029
|
|
https://github.com/rubygems/rubygems/commit/369f9b9311
Notes:
Merged: https://github.com/ruby/ruby/pull/12890
|
|
Command line arguments are strings, convert enum list elements to
strings to match.
https://github.com/ruby/optparse/commit/c5ec052efc
|
|
https://github.com/ruby/optparse/commit/0a0e977b7c
|
|
The following Location information has been added This is the information required for parse.y to be a universal parser:
```
❯ ruby --parser=prism --dump=parsetree -e "class A < B; end"
@ ProgramNode (location: (1,0)-(1,16))
+-- locals: []
+-- statements:
@ StatementsNode (location: (1,0)-(1,16))
+-- body: (length: 1)
+-- @ ClassNode (location: (1,0)-(1,16))
+-- locals: []
+-- class_keyword_loc: (1,0)-(1,5) = "class"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-- constant_path:
| @ ConstantReadNode (location: (1,6)-(1,7))
| +-- name: :A
+-- inheritance_operator_loc: (1,8)-(1,9) = "<"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-- superclass:
| @ ConstantReadNode (location: (1,10)-(1,11))
| +-- name: :B
+-- body: nil
+-- end_keyword_loc: (1,13)-(1,16) = "end"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-- name: :A
```
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12879
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12878
|
|
https://github.com/ruby/json/commit/2e015ff839
|
|
This test occasionally fail because it runs into a String instance
that had its `==` method removed.
I couldn't identify where this String comes from, but in general
when using `each_object` it's best to not assume returned objectd
are functional.
By just inverting the operands of `==` we ensure it's always
`String#==` that is called.
```
1) Error:
TestObjSpace#test_memsize_of_root_shared_string:
NoMethodError: undefined method '==' for #<String:0x00007f9b50e8c978>
/tmp/ruby/src/trunk-random1/test/objspace/test_objspace.rb:35:in 'block in TestObjSpace#test_memsize_of_root_shared_string'
/tmp/ruby/src/trunk-random1/test/objspace/test_objspace.rb:35:in 'ObjectSpace.each_object'
/tmp/ruby/src/trunk-random1/test/objspace/test_objspace.rb:35:in 'TestObjSpace#test_memsize_of_root_shared_string'
```
Notes:
Merged: https://github.com/ruby/ruby/pull/12870
|
|
TestSetTraceFunc#test_tracepoint_disable is a flaky and failing intermittently. Here is an example of failure logs.
```
1) Failure:
TestSetTraceFunc#test_tracepoint_disable [/home/runner/work/ruby/ruby/src/test/ruby/test_settracefunc.rb:857]:
<[:foo, :disable, :foo, :disable]> expected but was
<[:call, :call, :foo, :disable, :foo, :disable]>.
```
https://github.com/ruby/ruby/actions/runs/13619175633/job/38066208546?pr=12585#step:12:875
I printed values of TracePoint objects as follows, and checked the values when failing intermittently.
https://github.com/ruby/ruby/blob/7f9a6fc582fb5cfd88ab73a61782f39894a37ba6/test/ruby/test_settracefunc.rb#L848
Here is the log when the TestSetTraceFunc#test_tracepoint_disable failed intermittently.
`2025-03-05T09:08:37.4075411Z e: call, f: /home/runner/work/ruby/ruby/src/lib/tempfile.rb, l: 386, i: call, d: Tempfile::FinalizerManager` is an unexpected events. Thus, I modified test code so that we can filter out unexpected trace events.
```
2025-03-05T09:08:37.4075411Z e: call, f: /home/runner/work/ruby/ruby/src/lib/tempfile.rb, l: 386, i: call, d: Tempfile::FinalizerManager
2025-03-05T09:08:37.4085009Z e: call, f: /home/runner/work/ruby/ruby/src/test/ruby/test_settracefunc.rb, l: 808, i: foo, d: TestSetTraceFunc
2025-03-05T09:08:37.4086042Z e: call, f: <internal:trace_point>, l: 295, i: disable, d: TracePoint
2025-03-05T09:08:37.4115693Z e: call, f: /home/runner/work/ruby/ruby/src/test/ruby/test_settracefunc.rb, l: 808, i: foo, d: TestSetTraceFunc
2025-03-05T09:08:37.4116734Z e: call, f: <internal:trace_point>, l: 295, i: disable, d: TracePoint
```
Notes:
Merged-By: ono-max <onoto1998@gmail.com>
|
|
[Bug #21170]
st_table reserves -1 as a special hash value to indicate that an entry
has been deleted. So that that's a valid value to be returned from the
hash function, do_hash replaces -1 with 0 so that it is not mistaken for
the sentinel.
Previously, when upgrading an AR table to an ST table,
rb_st_add_direct_with_hash was used which did not perform the same
conversion, this could lead to a hash in a broken state where one if its
entries which was supposed to exist being marked as a tombstone.
The hash could then become further corrupted when the ST table required
resizing as the falsely tombstoned entry would be skipped but it would
be counted in num entries, leading to an uninitialized entry at index
15.
In most cases this will be really rare, unless using a very poorly
implemented custom hash function.
This also adds two debug assertions, one that st_add_direct_with_hash
does not receive the reserved hash value, and a second in
rebuild_table_with, which ensures that after we rebuild/compact a table
it contains the expected number of elements.
Co-authored-by: Alan Wu <alanwu@ruby-lang.org>
Notes:
Merged: https://github.com/ruby/ruby/pull/12852
|
