summaryrefslogtreecommitdiff
path: root/lib/irb/ruby-lex.rb
AgeCommit message (Collapse)Author
4 days[ruby/irb] Simplify regexp to account for prism error messagesKevin Newton
(https://github.com/ruby/irb/pull/954) https://github.com/ruby/irb/commit/e0c29be074 Co-authored-by: Stan Lo <stan001212@gmail.com>
2024-02-23[ruby/irb] Turn on frozen literal in filesStan Lo
(https://github.com/ruby/irb/pull/881) https://github.com/ruby/irb/commit/83d90550c2
2024-01-30[ruby/irb] Fix undef and alias indenttomoya ishida
(https://github.com/ruby/irb/pull/838) https://github.com/ruby/irb/commit/a641746b18
2023-10-12[ruby/irb] Decouple RubyLex from prompt and line_notomoya ishida
(https://github.com/ruby/irb/pull/701) * Remove instance variable prompt and line_no from RubyLex * Fix prompt test * Rename prompt generating method and make it private https://github.com/ruby/irb/commit/1ceb97fe2e
2023-10-04[ruby/irb] Clear all context usages in RubyLexStan Lo
(https://github.com/ruby/irb/pull/684) After this change, `RubyLex` will not interact with `Context` directly in any way. This decoupling has a few benefits: - It makes `RubyLex` easier to test as it no longer has a dependency on `Context`. We can see this from the removal of `build_context` from `test_ruby_lex.rb`. - It will make `RubyLex` easier to understand as it will not be affected by state changes in `Context` objects. - It allows `RubyLex` to be used in places where `Context` is not available. https://github.com/ruby/irb/commit/d5b262a076
2023-08-24[ruby/irb] Deprecate RubyLex and warn about referencing to itStan Lo
(https://github.com/ruby/irb/pull/692) `RubyLex` has always been a private component of IRB, so we should explicitly discourage usages of it. Also, it should be placed under the `IRB` module like other components. https://github.com/ruby/irb/commit/069b5625f7
2023-08-21[ruby/irb] Move input processing out of RubyLexStan Lo
(https://github.com/ruby/irb/pull/683) * Add a test case for Ctrl-C handling * Test symbol aliases with integration tests There are a few places that also need to check symbol aliases before `Irb#eval_input`. But since the current command test skip them, we don't have test coverage on them. * Move each_top_level_statement and readmultiline to Irb This will save RubyLex from knowning information about commands and aliases. https://github.com/ruby/irb/commit/69cb5b5615
2023-08-16[ruby/irb] Encapsulate input details in Statement objectsStan Lo
(https://github.com/ruby/irb/pull/682) * Introduce Statement class * Split Statement class for better clarity https://github.com/ruby/irb/commit/65e8e68690
2023-08-13[ruby/irb] Support seamless integration with ruby/debugStan Lo
(https://github.com/ruby/irb/pull/575) * Support native integration with ruby/debug * Prevent using multi-irb and activating debugger at the same time Multi-irb makes a few assumptions: - IRB will manage all threads that host sub-irb sessions - All IRB sessions will be run on the threads created by IRB itself However, when using the debugger these assumptions are broken: - `debug` will freeze ALL threads when it suspends the session (e.g. when hitting a breakpoint, or performing step-debugging). - Since the irb-debug integration runs IRB as the debugger's interface, it will be run on the debugger's thread, which is not managed by IRB. So we should prevent the 2 features from being used at the same time. To do that, we check if the other feature is already activated when executing the commands that would activate the other feature. https://github.com/ruby/irb/commit/d8fb3246be
2023-08-13[ruby/irb] Move IO configuration to IRB::IrbStan Lo
(https://github.com/ruby/irb/pull/681) It shouldn't be `RubyLex`'s responsibility to handle IO. So this moves the configuration to `IRB::Irb`. https://github.com/ruby/irb/commit/daff750076
2023-08-11[ruby/irb] Remove needless removal of trailing whitespace intomoya ishida
check_code_state (https://github.com/ruby/irb/pull/678) https://github.com/ruby/irb/commit/4a6af7d1ed
2023-08-11[ruby/irb] Move assignment check to RubyLexStan Lo
(https://github.com/ruby/irb/pull/670) Since assignment check relies on tokenization with `Ripper`, it feels like the responsibility of `RubyLex`. `Irb#eval_input` should simply get the result when calling `each_top_level_statement` on `RubyLex`. https://github.com/ruby/irb/commit/89d1adb3fd
2023-07-15[ruby/irb] Indent multiline percent literalstomoya ishida
(https://github.com/ruby/irb/pull/643) https://github.com/ruby/irb/commit/18bb4022a9
2023-06-30[ruby/irb] Re-implement prev_spaces feature for pasted codetomoya ishida
(https://github.com/ruby/irb/pull/607) https://github.com/ruby/irb/commit/9d97a192a5
2023-06-25[ruby/irb] Fix process_continue(rename to should_continue?) andtomoya ishida
check_code_block(rename to check_code_syntax) (https://github.com/ruby/irb/pull/611) https://github.com/ruby/irb/commit/b7f4bfaaa4
2023-06-24[ruby/irb] Omit nesting_level, use indent_level to build prompttomoya ishida
string (https://github.com/ruby/irb/pull/610) https://github.com/ruby/irb/commit/f01ff0811b
2023-06-20[ruby/irb] Improve indentation: bugfix, heredoc, embdoc, stringstomoya ishida
(https://github.com/ruby/irb/pull/515) * Implement heredoc embdoc and string indentation with bugfix * Fix test_ruby_lex's indentation value * Add embdoc indent test * Add workaround for lines==[nil] passed to auto_indent when exit IRB with CTRL+d
2023-06-19[ruby/irb] Add missing token that ignored by rippertomoya ishida
(https://github.com/ruby/irb/pull/608) https://github.com/ruby/irb/commit/1cd3b45402
2023-06-15[ruby/irb] Rewrite RubyLex to fix some bugs and make it possible totomoya ishida
add new features easily (https://github.com/ruby/irb/pull/500) * Add nesting level parser for multiple use (indent, prompt, termination check) * Rewrite RubyLex using NestingParser * Add nesting parser tests, fix some existing tests * Add description comment, rename method to NestingParser * Add comments and tweak code to RubyLex * Update NestingParser test * Extract list of ltype tokens to constants
2023-05-19[ruby/irb] Simplify each_top_level_statementtomoya ishida
(https://github.com/ruby/irb/pull/576) * Simplify each_top_level_statement, reduce instance vars * Update lib/irb/ruby-lex.rb Co-authored-by: Stan Lo <stan001212@gmail.com> * Remove unused ltype from TestRubyLex#check_state response * Remove unnecessary const path of TerminateLineInput * Combine duplicated code state check into method --------- https://github.com/ruby/irb/commit/172453cec4 Co-authored-by: Stan Lo <stan001212@gmail.com>
2023-05-18[ruby/irb] Refactor RubyLex's input/io methodsStan Lo
(https://github.com/ruby/irb/pull/583) 1. Make `RubyLex#set_input` simply assign the input block. This matches the behavior of `RubyLex#set_prompt`. 2. Merge `RubyLex#set_input`'s IO configuration logic with `#set_auto_indent` into `#configure_io`.
2023-04-05[ruby/irb] Drop Ruby 2.6 supportStan Lo
(https://github.com/ruby/irb/pull/555) * Remove all Ruby 2.6 support * Drop Ruby 2.6 specific testing conditions * Only run Ruby 2.7+ on CI * Bump Ruby requirement to 2.7+ https://github.com/ruby/irb/commit/3f714b616c
2023-01-14[ruby/irb] Store context in RubyLexStan Lo
Some background for this refactor: 1. Through a RubyLex instance's lifetime, the context passed to its methods should be the same. Given that `Context` is only initialised in `Irb#initialize`, this should be true. 2. When `RubyLex` is initialised, the context object should be accessible. This is also true in all 3 of `RubyLex.new`'s invocations. With the above observations, we should be able to store the context in `RubyLex` as an instance variable. And doing so will make `RubyLex`'s instance methods easier to use and maintain. https://github.com/ruby/irb/commit/5c8d3df2df
2023-01-12[ruby/irb] Remove redundant argument defaults from some RubyLexStan Lo
methods (https://github.com/ruby/irb/pull/502) * Remove unnecessary parameter defaults These methods are always called with tokens specified. So their default `@tokens` value is never used and is misleading. * Remove unnecessary context default * Require tokens for `RubyLex#check_state`
2023-01-11[ruby/irb] Formatting to header stylesHiroshi SHIBATA
https://github.com/ruby/irb/commit/cef125850d
2023-01-11[ruby/irb] Removed Release Version and Revisions for old VCS softwareHiroshi SHIBATA
https://github.com/ruby/irb/commit/07fae94862
2023-01-11[ruby/irb] Drop unused arguments in `RubyLex`Stan Lo
(https://github.com/ruby/irb/pull/504) * Simplify `RubyLex#set_prompt` It's optional argument is never used by any caller. * Remove the optional `p` argument from `RubyLex#set_input` The argument is only used in a test case, which can be easily replaced by a block argument.
2023-01-02[ruby/irb] Fix prompt and code mismatchtomoya ishida
(https://github.com/ruby/irb/pull/386) * fix prompt and code mismatch * Add test for prompt and code mismatch bug https://github.com/ruby/irb/commit/a5765d8177 Co-authored-by: Stan Lo <stan001212@gmail.com>
2022-12-27[ruby/irb] Refactor RubyLex#process_literal_typeMau Magnaguagno
(https://github.com/ruby/irb/pull/350) Simplify part of regex ``[_a-zA-Z0-9]`` with equivalent shorthand ``\w``. Replace case-when with match ``$1`` or default value ``?"``, making intention more clear.
2022-12-26[ruby/irb] fix indent depth calculation after heredoc and embdoctompng
https://github.com/ruby/irb/commit/b7973dd2d2
2022-12-26[ruby/irb] fix auto-indent after multiline stringtompng
https://github.com/ruby/irb/commit/f65ec49684
2022-11-21[ruby/irb] Add commands to start and use the debuggerTakashi Kokubun
(https://github.com/ruby/irb/pull/449) * Seamlessly integrate a few debug commands * Improve the break command support * Utilize skip_src option if available * Add step and delete commands * Write end-to-end tests for each debugger command * Add documentation * Add backtrace, info, catch commands https://github.com/ruby/irb/commit/976100c1c2
2022-11-03[ruby/irb] Allow non-identifier aliases like Pry's @ and $Takashi Kokubun
(https://github.com/ruby/irb/pull/426) * Allow non-identifier aliases * Move the configuration to IRB.conf * Avoid abusing method lookup for symbol aliases * Add more alias tests * A small optimization * Assume non-nil Context * Load IRB.conf earlier https://github.com/ruby/irb/commit/e23db5132e
2022-11-03[ruby/irb] Refactor RubyLex and its testsStan Lo
(https://github.com/ruby/irb/pull/427) * Make sure `RubyLex#set_input`'s context is always present in tests In real-world scenarios, the context should always be non-nil: https://github.com/ruby/irb/blob/master/lib/irb.rb#L489 So we should make sure our test setup reflects that. * Make context a required keyword Since in practice, `set_input`'s context should always be non-nil, its parameters should reflect that. And since `RubyLex#check_state` is only called by `#lex` and `#set_input`, both of which now always require context, we can assume its context should be non-nil too. https://github.com/ruby/irb/commit/1aeeb86203
2022-10-18[ruby/irb] Always use local variables in current context to parse code ↵tomoya ishida
(https://github.com/ruby/irb/pull/397) * Use local_variables for colorize, code_block_open check, nesting_level and assignment_expression check * Check if expression is an assignment BEFORE evaluating it. evaluate might define new localvars and change result of assignment_expression? * Add local_variables dependent code test * pend local variable dependent test on truffleruby code_block_open is not working on truffleruby * Always pass context to RubyLex#lex * Rename local_variable_assign_code generator method name * Add assignment expression truncate test * Add Context#local_variables and make generate_local_variables_assign_code more simple * Update lib/irb/input-method.rb Co-authored-by: Stan Lo <stan001212@gmail.com> * Add a comment why assignment expression check should be done before evaluate https://github.com/ruby/irb/commit/c8b3877281 Co-authored-by: Stan Lo <stan001212@gmail.com> Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
2022-10-18[ruby/irb] Fix code terminated check with heredoc and backtick ↵tomoya ishida
(https://github.com/ruby/irb/pull/390) * Fix backtick method def method call handled as backtick open * Fix handling heredoc in check_string_literal * Sort result of lexer.parse by pos in ruby<2.7. It's not sorted when the given code includes heredoc. * Update lib/irb/ruby-lex.rb Co-authored-by: Stan Lo <stan001212@gmail.com> * Update lib/irb/ruby-lex.rb Co-authored-by: Stan Lo <stan001212@gmail.com> * Add check_string_literal test for heredoc code that does not end with newline https://github.com/ruby/irb/commit/44bc712460 Co-authored-by: Stan Lo <stan001212@gmail.com>
2022-10-03[ruby/irb] Remove patches for Ruby 2.4st0012
They were added in https://github.com/ruby/irb/commit/b34f26a0dd3b589e6fc321a6248d173682c9202f https://github.com/ruby/irb/commit/0e760d2674
2022-10-03[ruby/irb] Remove unnecessary conditionst0012
https://github.com/ruby/irb/commit/daffb8bfe5
2022-10-03[ruby/irb] Refactor ripper_lex_without_warningst0012
https://github.com/ruby/irb/commit/0db0a8ddc5
2022-10-01[ruby/irb] Fix ripper_lex_without_warning duplicated heredoc tokentompng
https://github.com/ruby/irb/commit/45b539af39
2021-12-24Fix warning in `Lexer::Elem#[]`manga_osyo
Changed to use `#pos` `#event` `#tok` `#state` since using Lexer::Elem#[0~4] now gives a warning. see: https://github.com/ruby/ruby/commit/8944009be7418614ce7d4077807ac2b60d4d5d85 https://github.com/ruby/reline/commit/9adbb9af32
2021-12-03[ruby/irb] Examine indentation of in keyword when trying to type includeKaíque Kandy Koga
Use in_keyword_case_scope? Return fast https://github.com/ruby/irb/commit/8acc7f8dc7
2021-12-03[ruby/irb] Fix bug infinite loop when pasting multilines fo code in Ruby 2.6manga_osyo
Fix bug infinite loop when pasting multilines fo code in Ruby 2.6. This is not reproduced in Ruby 2.7. Changes added in https://github.com/ruby/irb/pull/242/files#diff-612b926e42ed78aed1a889ac1944f7d22229b3a489cc08f837a7f75eca3d3399R155 are also reflected in Ruby 2.6. https://github.com/ruby/irb/commit/0a77f75bf0
2021-12-02Compatibility with IRBschneems
Instead of accessing the struct as an array, access it via methods. There are other places inside of this file already using this API (for example https://github.com/ruby/ruby/blob/e0a5c3d2b71dfad038d7562fdd33f02ffd79232d/lib/irb/ruby-lex.rb#L829-L830). This commit moves all struct array-ish calls to use their method calls instead. It is also ~1.23 faster accessing values via a method instead of as an array according to this microbenchmark: ```ruby Elem = Struct.new(:pos, :event, :tok, :state, :message) do def initialize(pos, event, tok, state, message = nil) super(pos, event, tok, State.new(state), message) end # ... def to_a a = super a.pop unless a.empty? a end end class ElemClass attr_accessor :pos, :event, :tok, :state, :message def initialize(pos, event, tok, state, message = nil) @pos = pos @event = event @tok = tok @state = State.new(state) @message = message end def to_a if @message [@pos, @event, @tok, @state, @message] else [@pos, @event, @tok, @state] end end end # stub state class creation for now class State; def initialize(val); end; end ``` ```ruby Benchmark.ips do |x| x.report("struct") { struct[1] } x.report("class ") { from_class.event } x.compare! end; nil ``` ``` Warming up -------------------------------------- struct 1.624M i/100ms class 1.958M i/100ms Calculating ------------------------------------- struct 17.139M (± 2.6%) i/s - 86.077M in 5.025801s class 21.104M (± 3.4%) i/s - 105.709M in 5.015193s Comparison: class : 21103826.3 i/s struct: 17139201.5 i/s - 1.23x (± 0.00) slower ``` Notes: Merged: https://github.com/ruby/ruby/pull/5093
2021-10-07[ruby/irb] Update descriptions of methodsKaíque Kandy Koga
From Reidline to Reline Update description used in take_corresponding_syntax_to_kw_do and is_the_in_correspond_to_a_for methods Use possessive noun correctly Second element https://github.com/ruby/irb/commit/4fa9714d6f
2021-09-22[ruby/irb] Use typed spaces when the line is inside the here documentsKaíque Kandy Koga
Use first method instead of square brackets to support 2.5 and 2.6 versions Use tokens Clear check_newline_depth_difference https://github.com/ruby/irb/commit/6fec2a5d46
2021-09-10[ruby/irb] Support symbol with backtickaycabta
https://github.com/ruby/irb/commit/0aa2425883
2021-07-23[ruby/irb] Fix #256manga_osyo
Support int that follow on symbeg in IRB https://github.com/ruby/irb/commit/90cb27b1bd
2021-07-10[ruby/irb] Pass local variables from workspace binding to lexerJeremy Evans
This fixes at least an issue where irb will incorrectly assume code opens a heredoc when it does not, such as this code: ```ruby s1 = 'testing' s2 = 'this' s2 <<s1 p s1 s1 ``` Ruby parses the `s2 <<s1` as `s2.<<(s1)`, not as a heredoc, because `s2` is a local variable in scope. irb was using ripper without letting ripper know that `s2` was a local variable, so ripper would lex it as a heredoc instead of a method call. Fix the situation by prepending a line at line 0 with all local variable definitions in scope whenever lexing. This fixes the heredoc issue, and potentially other issues that depend on whether an identifier is a local variable or not. Fixes [Bug #17530] https://github.com/ruby/irb/commit/4ed2187f76
2021-04-26[ruby/irb] Fix typo ture -> true [ci skip]Ryuta Kamizono
https://github.com/ruby/irb/commit/783a0569e8