summaryrefslogtreecommitdiff
path: root/transcode.c
AgeCommit message (Collapse)Author
2025-11-01Fix memory leak in String#encode when StringValue raisesPeter Zhu
The following script leaks memory: 10.times do 100_000.times do "\ufffd".encode(Encoding::US_ASCII, fallback: proc { Object.new }) rescue end puts `ps -o rss= -p #{$$}` end Before: 450244 887748 1325124 1762756 2200260 2637508 3075012 3512516 3950020 4387524 After: 12236 12364 12748 13004 13388 13516 13772 13772 13772 13772
2025-10-29Fix memory leak in String#encode when fallback too bigPeter Zhu
The following script leaks memory: 10.times do 100_000.times do "\ufffd".encode(Encoding::US_ASCII, fallback: proc { "\uffee" }) rescue end puts `ps -o rss= -p #{$$}` end Before: 451196 889596 1328252 1766524 2204668 2643068 3081724 3520124 3958524 4396796 After: 12800 13056 13184 13312 13312 13312 13312 13312 13312 13312
2025-10-24Fix memory leak of transcoding when fallback raisesPeter Zhu
When the fallback function in transcode_loop raises, it will leak the memory in rb_econv_t. The following script reproduces the leak: 10.times do 100_000.times do "\ufffd".encode(Encoding::US_ASCII, fallback: proc { raise }) rescue end puts `ps -o rss= -p #{$$}` end Before: 451196 889980 1328508 1767676 2206460 2645372 3083900 3522428 3960956 4399484 After: 12508 12636 12892 12892 13148 13404 13532 13788 13916 13916
2025-08-22Fixes to encoding/transcoding for ractors.Luke Gruber
Not all ractor-related encoding issues were fixed by 1afc07e815051e2f73493f055f2130cb642ba12a. I found more by running my test-all branch with 3 ractors for each test.
2025-08-12Allow encodings to be autoloaded through transcoding functionsLuke Gruber
Make sure VM lock is not held when calling `load_transcoder_entry`, as that causes deadlock inside ractors. `String#encode` now works inside ractors, among others. Atomic load the rb_encoding_list Without this, wbcheck would sometimes hit a missing write barrier. Co-authored-by: John Hawthorn <john.hawthorn@shopify.com> Hold VM lock when iterating over global_enc_table.names This st_table can be inserted into at runtime when autoloading encodings. minor optimization when calling Encoding.list
2025-07-24[DOC] Tweak for String#encode!BurdetteLamar
2025-07-03Get transcoding to work across ractors by locking certain operationsLuke Gruber
Ex: `str.encode` and `str.encode!` work across ractors now. The global table `transcoder_table` needs a lock around each st_lookup/st_insert, and it's a two-level table so the second level also needs to be locked around insertion/deletion. In addition to this, the transcoder entries (values in the second-level hash table) need to be locked around retrieving them and loading them as they are loaded lazily. The transcoding objects (`Encoding::Converter`) can't be made shareable, so their operations don't need to be locked.
2024-12-17Fix constant documents (#12357)Soutaro Matsumoto
Delete duplicated `Document-const:` directives Those directives are attached to `rb_define_const` and `rb_file_const` where we don’t need them because the functions are supported by RDoc.
2023-12-07Free everything at shutdownAdam Hess
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown. Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org> Co-authored-by: Peter Zhu <peter@peterzhu.ca>
2023-10-27Adjust indent [ci skip]Nobuyoshi Nakada
2023-06-30Don't check for null pointer in calls to freePeter Zhu
According to the C99 specification section 7.20.3.2 paragraph 2: > If ptr is a null pointer, no action occurs. So we do not need to check that the pointer is a null pointer. Notes: Merged: https://github.com/ruby/ruby/pull/8004
2023-04-04[Feature #19579] Remove !USE_RVARGC code (#7655)Peter Zhu
Remove !USE_RVARGC code [Feature #19579] The Variable Width Allocation feature was turned on by default in Ruby 3.2. Since then, we haven't received bug reports or backports to the non-Variable Width Allocation code paths, so we assume that nobody is using it. We also don't plan on maintaining the non-Variable Width Allocation code, so we are going to remove it. Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-01-16[DOC] Use RDoc inclusions in transcode.cNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/7126
2022-11-16Using UNDEF_P macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6721
2022-09-10Using is_broken_string functionS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6346
2022-08-19Support Encoding::Converter newline: :lf and :lf_newline optionsJeremy Evans
Previously, newline: :lf was accepted but ignored. Where it should have been used was commented out code that didn't work, but unlike all other invalid values, using newline: :lf did not raise an error. This adds support for newline: :lf and :lf_newline, for consistency with newline: :cr and :cr_newline. This is basically the same as universal_newline, except that it only affects writing and not reading due to RUBY_ECONV_NEWLINE_DECORATOR_WRITE_MASK. Add tests for the File.open :newline option while here. Fixes [Bug #12436] Notes: Merged: https://github.com/ruby/ruby/pull/4590
2022-08-18[Bug #18964] Update the code range of appended portionNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6250
2022-07-21Expand tabs [ci skip]Takashi Kokubun
[Misc #18891] Notes: Merged: https://github.com/ruby/ruby/pull/6094
2022-06-17Using is_ascii_string to check encodingS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/5867
2022-02-26[DOC] Place a non-US-ASCII document in a document-specific scriptNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/5601
2022-02-25[DOC] Enhanced RDoc for some encoding methods (#5598)Burdette Lamar
In String, treats: #b #scrub #scrub! #unicode_normalize #unicode_normalize! #encode #encode! Also adds a note to IO.new (suggested by @jeremyevans). Notes: Merged-By: BurdetteLamar <BurdetteLamar@Yahoo.com>
2022-02-02Add `econv_get_encoding` functionS.H
Notes: Merged: https://github.com/ruby/ruby/pull/5509 Merged-By: nobu <nobu@ruby-lang.org>
2021-10-25[Feature #18239] Implement VWA for stringsPeter Zhu
This commit adds support for embedded strings with variable capacity and uses Variable Width Allocation to allocate strings. Notes: Merged: https://github.com/ruby/ruby/pull/4933
2021-10-03Using NIL_P macro instead of `== Qnil`S.H
Notes: Merged: https://github.com/ruby/ruby/pull/4925 Merged-By: nobu <nobu@ruby-lang.org>
2021-10-02Make encoding loading not issue warningJeremy Evans
Instead of relying on setting an unsetting ruby_verbose, which is not thread-safe, restructure require_internal and load_lock to accept a warn argument for whether to warn, and add rb_require_internal_silent to require without warnings. Use rb_require_internal_silent when loading encoding. Note this does not modify ruby_debug and errinfo handling, those remain thread-unsafe. Also silent requires when loading transcoders. Notes: Merged: https://github.com/ruby/ruby/pull/4887
2021-09-11Using SYMBOL_P macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/4798
2021-09-05Replace RBOOL macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/4791
2021-07-27Extracted repeatedly defined IDsNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/4684
2021-07-27Don't redefine #rb_intern over and over againNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/4684
2021-06-26Work around issue transcoding issue with non-ASCII compatible encodings and ↵Jeremy Evans
xml escaping When using a non-ASCII compatible source and destination encoding and xml escaping (the :xml option to String#encode), the resulting string was broken, as it used the correct non-ASCII compatible encoding, but contained data that was ASCII-compatible instead of compatible with the string's encoding. Work around this issue by detecting the case where both the source and destination encoding are non-ASCII compatible, and transcoding the source string from the non-ASCII compatible encoding to UTF-8. The xml escaping code will correctly handle the UTF-8 source string and the return the correctly encoded and escaped value. Fixes [Bug #12052] Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org> Notes: Merged: https://github.com/ruby/ruby/pull/4605 Merged-By: jeremyevans <code@jeremyevans.net>
2021-04-26Fix some typos by spell checkerRyuta Kamizono
Notes: Merged: https://github.com/ruby/ruby/pull/4414
2021-01-06fix result of example [ci skip]Nobuhiro IMAI
Notes: Merged: https://github.com/ruby/ruby/pull/4031
2020-12-22Data: delete卜部昌平
Has been deprecated since 684bdf6171b76f5bc5e4f05926a5ab01ec2b4fd5. Matz says in [ruby-core:83954] that Data should be an alias of Object. Because rb_cData has not been deprecated, let us deprecate the constant to make it a C-level synonym of rb_cObject. Notes: Merged: https://github.com/ruby/ruby/pull/3961
2020-12-03Update documentation for String#encode{,!} [ci skip]Jeremy Evans
These methods take keywords, not a hash. From tjschuck (T.J. Schuck) Fixes [Bug #17364]
2020-08-15RARRAY_AREF: convert into an inline function卜部昌平
RARRAY_AREF has been a macro for reasons. We might not be able to change that for public APIs, but why not relax the situation internally to make it an inline function. Notes: Merged: https://github.com/ruby/ruby/pull/3419
2020-06-29make_econv_exception: do not goto into a branch卜部昌平
I'm not necessarily against every goto in general, but jumping into a branch is definitely a bad idea. Better refactor. Notes: Merged: https://github.com/ruby/ruby/pull/3247
2020-05-11sed -i 's|ruby/impl|ruby/internal|'卜部昌平
To fix build failures. Notes: Merged: https://github.com/ruby/ruby/pull/3079
2020-05-11sed -i s|ruby/3|ruby/impl|g卜部昌平
This shall fix compile errors. Notes: Merged: https://github.com/ruby/ruby/pull/3079
2020-04-08Merge pull request #2991 from shyouhei/ruby.h卜部昌平
Split ruby.h Notes: Merged-By: shyouhei <shyouhei@ruby-lang.org>
2020-02-24Fixed symbol misused as IDNobuyoshi Nakada
`rb_funcallv_public` and `rb_respond_to` require an `ID`, not a `Symbol`. [Bug #16649]
2020-02-07more on NULL versus functions.卜部昌平
Function pointers are not void*. See also ce4ea956d24eab5089a143bba38126f2b11b55b6 8427fca49bd85205f5a8766292dd893f003c0e48
2020-01-11Warn when :newline precedes other newline optionsNobuyoshi Nakada
2019-12-26decouple internal.h headers卜部昌平
Saves comitters' daily life by avoid #include-ing everything from internal.h to make each file do so instead. This would significantly speed up incremental builds. We take the following inclusion order in this changeset: 1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very first thing among everything). 2. RUBY_EXTCONF_H if any. 3. Standard C headers, sorted alphabetically. 4. Other system headers, maybe guarded by #ifdef 5. Everything else, sorted alphabetically. Exceptions are those win32-related headers, which tend not be self- containing (headers have inclusion order dependencies). Notes: Merged: https://github.com/ruby/ruby/pull/2711
2019-11-18Deprecate taint/trust and related methods, and make the methods no-opsJeremy Evans
This removes the related tests, and puts the related specs behind version guards. This affects all code in lib, including some libraries that may want to support older versions of Ruby. Notes: Merged: https://github.com/ruby/ruby/pull/2476
2019-11-18Warn on access/modify of $SAFE, and remove effects of modifying $SAFEJeremy Evans
This removes the security features added by $SAFE = 1, and warns for access or modification of $SAFE from Ruby-level, as well as warning when calling all public C functions related to $SAFE. This modifies some internal functions that took a safe level argument to no longer take the argument. rb_require_safe now warns, rb_require_string has been added as a version that takes a VALUE and does not warn. One public C function that still takes a safe level argument and that this doesn't warn for is rb_eval_cmd. We may want to consider adding an alternative method that does not take a safe level argument, and warn for rb_eval_cmd. Notes: Merged: https://github.com/ruby/ruby/pull/2476
2019-07-24transcode.c (rb_econv_open0): remove unused codeYusuke Endoh
Coverity Scan found this.
2019-07-14transcode.c (rb_trans_conv): remove unnecessary assignmentsYusuke Endoh
This change will suppress Coverity Scan warnings
2018-12-06Prefer rb_check_arity when 0 or 1 argumentsnobu
Especially over checking argc then calling rb_scan_args just to raise an ArgumentError. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66238 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-01Fix call-seq of Encoding::Converter#putback [ci skip]kazu
[Fix GH-1999] From: Takayuki Matsubara <takayuki.1229@gmail.com> git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65487 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-29* expand tabs.svn
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64881 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-15 19:31:37 +0000