summaryrefslogtreecommitdiff
path: root/ext/zlib/zlib.c
AgeCommit message (Collapse)Author
2018-01-08zlib: reduce garbage on Zlib::GzipReader#readpartialnormal
For garbage-concious users who use the `outbuf' argument of `readpartial' to supply a destination buffer, this provides a drastic reduction in garbage when inflating large inputs in a streaming fashion. This results in a anonymous RSS reduction in the reader similar to the reduction in the writer from r61631. Results using the test script from r61631 <https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=61631> Before: writer 7.359999 0.000000 7.359999 ( 7.360639) writer RssAnon: 4040 kB reader 6.346667 0.070000 6.416667 ( 7.387654) reader RssAnon: 98272 kB After: writer 7.309999 0.000000 7.309999 ( 7.310651) writer RssAnon: 4048 kB reader 6.146666 0.003333 6.149999 ( 7.334868) reader RssAnon: 4300 kB * ext/zlib/zlib.c (struct read_raw_arg): new struct (gzfile_read_raw_partial): use read_raw_arg (gzfile_read_raw_rescue): ditto (gzfile_read_raw): accept outbuf, use read_raw_arg (gzfile_read_raw_ensure): accept outbuf (gzfile_read_header): ditto (gzfile_check_footer): ditto (gzfile_read_more): ditto (gzfile_read_raw_until_zero): adjust for changes (gzfile_fill): ditto (gzfile_readpartial): ditto (gzfile_read_all): ditto (gzfile_getc): ditto (gzfile_reader_end_run): ditto (gzfile_reader_get_unused): ditto (rb_gzreader_initialize): ditto (gzreader_skip_linebreaks): ditto (gzreader_gets): ditto (zlib_gunzip_run): ditto [ruby-core:84660] [Feature #14319] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61665 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-05zlib: reduce garbage on gzip writes (deflate)normal
Zlib::GzipWriter generated large amounts of garbage from (struct zstream).input. Reuse the .input field when it is hidden, and recycle it when its lifetime is over. This change alone reduced memory usage of the writer from 90MB to 4.5MB. For the detached buffer of compressed data used by gzfile_write_raw, we can only clear the string (not recycle it) since user code may hold references to it (but the data would be clobbered, anyways). This reduced memory usage slightly by around 0.5MB (because it's smaller compressed data). Combined, these changes reduce the anonymous RSS memory of a dedicated writer process from over 90MB to under 4MB. before: # user system total real writer 7.823332 0.053333 7.876665 ( 7.881464) writer RssAnon: 92944 kB reader 6.969999 0.076666 7.046665 ( 7.906377) reader RssAnon: 109820 kB after: writer 7.359999 0.000000 7.359999 ( 7.360639) writer RssAnon: 4040 kB reader 6.346667 0.070000 6.416667 ( 7.387654) reader RssAnon: 98272 kB Script used: ------- require 'zlib' require 'benchmark' nr = 16384 * 2 def stats(pfx, bm) str = "#{bm}#{File.readlines("/proc/#$$/status").grep(/^RssAnon:/)[0]}" puts str.gsub!(/^/m, pfx) end rd, wr = IO.pipe pid = fork do buf = ((0..255).map(&:chr).join * 128).freeze rd.close gzip = Zlib::GzipWriter.new(wr) bm = Benchmark.measure do nr.times { gzip.write(buf) } gzip.close wr.close end stats('writer ', bm) end wr.close buf = '' gunzip = Zlib::GzipReader.new(rd) n = 0 bm = Benchmark.measure do begin gunzip.readpartial(16384, buf) n += buf.size rescue EOFError break end while true end stats('reader ', bm) Process.waitall ------- * ext/zlib/zlib.c (zstream_discard_input): reuse or recycle hidden input (zstream_reset_input): clear hidden input (zstream_run): detach input and recycle after use (gzfile_write_raw): clear buffer after write [ruby-core:84638] [Feature #14315] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61631 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-05zlib: remove redundant rb_obj_revealnormal
No need to reveal strings freshly created with rb_str_new. * ext/zlib/zlib.c (zstream_detach_input): remove redundant call git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61612 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-23zlib.c: multiple arguments to writenobu
* ext/zlib/zlib.c (rb_gzwriter_write): accepts multiple arguments. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-07zlib.c: ensure to freenobu
* ext/zlib/zlib.c (zlib_gunzip): gz0 is a structure variable on the stack, no longer valid after exit by an exception. ensure to free instead. [Bug #13982] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60131 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-06zlib.c: memory leak in gunzipnobu
* ext/zlib/zlib.c (zlib_gunzip): clear zstream to fix memory leak. [ruby-core:83162] [Bug #13982] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60130 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-09-27ext: adjust indent [ci skip]nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-14zlib.c: fix unnormalized Fixnumnobu
* ext/zlib/zlib.c (rb_gzfile_total_out): cast to long not to result in an unsigned long to normalized to Fixnum on LLP64 platforms. [ruby-core:81488] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59337 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-14Zlib::GzipReader#pos underflows after calling #ungetbyte or #ungetc at start ↵naruse
of file [Bug #13616] patched by Andrew Haines <andrew@haines.org.nz> [ruby-core:81488] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59333 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-01zlib.c: no buf_filled in zstreamnobu
* ext/zlib/zlib.c (zstream): manage capacity and size of `buf` instead of size and separated member `buf_filled`. reported by Christian Jalio (jalio) at https://hackerone.com/reports/211958 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-01zlib.c: zstream_buffer_ungetbytenobu
* ext/zlib/zlib.c (zstream_buffer_ungetbyte): simplify by using zstream_buffer_ungets(). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58525 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-01zlib.c: zstream_expand_buffer_non_streamnobu
* ext/zlib/zlib.c (zstream_expand_buffer_non_stream): rename from zstream_expand_buffer_without_gvl() and replace duplicate code in zstream_expand_buffer(). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-20rdoc for r57126 [ci skip]nobu
* ext/zlib/zlib.c: [DOC] update as keyword arguments. [Feature #13020] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57129 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-20Zlib.gzip uses kwargs instead of argc [Feature #13020]naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57126 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-11fix Zlib.gzip/gunzip documentnaruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57049 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-10zlib.c: replace with substringnobu
* ext/zlib/zlib.c (zstream_discard_input): replace with unread portion substring, not modifying the input buffer directly. [ruby-core:78567] [Bug #13021] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-10Fix document of gunzip and gzip [ci skip]kazu
- fix a typo (`GzipReadr` -> `GzipReader`) - `Zlib::GzipReader.new` does not take block - fix encoding git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57037 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-09Zlib.gzip and Zlib.gunzip [Feature #13020]naruse
Encode and Decode gzip data without creating files. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57035 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-10-26[DOC] replace Fixnum with Integer [ci skip]nobu
* numeric.c: [DOC] update document for Integer class. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56492 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-06-17Unnecessary checksnobu
* ext/bigdecimal/bigdecimal.c: FIX2INT and FIX2UINT imply the check for Fixnum. * ext/zlib/zlib.c: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55430 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-05-21* ext/zlib/zlib.c: remove hacky macro introduced at r30437.naruse
* ext/zlib/zlib.c (gzfile_make_header): cast as long (instead of int). * ext/zlib/zlib.c (gzfile_make_footer): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55105 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-09* ext/zlib/zlib.c: Document mtime header behavior with patch by @schneemszzak
Fixes [GH-1129]: https://github.com/ruby/ruby/pull/1129 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53785 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-19zlib.c: Fix typo [ci skip]nobu
* ext/zlib/zlib.c (Init_zlib): [DOC] Fix double-word typo and grammatical error. [Fix GH-1162] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53200 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15* ext/zlib/zlib.c: fix a typo.hsbt
[ci skip][fix GH-1149] Patch by @crismali git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53118 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-10-25use rb_check_arity [ci skip]nobu
* ext: use rb_check_arity and rb_error_arity to raise ArgumentError. [Feature #9025] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52275 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-07-10zlib.c: GC guardsnobu
* ext/zlib/zlib.c: prevent from GC. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51212 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-07-10zlib.c: protoizenobu
* ext/zlib/zlib.c (do_checksum): protoize the definition. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51211 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-06-11* ext/zlib/zlib.c: Fix indentation for rdoc.hsbt
[Bug #11221][ruby-core:69465] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50834 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-05-10* ext/zlib/zlib.c (rb_gzreader_external_encoding):glass
define GzipReader#external_encoding. [Bug #10900] * test/zlib/test_zlib.rb: test for above. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50460 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-03-08* ext/zlib/zlib.c (rb_gzfile_close): Don't raise on doubleakr
close for consistent to IO#close. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49893 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-10* ext/zlib/zlib.c: fix document of method signatures.hsbt
[Bug #10668][ruby-core:67186][ci skip] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49202 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-04ext/zlib/zlib.c: TypedData conversionnormal
* ext/zlib/zlib.c (zstream_mark, zstream_free): update signature (gzfile_mark, gzfile_free): ditto (zstream_memsize): new function for rb_data_type->dsize (gzfile_memsize): ditto (zstream_data_type, gzfile_data_type): new data types (zstream_new): Data_Make_Struct => TypedData_Make_Struct (gzfile_new): ditto (get_zstream, get_gzfile): Data_Get_Struct => TypedData_Get_Struct (rb_zstream_flush_next_in): ditto (rb_zstream_flush_next_out): ditto (rb_zstream_avail_out): ditto (rb_zstream_avail_in): ditto (rb_zstream_closed_p): ditto (rb_deflate_initialize): ditto (rb_deflate_init_copy): ditto (rb_inflate_initialize): ditto (gzfile_ensure_close): ditto (rb_gzfile_closed_p): ditto (rb_gzfile_path): ditto (rb_gzwriter_initialize): ditto (rb_gzreader_initialize): ditto (rb_gzreader_unused): ditto [ruby-core:65377] [Feature #10319] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47791 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-09-30protoize no-arguments functionsnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47744 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-09-17ext/zlib/zlib.c (struct gzfile): pack (288 => 272 bytes) on 64-bitnormal
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47619 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-09-17ext/zlib/zlib.c (zlib_mem_alloc): check overflownormal
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47613 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-08-30zlib: GzipReader#rewind preserves ZSTREAM_FLAG_GZFILEnormal
* ext/zlib/zlib.c (gzfile_reset): preserve ZSTREAM_FLAG_GZFILE [Bug #10101] * test/zlib/test_zlib.rb (test_rewind): test each_byte We must preserve the ZSTREAM_FLAG_GZFILE flag to prevent zstream_detach_buffer from: a) returning Qnil and breaking out of the `each_byte' loop b) yielding a large string to each_byte Note: the test case in bug report takes a long time. I found this bug because I noticed the massive time descrepancy between `each_byte' and `readbyte' loop before this patch. With this patch, `each_byte' and `readbyte' both take very long. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47327 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-07-27* ext/zlib/zlib.c: [DOC] Remove default value of Zlib constants, aszzak
they may change in the implementation without notice. Patched by @robin850 [Fixes GH-682] https://github.com/ruby/ruby/pull/682 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46976 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-11* ext/socket/ifaddr.c (IS_IFADDRS): Unused macro removed.akr
* ext/strscan/strscan.c (BUFSIZE): Ditto. * ext/zlib/zlib.c (OBJ_IS_FREED): Ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@45922 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-04-18zlib.c: return unshared stringnobu
* ext/zlib/zlib.c (zstream_shift_buffer): create new copied string since it cannot be shared ever. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@45625 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-02-21class.c: do nothing if copying selfnobu
* class.c (rb_mod_init_copy): do nothing if copying self. [ruby-dev:47989] [Bug #9535] * hash.c (rb_hash_initialize_copy): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@45076 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-01-28* ext/zlib/zlib.c (rb_zlib_adler32): [DOC] Add example for adler32zzak
Patch by Vajrasky Kok [Bug #9307] [ci skip] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@44723 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-01-12ext: use rb_sprintf() and rb_vsprintf() with PRIsVALUEnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@44572 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-08-28* ext/zlib/zlib.c (zstream_run): Fix handling of deflate streams thatdrbrain
need a dictionary but are being decompressed by Zlib::Inflate.inflate (which has no option to set a dictionary). Now Zlib::NeedDict is raised instead of crashing. [ruby-trunk - Bug #8829] * test/zlib/test_zlib.rb (TestZlibInflate): Test for the above. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@42720 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-05-31zlib.c: check EOFnobu
* ext/zlib/zlib.c (gzfile_read, gzfile_read_all, gzfile_getc), (gzreader_gets): check EOF. [ruby-core:55220] [Bug #8467] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@41014 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-05-13* include/ruby/ruby.h: constify RBasic::klass and addko1
RBASIC_CLASS(obj) macro which returns a class of `obj'. This change is a part of RGENGC branch [ruby-trunk - Feature #8339]. * object.c: add new function rb_obj_reveal(). This function reveal interal (hidden) object by rb_obj_hide(). Note that do not change class before and after hiding. Only permitted example is: klass = RBASIC_CLASS(obj); rb_obj_hide(obj); .... rb_obj_reveal(obj, klass); TODO: API design. rb_obj_reveal() should be replaced with others. TODO: modify constified variables using cast may be harmful for compiler's analysis and optimizaton. Any idea to prohibt inserting RBasic::klass directly? If rename RBasic::klass and force to use RBASIC_CLASS(obj), then all codes such as `RBASIC(obj)->klass' will be compilation error. Is it acceptable? (We have similar experience at Ruby 1.9, for example "RARRAY(ary)->ptr" to "RARRAY_PTR(ary)". * internal.h: add some macros. * RBASIC_CLEAR_CLASS(obj) clear RBasic::klass to make it internal object. * RBASIC_SET_CLASS(obj, cls) set RBasic::klass. * RBASIC_SET_CLASS_RAW(obj, cls) same as RBASIC_SET_CLASS without write barrier (planned). * RCLASS_SET_SUPER(a, b) set super class of a. * array.c, class.c, compile.c, encoding.c, enum.c, error.c, eval.c, file.c, gc.c, hash.c, io.c, iseq.c, marshal.c, object.c, parse.y, proc.c, process.c, random.c, ruby.c, sprintf.c, string.c, thread.c, transcode.c, vm.c, vm_eval.c, win32/file.c: Use above macros and functions to access RBasic::klass. * ext/coverage/coverage.c, ext/readline/readline.c, ext/socket/ancdata.c, ext/socket/init.c, * ext/zlib/zlib.c: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40691 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-04-16* ext/socket/option.c: Document synonymous methods, by windwiny [GH-277]zzak
* ext/stringio/stringio.c: ditto * ext/io/wait/wait.c: ditto * ext/gdbm/gdbm.c: ditto * ext/dl/cfunc.c: ditto * ext/zlib/zlib.c: ditto * ext/win32ole/win32ole.c: ditto * ext/dbm/dbm.c: ditto * ext/json/generator/generator.c: ditto * ext/date/date_core.c: ditto git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40319 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-04-01* ext/zlib/zlib.c (rb_gzfile_set_mtime): Use NUM2UINT.akr
The old logic doesn't work well on LP64 platforms as: .. -2**63-1 => error, -2**63 .. -2**62-1 => success, -2**62 .. -2**31-1 => error, -2**31 .. 2**31-1 => success, 2**31 .. 2**62-1 => error, 2**62 .. 2**64-1 => success, 2**64 .. => error. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40040 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-04-01* ext/zlib/zlib.c (Zlib::Inflate.new):eregon
Fix documentation syntax and naming errors. Based on patch by Robin Dupret. Fix GH-271. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40039 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-01-01* ext/zlib/zlib.c (Zlib::GzipFile): Fix typo by zed_0xffzzak
[Fixes Github #229] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@38675 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-12-23zlib.c: function namenobu
* ext/zlib/zlib.c (rb_gzreader_lines): fix function name. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@38569 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-26 09:20:45 +0000