summaryrefslogtreecommitdiff
path: root/transcode.c
AgeCommit message (Collapse)Author
2008-08-13* transcode.c (transcode_search_path): return the length of converterakr
path. (load_transcoder_entry): renamed from load_transcoder. (load_transcoder): new function for loding transcoder by encoding names. (rb_transcoding_open_by_transcoder): extracted from rb_transcoding_open. (rb_transcoding_open): use load_transcoder and rb_transcoding_open_by_transcoder. (rb_trans_open_by_transcoder_entries): new function. (trans_open_i): construct entries array. (rb_trans_open): use rb_trans_open_by_transcoder_entries. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18551 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (econv_primitive_convert): add output_size argument.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18548 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (rb_trans_conv): report last transcode_obuf_full.akr
(econv_max_output): new method Encoding::Converter#max_output. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18547 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (str_transcode): fix error message.naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18545 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (rb_cEncodingConverter): new class Encoding::Converter.akr
(econv_free): new function. (econv_s_allocate): ditto. (econv_init): ditto. (check_econv): ditto. (econv_primitive_convert): new method. (Init_transcode): define Encoding::Converter. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18544 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (rb_trans_open): free ts before raise.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18535 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode_data.h (rb_trans_elem_t): new field: from and to.akr
* transcode.c (trans_open_i): just record from and to. (rb_trans_open): load transcodings. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18531 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (trans_open_i): check the result of rb_transcoding_open.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode_data.h (TRANSCODE_ERROR): removed.akr
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid (but unique to ASCII-8BIT) character. * transcode.c (rb_eConversionUndefined): new error. (rb_eInvalidByteSequence): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode_data.h (rb_transcoder): add resetstate_func field forakr
resetting a state of stateful encoding. * enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify finish_eucjp_to_iso2022jp for resetstate_func. * tool/transcode-tblgen.rb: specify NULL for resetstate_func. * transcode.c (output_replacement_character): call resetstate_func before appending the replacement character. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode.c (get_replacement_character): extracted fromakr
output_replacement_character. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18502 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode.c (rb_trans_conv): find second last error.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode_data.h (rb_trans_result_t): new type.akr
(rb_trans_elem_t): new type. (rb_trans_t): new type. * transcode.c (transcode_dispatch_cb): removed. (transcode_dispatch): removed. (rb_transcoding_result_t): moved to rb_trans_result_t in transcode_data.h. (transcode_restartable0): goto follow_info when FUNsi. (rb_transcoding_open): use get_transcoder_entry. (rb_trans_open): new function. (rb_trans_conv): ditto. (rb_trans_close): ditto. (trans_open_i): ditto. (trans_sweep): ditto. (more_output_buffer): take rb_trans_t instead of rb_transcoding as an argument. (transcode_loop): take from_encoding and to_encoding instead of tr as arguments. use rb_trans_open/rb_trans_conv/rb_trans_close. (str_transcode): don't use transcode_dispatch. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode_data.h (rb_transcoding): rename fields.akr
readlen -> recognized_len. feedlen -> readagain_len. * transcode.c: follow the field renaming. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18481 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* transcode.c (transcode_restartable0): invalid handling simplified.akr
(transcode_restartable): use PARTIAL_INPUT for converting buffered input. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18476 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* transcode_data.h (rb_transcoding): new field: flags.akr
* transcode.c (load_transcoder): extracted from transcode_dispatch_cb. (rb_transcoding_result_t): renamed from transcode_result_t. (rb_transcoding_open): new function. (rb_transcoding_convert): ditto. (rb_transcoding_close): ditto. (transcode_loop): use rb_transcoding_open, rb_transcoding_convert and rb_transcoding_close. (str_transcode): don't need rb_transcoding. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* transcode.c: rename my_transcoding to tc and my_transcoder to tr.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18468 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* transcode_data.h (rb_transcoding): add feedlen field.akr
* transcode.c (transcode_restartable0): renamed from transcode_restartable. save input buffer into feed buffer if next character is started the point before input buffer. for example, "\x00\xd8\x01" then "\x02" in UTF-16LE. \x02 causes invalid and next character is started from \x01. (transcode_restartable): new function to call transcode_restartable0. if feed buffer is not empty, convert it at first. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18467 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* transcode.c (transcode_char_start): refactored to remove readlenakr
argument. (transcode_restartable): refactored to remove readlen variable. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18464 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09r18455 reverted.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18462 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09* transcode.c (transcode_loop): take destination and resize functionakr
as parameters. (more_output_buffer): ditto. (str_transcoding_resize): argument changed from rb_transcoding* to VALUE. (str_transcode): call transcode_loop with destination string and its resize function. * transcode_data.h (rb_transcoding): move ruby_string_dest and flush_func to transcode_loop parameters. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18458 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09* transcode.c (transcode_char_start): arguments changed.akr
(transcode_restartable): arguments changed to avoid *in_pos points out of buffer by decreasing *in_pos. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18455 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09* transcode_data.h (rb_transcoding): new fields: next_info and next_byte.akr
* transcode.c (transcode_restartable): save/restore next_info and next_byte. sync readlen and in_p when invalid. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09* transcode.c (transcode_restartable): my_transcoder argument removed.akr
(transcode_loop): my_transcoder argument removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18453 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09* transcode_data.h (rb_transcoding): add fields for restartable akr
transcoding. (rb_transcoder): add max_input field. from_unit_length field is renamed to input_unit_length. * tool/transcode-tblgen.rb: generate max_input field. * enc/trans/iso2022.erb.c: follow rb_transcoder change. * enc/trans/utf_16_32.erb.c: ditto. * transcode.c (PARTIAL_INPUT): new constant. (transcode_char_start): new function. (transcode_result_t): new type. (transcode_restartable): new function. (more_output_buffer): new function. (transcode_loop): use transcode_restartable. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-08* transcode_data.h (rb_transcoder): from_unit_length field added.akr
from_utf8 field removed. * tool/transcode-tblgen.rb: generate offsets range. follow rb_transcoder change. * transcode.c (transcode_loop): don't use from_utf8. make invalid region from_unit_length wise. * enc/trans/iso2022.erb.c: follow rb_transcoder and transcode_generate_node change. * enc/trans/utf_16_32.erb.c: follow rb_transcoder and transcode_generate_node change. explicit :invalid map removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-07* transcode_data.h (rb_transcoding): new field "stateful".akr
(rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. nute(23:52:53)% head -40 ChangeLog Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org> * transcode_data.h (rb_transcoding): new field "stateful". (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. * enc/trans/utf_16_32.erb.c: follow argument change of FUNso. [ruby-dev:35798] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-06* tool/transcode-tblgen.rb: distinguish UNDEF and INVALID.akr
[ruby-dev:35709] * transcode.c (transcode_loop): don't need rb_enc_mbclen now. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05* transcode.c (str_encode): no need to duplicate first.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05* transcode_data.h (TRANSCODE_ERROR): common transcode failurenobu
exception, would be changed later. * enc/trans/japanese.c (UNSUPPORTED_MODE): unsupported mode transition exception. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18363 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-04* marshal.c (dump_ensure), process.c (run_exec_dup2),nobu
string.c (rb_str_replace), transcode.c (transcode_dispatch): fixed memory leaks. based on patches from shinichiro.h <shinichiro.hamaji AT gmail.com> at [ruby-dev:35751]. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18341 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-01* transcode.c (transcode_loop): undefined character is replaced withnaruse
only one character. [ruby-dev:35709] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18312 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31* transcode.c (output_replacement_character):naruse
rename from _get_replacement_character. * transcode.c (output_replacement_character): fix replacement on UTF-32{BE,LE}. [ruby-dev:35705] * transcode.c (transcode_loop): ditto. * test/ruby/test_transcode.rb (test_invalid_replace): add for above. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31* transcode.c (get_replacement_character): use U+FFFD as replacementnaruse
character when convert to Unicode. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): rename from test_public_review_issue_121. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): enable option2. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18294 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31* transcode.c (get_replacement_character): fix: invalid byte sequencenaruse
is always replaced "\x00?". git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18293 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-14* transcode.c (transcode_loop): constified.nobu
* transcode.c (str_transcode): rb_str_set_len() sets a delimiter. * transcode_data.h (rb_transcoder): constified preprocessor and postprocessor input. * enc/trans/japanese.c: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18066 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-14* transcode.c (get_replacement_character): temporary function,naruse
get characters for replacement mode. * transcode.c (transcode_loop): add undef key and replace value. * transcode.c (str_transcode): ditto. * transcode.c (Init_transcode): define sym_undef and sym_replace. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18062 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-12* configure.in (sizeof ptrdiff_t): check for size of ptrdiff_t.nobu
* include/ruby/ruby.h (PRI?PTRDIFF, PRI?SIZE): printf conversion specifiers for ptrdiff_t and size_t/ssize_t. * insns.def (leave), marshal.c (long_toobig), transcode.c (str_transcode), vm_dump.c (control_frame_dump, stack_dump_each), (debug_print_register, debug_print_pre): t and z length modifiers are C99. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-06-07* io.c (rb_f_open), re.c (rb_reg_search), transcode.c (str_transcode):nobu
suppress warnings. * util.c (quorem, rv_alloc, nrv_alloc): only used in dtoa(). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16873 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-06-01* insns.def (DEFINE_INSN): subtract of pointers is ptrdiff_t.naruse
this is not int on 64bit system. * vm_dump.c (control_frame_dump): ditto. * vm_dump.c (stack_dump_each): ditto. * vm_dump.c (debug_print_register): ditto. * vm_dump.c (debug_print_pre): ditto. * transcode.c (str_transcode): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16735 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-05-23* transcode.c (rb_str_transcode): argc is 1, and argv is &to.naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-05-20* transcode.c, include/ruby/encodng.h (rb_str_transcode):naruse
C API of encoding conversion for Ruby object. VALUE rb_str_transcode(VALUE str, VALUE to). * transcode.c (str_encode, str_encode_bang): rename from rb_tr_transcode or rb_str_transcode_bang. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16496 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-04-07* encoding.c (enc_init_db): moved to enc/encdb.c.nobu
* transcode.c (init_transcoder_table): moved to enc/trans/transdb.c. * enc/depend (enc/encdb.o enc/trans/transdb.o): depend on corresponding headers. * common.mk (COMMONOBJS): moved transcode.o from OBJS git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15915 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-03-06* transcode.c (rb_str_transcode_bang): set coderange.naruse
* transcode.c (rb_str_transcode): use rb_str_transcode_bang. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15707 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-03-05Web Mar 5 17:43:43 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst
* transcode.c (transcode_loop): Adjusted detection of invalid (ill-formed) UTF-8 sequences. Fixing potential security issue, see http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes. * test/ruby/test_transcode.rb: Added two tests for above fix. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15692 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-02-21Thu Feb 21 17:15:15 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst
* transcode.c: Added basic support for passing options to String#encode via a hash. Currently only one option, with one value, is supported: invalid: :ignore (dropping invalid byte sequences instead of producing an error). Option naming is not yet stable! * test/ruby/test_transcode.rb: Added a single test for invalid: :ignore option. Not more tests because most data does not yet distinguish between INVALID and UNKNOWN. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15565 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-29* enc/trans/japanese.c (rb_to_Windows_31J): to 'Windows-31J'.naruse
* common.mk: add rules for transdb.h. * transcode.c (init_transcoder_table): use transdb.h. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21Mon Jan 21 19:42:42 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst
* transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb: added UTF-32BE and UTF-32LE conversions. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15156 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21* transcode.c (str_transcode): initialize transcoder innobu
rb_transcoding. [ruby-dev:33234] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15153 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21* transcode.c (str_transcode): initialize transcoder innobu
rb_transcoding. [ruby-dev:33234] * transcode_data.h (rb_transcoding): transcoder constified. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15152 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-17 06:10:23 +0000