summaryrefslogtreecommitdiff
path: root/test/ruby/test_econv.rb
AgeCommit message (Collapse)Author
2008-09-03* include/ruby/io.h (rb_io_t): new fields: encs.ecopts andakr
writeconv_pre_ecopts. (MakeOpenFile): initialize them. * include/ruby/encoding.h (rb_str_transcode): take ecopts argument. (rb_econv_flags): removed. (rb_econv_prepare_opts): declared. (rb_econv_open_opts): declared. * io.c (make_writeconv): use rb_econv_open_opts. (make_readconv): ditto. (io_fwrite): follow rb_str_transcode change. (rb_io_extract_modeenc): use rb_econv_prepare_opts. (rb_file_open_generic): initialize encs.ecopts. (rb_file_open_internal): ditto. (rb_io_reopen): ditto. (argf_ecopts): defined. (argf_next_argv): set encs.ecopts. (io_encoding_set): use rb_econv_prepare_opts. (argf_set_encoding): set argf_ecopts. * gc.c (gc_mark_children): mark encs.ecopts and writeconv_pre_ecopts in T_FILE. * transcode.c (transcode_loop): take ecopts argument. use rb_econv_open_opts. (rb_econv_flags): removed. (rb_econv_prepare_opts): defined. (rb_econv_open_opts): defined. (str_transcode0): take ecopts. (str_transcode): use rb_econv_prepare_opts. (rb_str_transcode): take ecopts. (econv_init): accept hash argument. (econv_insert_output): follow rb_str_transcode change. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19109 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* include/ruby/encoding.h (rb_econv_set_replacemenet): declared.akr
* transcode.c (rb_econv_t): new fields: replacement_str, replacement_len, replacement_enc and replacement_allocated. (get_replacement_character): make len as size_t. (rb_econv_open_by_transcoder_entries): initialize the new fields. (rb_econv_close): deallocate replacement_str if it allocated. (make_replacement): new function. (output_replacement_character): use make_replacement. (rb_econv_set_replacemenet): defined. (econv_get_replacement): new method. (econv_set_replacement): new method. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19108 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* enc/trans/japanese.trans: new file.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* transcode.c (transcode.c): set source_encoding andakr
destination_encoding as encoding object. (ecerr_source_encoding): new method. (ecerr_destination_encoding): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18993 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* transcode.c (make_econv_exception): rename instance variable namesakr
for storing encoding names. (ecerr_source_encoding_name): method renamed. (ecerr_destination_encoding_name): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18987 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* transcode.c (econv_putback): associate encoding to the result.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18986 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* transcode.c (econv_insert_output): raise ArgumentError on failure.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18984 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* enc/trans/single_byte.trans (us_ascii_map): don't define 8bit bytes.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18982 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* transcode.c (econv_last_error): new method.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18980 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* transcode.c (econv_primitive_convert): make two arguments,akr
destination_byteoffset and destination_bytesize, optional. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18979 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30* transcode.c (econv_putback): make max argument optional.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18967 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30* transcode.c (rb_econv_putback): putback from last byte.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30* transcode.c (econv_putback): renamed from econv_primitive_putback.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18965 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30* transcode.c (econv_insert_output): renamed fromakr
econv_primitive_insert_output. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18964 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30* test/ruby/test_econv.rb: follow econv_primitive_errinfo change.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18963 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28* transcode.c (econv_finish): new method.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18913 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28* transcode.c (econv_convert): new method.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18912 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28* transcode.c (econv_primitive_convert): accept nil asakr
destination_bytesize for unlimited destination size. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18911 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28* transcode.c (econv_primitive_convert): accept nil as input for emptyakr
input. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18910 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26* include/ruby/encoding.h (rb_econv_result_t): enumeration constant:akr
econv_incomplete_input. * io.c (finish_writeconv): check econv_incomplete_input. * transcode.c (transcode_restartable0): return econv_incomplete_input for unexpected end of source buffer. (trans_sweep): check econv_incomplete_input. (rb_trans_conv): ditto. (rb_econv_convert0): ditto. (rb_econv_convert): ditto. (transcode_loop): ditto. (make_econv_exception): change message for econv_incomplete_input. (econv_result_to_symbol): return :incomplete_input for econv_incomplete_input. (ecerr_incomplete_input): new method. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18875 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25* transcode.c (make_econv_exception): show readagain part for invalidakr
byte sequence exception. store the part as an instance variable. (ecerr_readagain_bytes): new method to access the readagain part. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18850 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25* transcode.c (rb_econv_open_by_transcoder_entries): initializeakr
last_error. num_trans may be zero. (rb_econv_convert0): num_trans may be zero. (rb_econv_putbackable): ditto. (rb_econv_putback): ditto. (rb_econv_convert): input_ptr and output_ptr may be NULL. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18835 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24* include/ruby/encoding.h (rb_econv_open_exc): declared.akr
* transcode.c (rb_eNoConverter): new exception. (rb_econv_open_exc): new function. (transcode_loop): use rb_econv_open_exc. * io.c (make_writeconv): use rb_econv_open_exc. (make_readconv): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18803 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-23* include/ruby/encoding.h (ECONV_INVALID_MASK): defined.akr
(ECONV_INVALID_IGNORE): defined. (ECONV_INVALID_REPLACE): defined. (ECONV_UNDEF_MASK): defined. (ECONV_UNDEF_IGNORE): defined. (ECONV_UNDEF_REPLACE): defined. * transcode.c (INVALID_IGNORE): removed. (INVALID_REPLACE): removed. (UNDEF_IGNORE): removed. (UNDEF_REPLACE): removed. (rb_econv_convert0): renamed from rb_econv_convert. (rb_econv_convert): defined to call rb_econv_convert0 with replace/ignore behavior moved from transcode_loop. (transcode_loop): replace/ignore behavior removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18787 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-22* include/ruby/io.h (FMODE_TEXTMODE): defined.akr
* include/ruby/encoding.h (rb_econv_t): new field: flags. (rb_econv_binmode): declared. * io.c (io_unread): text mode hack removed. (NEED_NEWLINE_DECODER): defined. (NEED_NEWLINE_ENCODER): defined. (NEED_READCONV): defined. (NEED_WRITECONV): defined. (TEXTMODE_NEWLINE_ENCODER): defined for windows. (make_writeconv): setup converter with TEXTMODE_NEWLINE_ENCODER for text mode. (io_fwrite): use NEED_WRITECONV. character code conversion is disabled if fptr->writeconv_stateless is nil. (make_readconv): setup converter with ECONV_UNIVERSAL_NEWLINE_DECODER for text mode. (read_all): use NEED_READCONV. (appendline): use NEED_READCONV. (rb_io_getline_1): use NEED_READCONV. (io_getc): use NEED_READCONV. (rb_io_ungetc): use NEED_READCONV. (rb_io_binmode): OS-level text mode test removed. call rb_econv_binmode. (rb_io_binmode_m): call rb_io_binmode_m with write_io as well. (rb_io_flags_mode): return mode string including "t". (rb_io_mode_flags): detect "t" for text mode. (rb_sysopen): always specify O_BINARY. * transcode.c (rb_econv_open_by_transcoder_entries): initialize flags. (rb_econv_open): if source and destination encoding is both empty string, open newline converter. last_tc will be NULL in this case. (rb_econv_encoding_to_insert_output): last_tc may be NULL now. (rb_econv_string): ditto. (output_replacement_character): ditto. (transcode_loop): ditto. (econv_init): ditto. (econv_inspect): ditto. (rb_econv_binmode): new function. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-17* transcode.c (econv_primitive_putback): new method.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18671 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-17* transcode.c (make_econv_exception): add several instance variablesakr
to exception object. (ecerr_source_encoding): new method: Encoding::ConversionUndefined#source_encoding and Encoding::InvalidByteSequence#source_encoding. (ecerr_destination_encoding): new method: Encoding::ConversionUndefined#destination_encoding and Encoding::InvalidByteSequence#destination_encoding. (econverr_error_char): new method: Encoding::ConversionUndefined#error_char. (econverr_error_bytes): new method: Encoding::ConversionUndefined#error_bytes. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18669 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-16* include/ruby/encoding.h (rb_econv_t): add fields: in_buf_start,akr
in_data_start, in_data_end, in_buf_end and last_trans_index. (rb_econv_output): removed. (rb_econv_insert_output): declared. (rb_econv_encoding_to_insert_output): declared. * enc/trans/newline.trans (rb_universal_newline): stateful_type changed. * transcode.c (transcode_restartable0): initialize inchar_start, tc->recognized_len and next_table at beginning of the loop. (rb_econv_open_by_transcoder_entries): initialize new fields. (rb_econv_open): setup last_trans_index. (trans_sweep): last out_buf_start can be non-NULL now. (rb_econv_convert): check last out_buf_start and in_buf_start at first. (rb_econv_output_with_destination_encoding): removed. (econv_just_convert): removed. (rb_econv_output): removed. (econv_primitive_output): method removed. (rb_econv_encoding_to_insert_output): new function. (allocate_converted_string): new function. (rb_econv_insert_output): new function. (econv_primitive_insert_output): new method. (output_replacement_character): use rb_econv_insert_output. unused arguments removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18654 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15add a test.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18648 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15* transcode.c (econv_just_convert): extracted from rb_econv_output.akr
(rb_econv_output): use econv_just_convert. (econv_primitive_output): new method. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18647 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15* include/ruby/encoding.h (rb_econv_t): new field: last_error.akr
* transcode.c (rb_trans_conv): new argument: result_position_ptr. (rb_econv_convert): fill last_error. (econv_result_to_symbol): extracted from econv_primitive_convert. (econv_primitive_errinfo): new method. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18643 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15add tests.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18638 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15add a test.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18636 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15add tests.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18635 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15* transcode.c (econv_init): don't create dummy encoding ifakr
rb_econv_open is failed. (make_dummy_encoding): new function extracted from make_encoding. (make_encoding): removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18634 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* transcode.c (econv_primitive_convert): set destination_bufferakr
encoding. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18630 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* include/ruby/encoding.h (rb_econv_result_t): moved fromakr
transcode_data.h. (rb_econv_elem_t): ditto. (rb_econv_t): ditto. source_encoding and destination_encoding field is added. (rb_econv_open): declared. (rb_econv_convert): ditto. (rb_econv_close): ditto. * transcode.c (rb_econv_open_by_transcoder_entries): initialize source_encoding and destination_encoding field as NULL. (rb_econv_open): make it external linkage. (rb_econv_close): ditto. (rb_econv_convert): ditto. renamed from rb_econv_conv. (make_encoding): new function. (econv_init): use make_encoding and store rb_encoding* in rb_econv_t. (econv_source_encoding): new method Encoding::Converter#source_encoding. (econv_destination_encoding): new method Encoding::Converter#destination_encoding. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18625 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* transcode.c (econv_init): accept Encoding object as source_encodingakr
and destination_encoding. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18623 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* transcode_data.h (transcode_invalid_byte_sequence): renamed fromakr
transcode_invalid_input. (transcode_destination_buffer_full): renamed from transcode_obuf_full. (transcode_source_buffer_empty): renamed from transcode_ibuf_empty. (rb_econv_result_t): renamed from rb_trans_result_t. (rb_econv_elem_t): renamed from rb_trans_elem_t. (rb_econv_t): renamed from rb_trans_t. * transcode.c (UNIVERSAL_NEWLINE_DECODER): renamed from UNIVERSAL_NEWLINE. (CRLF_NEWLINE_ENCODER): renamed from CRLF_NEWLINE. (CR_NEWLINE_ENCODER): renamed from CR_NEWLINE. (rb_econv_open): renamed from rb_trans_open. (rb_econv_close): renamed from rb_trans_close. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18618 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14add a test.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18616 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* transcode.c (rb_econv_conv): new function. it don't consume inputakr
too much, even for multilevel conversion. (transcode_loop): use rb_econv_conv. (econv_primitive_convert): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18610 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* transcode_data.h (rb_trans_result_t): new enumeration constant:akr
transcode_output_followed_by_input. * transcode.c (OUTPUT_FOLLOWED_BY_INPUT): new flag. (transcode_restartable0): suspend when output followed by input if OUTPUT_FOLLOWED_BY_INPUT is specified. (trans_sweep): check OUTPUT_FOLLOWED_BY_INPUT. (rb_trans_conv): support OUTPUT_FOLLOWED_BY_INPUT. (econv_primitive_convert): return :output_followed_by_input for transcode_output_followed_by_input. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18608 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14test updated.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18598 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13* transcode.c (econv_primitive_convert): add output_byteoffsetakr
argument. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18591 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13* transcode.c (econv_init): make flags argument optional.akr
(econv_primitive_convert): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13* enc/trans/newline.trans (rb_crlf_newline): new transcoder.akr
(rb_cr_newline): new transcoder. * transcode.c (trans_open_i): one more exra room for input newline converter. (rb_trans_open): crlf newline and cr newline implemented. (Init_transcode): Encoding::Converter::CRLF_NEWLINE and Encoding::Converter::LF_NEWLINE defined. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18557 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13* enc/trans/newline.trans: new file.akr
* transcode_data.h (rb_trans_t): add last_tc field. * transcode.c (UNIVERSAL_NEWLINE): defined. (CRLF_NEWLINE): defined. (CR_NEWLINE): defined. (rb_trans_open_by_transcoder_entries): initialize last_tc. (trans_open_i): allocate one more room for newline converter. (rb_trans_open): universal newline implemented. (more_output_buffer): take max_output argument instead ts. (output_replacement_character): take tc argument instead of ts. (transcode_loop): use last_tc field. (econv_init): add flags argument for rb_trans_open. (Init_transcode): Encoding::Converter::UNIVERSAL_NEWLINE defined. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18556 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (econv_primitive_convert): add output_size argument.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18548 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (rb_trans_conv): report last transcode_obuf_full.akr
(econv_max_output): new method Encoding::Converter#max_output. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18547 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode.c (rb_cEncodingConverter): new class Encoding::Converter.akr
(econv_free): new function. (econv_s_allocate): ditto. (econv_init): ditto. (check_econv): ditto. (econv_primitive_convert): new method. (Init_transcode): define Encoding::Converter. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18544 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-21 14:33:55 +0000