summaryrefslogtreecommitdiff
path: root/test/ruby/test_transcode.rb
AgeCommit message (Collapse)Author
2009-03-06* {ext,lib,test}/**/*.rb: removed trailing spaces.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@22784 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-22Recover default_default_internal.naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21741 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-21Add test for String#encode.naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-14* enc/trans/gb18030.trans, gb18030-tbl.rb:duerst
new Chinese GB18030 transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) * transcode_data.h, transcode.c, tool/transcode_tblgen.rb: added support for GB18030-specific 4-byte sequences (with Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-04* enc/trans/gbk.trans, gbk-tbl.rb:duerst
new Chinese GBK transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21315 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-04* test/ruby/test_transcode.rb: added tests for GB2312duerst
(from Yoshihiro Kambayashi) * enc/trans/chinese.trans: set valid byte patterns for GB2312 and GB12345 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21314 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-04* enc/trans/big5.trans, big5-tbl.rb:duerst
new Chinese Big5 transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21313 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-12-25* transcode.c (str_transcode0): set encoding when String#encode wasnaruse
given explicit but the same destination and source encoding. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21047 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-12-09* enc/trans/single_byte.trans, cp850-tbl.rb, cp852-tbl.rb,duerst
cp855-tbl.rb, koi8-r-tbl.rb, koi8-u-tbl.rb, tis-620-tbl.rb: new single-byte transcodings (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi), small cosmetic fixes git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20599 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-11-11* test/ruby/test_transcode.rb: unnecessary setup methodduerst
(setup_really_needed?) removed git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20179 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-11-11* enc/trans/single_byte.trans, macgreek-tbl.rb, macroman-tbl.rb,duerst
macromania-tbl.rb, macturkish-tbl.rb, macukraine-tbl.rb, ibm437-tbl.rb, ibm852-tbl.rb, ibm855-tbl.rb, ibm857-tbl.rb, ibm860-tbl.rb, ibm861-tbl.rb, ibm862-tbl.rb, ibm863-tbl.rb, ibm865-tbl.rb, ibm866-tbl.rb, ibm869-tbl.rb, ibm775-tbl.rb: new single-byte transcodings (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20178 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-31* enc/trans/single_byte.trans, maccroatioan-tbl.rb,duerst
maccyrillic-tbl.rb, maciceland-tbl.rb: new single-byte transcodings (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20075 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-19* enc/trans/single_byte.trans: adding WINDOWS-wwww encodingsduerst
(wwww = 874/1250/1251/1253/1254/1255/1256/1257) (contributed by Yoshihiro Kambayashi) * enc/trans/windows-wwww-tbl.rb: 8 new files (contributed by Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added test_windows_wwww (contributed by Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-15* test/ruby/test_transcode.rb (TestTranscode#test_errors):matz
String#encode now works without any argument. [ruby-dev:36740] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19796 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-14* enc/trans/single_byte.trans: added windows-1252duerst
* enc/trans/windows-1252-tbl.rb: new file (contributed by Yoshihiro Kambayashi) * tool/transcode-tblgen.rb: listed windows-1252 as '1byte' * test/ruby/test_transcode.rb: added test_windows_1252 (contributed by Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19778 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-25* transcode.c (rb_eUndefinedConversionError): renamed fromakr
rb_eConversionUndefinedError. (rb_eConverterNotFoundError): renamed from rb_eNoConverterError. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19554 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-18* transcode.c: add "Error" suffix for Encoding exception classes.matz
a patch from Tadashi Saito <shiba at mail2.accsnet.ne.jp> in [ruby-dev:36346]. * encoding.c (Init_Encoding): rename EncodingCompatibilityError to Encoding::CompatibilityError. [ruby-dev:36366] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19407 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14* transcode.c (str_encode): returns duplicated string if nothingnobu
changed. [ruby-core:18578] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19349 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12add tests.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19301 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08add tests.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19252 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-05add tests.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19151 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-04* include/ruby/encoding.h (ECONV_INVALID_IGNORE): removed becauseakr
it tend to cause security problem. If the behaviour is really required, ECONV_INVALID_REPLACE with empty string can be used. For example, CVE-2006-2313, CVE-2008-1036, [ruby-core:15645] (ECONV_UNDEF_IGNORE): ditto. * transcode.c (rb_econv_convert): follow the above change. (econv_opts): ditto. (Init_transcode): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19123 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* include/ruby/io.h (rb_io_t): new fields: encs.ecopts andakr
writeconv_pre_ecopts. (MakeOpenFile): initialize them. * include/ruby/encoding.h (rb_str_transcode): take ecopts argument. (rb_econv_flags): removed. (rb_econv_prepare_opts): declared. (rb_econv_open_opts): declared. * io.c (make_writeconv): use rb_econv_open_opts. (make_readconv): ditto. (io_fwrite): follow rb_str_transcode change. (rb_io_extract_modeenc): use rb_econv_prepare_opts. (rb_file_open_generic): initialize encs.ecopts. (rb_file_open_internal): ditto. (rb_io_reopen): ditto. (argf_ecopts): defined. (argf_next_argv): set encs.ecopts. (io_encoding_set): use rb_econv_prepare_opts. (argf_set_encoding): set argf_ecopts. * gc.c (gc_mark_children): mark encs.ecopts and writeconv_pre_ecopts in T_FILE. * transcode.c (transcode_loop): take ecopts argument. use rb_econv_open_opts. (rb_econv_flags): removed. (rb_econv_prepare_opts): defined. (rb_econv_open_opts): defined. (str_transcode0): take ecopts. (str_transcode): use rb_econv_prepare_opts. (rb_str_transcode): take ecopts. (econv_init): accept hash argument. (econv_insert_output): follow rb_str_transcode change. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19109 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* enc/trans/japanese.trans: new file.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* enc/euc_jp.c (euc-jp-ms): euc-jp-ms is not EUC-JP not an alias ofnaruse
eucJP-ms. * enc/trans/japanese.trans (eucJP-ms): eucJP-ms is the correct name of the encoding in Ruby. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19021 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* test/ruby/test_transcode.rb (test_windows_31j): added.naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19013 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* tool/transcode-tblgen.rb: add table generator from Citrus maps.naruse
* enc/trans/japanese.trans: use Citrus maps. * enc/trans/CP: add maps from Citrus. * enc/trans/JIS: ditto. * test/ruby/test_transcode.rb: Shift_JIS and EUC-JP doesn't support IBM extended characters. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19003 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26* test/ruby/test_transcode.rb: added test_euc_jpduerst
(contributed by Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18862 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24* include/ruby/encoding.h (rb_econv_open_exc): declared.akr
* transcode.c (rb_eNoConverter): new exception. (rb_econv_open_exc): new function. (transcode_loop): use rb_econv_open_exc. * io.c (make_writeconv): use rb_econv_open_exc. (make_readconv): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18803 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-22* test/ruby/test_transcode.rb: test_shift_jis:duerst
fixed comment strings (see r18291) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18772 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode_data.h (TRANSCODE_ERROR): removed.akr
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid (but unique to ASCII-8BIT) character. * transcode.c (rb_eConversionUndefined): new error. (rb_eInvalidByteSequence): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11add a test.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18504 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode_data.h (rb_transcoder): add resetstate_func field forakr
resetting a state of stateful encoding. * enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify finish_eucjp_to_iso2022jp for resetstate_func. * tool/transcode-tblgen.rb: specify NULL for resetstate_func. * transcode.c (output_replacement_character): call resetstate_func before appending the replacement character. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode.c (rb_trans_conv): find second last error.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode_data.h (rb_trans_result_t): new type.akr
(rb_trans_elem_t): new type. (rb_trans_t): new type. * transcode.c (transcode_dispatch_cb): removed. (transcode_dispatch): removed. (rb_transcoding_result_t): moved to rb_trans_result_t in transcode_data.h. (transcode_restartable0): goto follow_info when FUNsi. (rb_transcoding_open): use get_transcoder_entry. (rb_trans_open): new function. (rb_trans_conv): ditto. (rb_trans_close): ditto. (trans_open_i): ditto. (trans_sweep): ditto. (more_output_buffer): take rb_trans_t instead of rb_transcoding as an argument. (transcode_loop): take from_encoding and to_encoding instead of tr as arguments. use rb_trans_open/rb_trans_conv/rb_trans_close. (str_transcode): don't use transcode_dispatch. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-08add several tests for UTF-32LE.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-08* transcode_data.h (rb_transcoder): from_unit_length field added.akr
from_utf8 field removed. * tool/transcode-tblgen.rb: generate offsets range. follow rb_transcoder change. * transcode.c (transcode_loop): don't use from_utf8. make invalid region from_unit_length wise. * enc/trans/iso2022.erb.c: follow rb_transcoder and transcode_generate_node change. * enc/trans/utf_16_32.erb.c: follow rb_transcoder and transcode_generate_node change. explicit :invalid map removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-07* transcode_data.h (rb_transcoding): new field "stateful".akr
(rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. nute(23:52:53)% head -40 ChangeLog Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org> * transcode_data.h (rb_transcoding): new field "stateful". (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. * enc/trans/utf_16_32.erb.c: follow argument change of FUNso. [ruby-dev:35798] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-06add tests for [ruby-dev:35726] and [ruby-dev:35709].akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31* transcode.c (output_replacement_character):naruse
rename from _get_replacement_character. * transcode.c (output_replacement_character): fix replacement on UTF-32{BE,LE}. [ruby-dev:35705] * transcode.c (transcode_loop): ditto. * test/ruby/test_transcode.rb (test_invalid_replace): add for above. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31* test/ruby/test_transcode.rb (test_unicode_public_review_issue_121):naruse
fix option1 and 3. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31* transcode.c (get_replacement_character): use U+FFFD as replacementnaruse
character when convert to Unicode. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): rename from test_public_review_issue_121. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): enable option2. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18294 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31test/ruby/test_transcode.rb: added test_shift_jisduerst
(contributed by Yoshihiro Kambayashi) and test_public_review_issue_121 (see http://www.unicode.org/review/pr-121.html) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18291 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-25* test/ruby/test_transcode.rb: refactoring/cleanup ofduerst
test_iso_2022_jp(_1) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18203 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-22* test/ruby/test_transcode.rb: added two commentsduerst
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18160 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-06-04* test/ruby/test_transcode.rb: add tests for iso-2022-jp.mame
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16821 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-03-16Sun Mar 16 18:07:07 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst
* enc/trans/utf_16_32.c: bug fix (some invalid UTF-8 sequences were legal) * test/ruby/test_transcode.rb: test for above bug git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15786 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-03-05Web Mar 5 17:43:43 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst
* transcode.c (transcode_loop): Adjusted detection of invalid (ill-formed) UTF-8 sequences. Fixing potential security issue, see http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes. * test/ruby/test_transcode.rb: Added two tests for above fix. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15692 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-02-21Thu Feb 21 17:15:15 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst
* transcode.c: Added basic support for passing options to String#encode via a hash. Currently only one option, with one value, is supported: invalid: :ignore (dropping invalid byte sequences instead of producing an error). Option naming is not yet stable! * test/ruby/test_transcode.rb: Added a single test for invalid: :ignore option. Not more tests because most data does not yet distinguish between INVALID and UNKNOWN. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15565 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-02-07* enc/trans/korean.c: add support for CP949 by Park Ji-In. [ruby-dev:33626]naruse
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15393 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-30 23:55:34 +0000