summaryrefslogtreecommitdiff
path: root/tool
AgeCommit message (Collapse)Author
2008-09-04* tool/transcode-tblgen.rb (citrus_decode_mapsrc): support older 1.8.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19116 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* transcode_data.h (rb_transcoding): remove stateful field.akr
add state field. (TRANSCODING_STATE): defined. (rb_transcoder): add fields: state_size, state_init_func, state_fini_func. change rb_transcoding* argument to void*. * transcode.c (transcode_restartable0): use TRANSCODING_STATE for first arguments of transcoder functions. (rb_transcoding_open_by_transcoder): initialize state field. (rb_transcoding_close): finalize state field. * tool/transcode-tblgen.rb: provide state size/init/fini. * enc/trans/newline.trans (universal_newline_init): defined. (fun_so_universal_newline): take void* as a state pointer. (rb_universal_newline): provide state size/init/fini. (rb_crlf_newline): ditto. (rb_cr_newline): ditto. * enc/trans/iso2022.trans (iso2022jp_init): defined. (fun_si_iso2022jp_to_eucjp): take void* as a state pointer. (fun_so_iso2022jp_to_eucjp): ditto. (fun_so_eucjp_to_iso2022jp): ditto. (iso2022jp_reset_sequence_size): ditto. (finish_eucjp_to_iso2022jp): ditto. (rb_ISO_2022_JP_to_EUC_JP): provide state size/init/fini. (rb_EUC_JP_to_ISO_2022_JP): ditto. * enc/trans/utf_16_32.trans (fun_so_from_utf_16be): take void* as a state pointer. (fun_so_to_utf_16be): ditto. (fun_so_from_utf_16le): ditto. (fun_so_to_utf_16le): ditto. (fun_so_from_utf_32be): ditto. (fun_so_to_utf_32be): ditto. (fun_so_from_utf_32le): ditto. (fun_so_to_utf_32le): ditto. (rb_from_UTF_16BE): provide state size/init/fini. (rb_to_UTF_16BE): ditto. (rb_from_UTF_16LE): ditto. (rb_to_UTF_16LE): ditto. (rb_from_UTF_32BE): ditto. (rb_to_UTF_32BE): ditto. (rb_from_UTF_32LE): ditto. (rb_to_UTF_32LE): ditto. * enc/trans/japanese.trans (fun_so_eucjp2sjis): take void* as a state pointer. (fun_so_sjis2eucjp): ditto. (rb_eucjp2sjis): provide state size/init/fini. (rb_sjis2eucjp): provide state size/init/fini. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19096 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* transcode_data.h (WORDINDEX_SHIFT_BITS): defined.akr
(WORDINDEX2INFO): defined. (INFO2WORDINDEX): defined. * tool/transcode-tblgen.rb: use WORDINDEX2INFO. * transcode.c: use INFO2WORDINDEX. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* transcode_data.h (rb_transcoder): new field: byte_array_length andakr
word_array_length. * tool/transcode-tblgen.rb (transcode_generated_code): generate byte_array_length and word_array_length. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19088 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* tool/transcode-tblgen.rb (ArrayCode): new class.akr
(ActionMap#gen_array_code): moved to ArrayCode. (ActionMap#numelt_array_code): ditto. (ActionMap#array_code_insert_at_last): ditto. (TRANSCODE_GENERATED_BYTES_CODE): use ArrayCode. (TRANSCODE_GENERATED_WORDS_CODE): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03* tool/transcode-tblgen.rb (ActionMap#gen_array_code): extracted fromakr
generate_lookup_node. (ActionMap#numelt_array_code): ditto. (ActionMap#array_code_insert_at_last): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-02* transcode_data.h (base_element): removed.akr
(BYTE_LOOKUP): removed. (BYTE_LOOKUP_BASE): don't cast. (BYTE_LOOKUP_INFO): ditto. (PType): unsigned int, instead of uintptr_t. (rb_transcoding): change type of next_field, conv_tree_start and word_array. * tool/transcode-tblgen.rb: generate word_array as array of unsigned int. * transcode.c (transcode_restartable0): follow the above type change. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19070 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-02* tool/transcode-tblgen.rb: add prefix for byte_array and word_array.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: comment removed in generated code.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19047 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: define TRANSCODE_TABLE_INFO in generatedakr
code. use it in rb_transcoder. * enc/trans/newline.trans: use TRANSCODE_TABLE_INFO. * enc/trans/iso2022.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: record infos and BYTE_LOOKUPs as index ofakr
word_array to avoid relocation. * transcode.c (transcode_restartable0): add word_array to get infos and BYTE_LOOKUPs. * transcode_data.h (BYTE_LOOKUP_INFO): change return type to uintptr_t. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: don't need to cast offsets array.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: record offsets array as index ofakr
byte_array to avoid relocation. * transcode.c (transcode_restartable0): add byte_array to get offsets array. * transcode_data.h (BYTE_LOOKUP_BASE): change return type to uintptr_t. (rb_transcoder): add fields: byte_array, word_array and word_size. * enc/trans/newline.trans: follow rb_transcoder change. * enc/trans/iso2022.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: make infos arrays and BYTE_LOOKUPs intoakr
single array. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* transcode_data.h (BYTE_LOOKUP): change to uintptr_t array.akr
(BYTE_LOOKUP_BASE): follow the type change. (BYTE_LOOKUP_INFO): ditto. (PType): ditto. (rb_transcoding): ditto. * tool/transcode-tblgen.rb: follow the type change. * transcode.c: ditto. * enc/trans/newline.trans: ditto. * enc/trans/iso2022.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: gather infos arrays and BYTE_LOOKUPs.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19036 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: make offsets arrays into single array.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19032 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: gather offsets array at top.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19031 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb: ValidEncoding['eucJP-ms'] defined.akr
"\xA2\xAF".encode("utf-8", "eucJP-ms") should raise Encoding::ConversionUndefined, not Encoding::InvalidByteSequence. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19029 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01* tool/transcode-tblgen.rb (transcode_compile_tree): use the firstnaruse
mapping when some mappings are given for a character. [ruby-dev:36068] * tool/transcode-tblgen.rb: expandtab. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19018 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 * tool/transcode-tblgen.rb: set ERB source filename for error message.usa
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19016 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* tool/transcode-tblgen.rb: change "illegal" to "invalid".akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* tool/transcode-tblgen.rb (transcode_generated_code): defined forakr
generating table at once. (transcode_tblgen): returns an empty string. (transcode_generate_node): ditto. * enc/trans/newline.trans: use transcode_generated_code. * enc/trans/iso2022.trans: ditto. * enc/trans/single_byte.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/japanese.trans: ditto. * enc/trans/korean.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19006 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* tool/transcode-tblgen.rb (citrus_decode_mapsrc): print loggingakr
message on STDERR. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19005 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31* tool/transcode-tblgen.rb: add table generator from Citrus maps.naruse
* enc/trans/japanese.trans: use Citrus maps. * enc/trans/CP: add maps from Citrus. * enc/trans/JIS: ditto. * test/ruby/test_transcode.rb: Shift_JIS and EUC-JP doesn't support IBM extended characters. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19003 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25* common.mk (dist): tool/make-snapshot is executable.yugui
* tool/make-snapshot (package): supported release numbers 1.9.0-x. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18857 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15* transcode_data.h (rb_transcoder_stateful_type_t): defined.akr
(rb_transcoder): add field: stateful_type. * tool/transcode-tblgen.rb: generate stateful_type field as stateless_converter. * enc/trans/iso2022.trans: follow rb_transcoder change. * enc/trans/newline.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18650 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* include/ruby/encoding.h (rb_econv_output): declared.akr
* transcode_data.h (rb_transcoder): add resetsize_func field. * enc/trans/iso2022.trans (iso2022jp_reset_sequence_size): defined. (rb_EUC_JP_to_ISO_2022_JP): provede resetsize_func. * tool/transcode-tblgen.rb: set NULL for resetsize_func. * transcode.c (rb_econv_output): new function for inserting output. (output_replacement_character): use rb_econv_output. (transcode_loop): check return value of output_replacement_character. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18628 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14* tool/transcode-tblgen.rb: check unexpected actions.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18619 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13* tool/make-snapshot: fix for clean up.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18553 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13* tool/make-snapshot: try to find exported directory.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18550 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* tool/make-snapshot: fix for enc.mk.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18542 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* tool/transcode-tblgen.rb (#transcode_tblgen): slight messagematz
improvement. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18529 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12* transcode_data.h (TRANSCODE_ERROR): removed.akr
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid (but unique to ASCII-8BIT) character. * transcode.c (rb_eConversionUndefined): new error. (rb_eInvalidByteSequence): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* transcode_data.h (rb_transcoder): add resetstate_func field forakr
resetting a state of stateful encoding. * enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify finish_eucjp_to_iso2022jp for resetstate_func. * tool/transcode-tblgen.rb: specify NULL for resetstate_func. * transcode.c (output_replacement_character): call resetstate_func before appending the replacement character. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11* enc/trans/iso2022.trans: renamed from iso2022.erb.c.akr
* enc/trans/single_byte.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/korean.trans: ditto. * enc/trans/japanese.trans: ditto. * enc/depend: follow the renaming. * tool/build-transcode: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18488 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* tool/transcode-tblgen.rb: record checksum ofakr
transcode-tblgen.rb itself. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18473 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10* enc/Makefile.in (make-workdir): use MAKEDIRS.nobu
* enc/depend: makes target directory before compile/link. * tool/transcode-tblgen.rb: creates target directory. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18469 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09* transcode_data.h (rb_transcoding): add fields for restartable akr
transcoding. (rb_transcoder): add max_input field. from_unit_length field is renamed to input_unit_length. * tool/transcode-tblgen.rb: generate max_input field. * enc/trans/iso2022.erb.c: follow rb_transcoder change. * enc/trans/utf_16_32.erb.c: ditto. * transcode.c (PARTIAL_INPUT): new constant. (transcode_char_start): new function. (transcode_result_t): new type. (transcode_restartable): new function. (more_output_buffer): new function. (transcode_loop): use transcode_restartable. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-08* transcode_data.h (rb_transcoder): from_unit_length field added.akr
from_utf8 field removed. * tool/transcode-tblgen.rb: generate offsets range. follow rb_transcoder change. * transcode.c (transcode_loop): don't use from_utf8. make invalid region from_unit_length wise. * enc/trans/iso2022.erb.c: follow rb_transcoder and transcode_generate_node change. * enc/trans/utf_16_32.erb.c: follow rb_transcoder and transcode_generate_node change. explicit :invalid map removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-07* transcode_data.h (rb_transcoding): new field "stateful".akr
(rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. nute(23:52:53)% head -40 ChangeLog Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org> * transcode_data.h (rb_transcoding): new field "stateful". (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. * enc/trans/utf_16_32.erb.c: follow argument change of FUNso. [ruby-dev:35798] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-06* tool/transcode-tblgen.rb (transcode_generate_node): codeakr
argument removed. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18395 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-06* tool/transcode-tblgen.rb: distinguish UNDEF and INVALID.akr
[ruby-dev:35709] * transcode.c (transcode_loop): don't need rb_enc_mbclen now. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05* tool/transcode-tblgen.rb (ActionMap#eql?): use == to compare @map.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18383 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05* tool/transcode-tblgen.rb: show generating tables in verbose mode.akr
(transcode_generate_node): call ActionMap#generate_node with showing table name. * enc/trans/utf_16_32.erb.c: use transcode_generate_node. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18382 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05* common.mk (transcodes), tool/build-transcode: generates transcodenobu
sources. * enc/trans/{japanese,korean,single_byte,utf_16_32}.c: to be autogenerated now. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18376 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05* tool/build-transcode: new file.akr
* tool/transcode-tblgen.rb: new file. * enc/trans/make_transdb.rb: exclude *.erb.c. * enc/depend: exclude *.erb.c. * enc/trans/utf_16_32.erb.c: new file. * enc/trans/single_byte.erb.c: new file. * enc/trans/japanese.erb.c: new file. * enc/trans/korean.erb.c: new file. * enc/trans/iso-8859-2-tbl.rb: new file. * enc/trans/iso-8859-3-tbl.rb: new file. * enc/trans/iso-8859-4-tbl.rb: new file. * enc/trans/iso-8859-5-tbl.rb: new file. * enc/trans/iso-8859-6-tbl.rb: new file. * enc/trans/iso-8859-7-tbl.rb: new file. * enc/trans/iso-8859-8-tbl.rb: new file. * enc/trans/iso-8859-9-tbl.rb: new file. * enc/trans/iso-8859-10-tbl.rb: new file. * enc/trans/iso-8859-11-tbl.rb: new file. * enc/trans/iso-8859-13-tbl.rb: new file. * enc/trans/iso-8859-14-tbl.rb: new file. * enc/trans/iso-8859-15-tbl.rb: new file. * enc/trans/eucjp-tbl.rb: new file. * enc/trans/sjis-tbl.rb: new file. * enc/trans/euckr-tbl.rb: new file. * enc/trans/utf_16_32.c: regenerated. * enc/trans/single_byte.c: regenerated. * enc/trans/japanese.c: regenerated. * enc/trans/korean.c: regenerated. [ruby-dev:35730] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18373 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-01* tool/instruction.rb: RubyVM is not module.ko1
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17776 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-01* compile.c, insns.def, vm.c, vm_core.h: remove some insnsko1
(undef, alias, definemethod). Call RubyVM::FrozenCore's singleton method instead. Add "putiseq" and "putspecialobject" instructions. * id.c, id.h: add ids for above. * tool/parse.rb: "VM" no longer exists. Use RubyVM instead. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17755 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-06-21* tool/make-snapshot: ported to ruby.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17495 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-30 21:50:04 +0000