summaryrefslogtreecommitdiffstats
path: root/transcode.c
Commit message (Collapse)AuthorAgeFilesLines
* * transcode.c (rb_econv_conv): new function. it don't consume inputakr2008-08-141-5/+28
| | | | | | | | | | too much, even for multilevel conversion. (transcode_loop): use rb_econv_conv. (econv_primitive_convert): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18610 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_trans_result_t): new enumeration constant:akr2008-08-141-8/+68
| | | | | | | | | | | | | | | | transcode_output_followed_by_input. * transcode.c (OUTPUT_FOLLOWED_BY_INPUT): new flag. (transcode_restartable0): suspend when output followed by input if OUTPUT_FOLLOWED_BY_INPUT is specified. (trans_sweep): check OUTPUT_FOLLOWED_BY_INPUT. (rb_trans_conv): support OUTPUT_FOLLOWED_BY_INPUT. (econv_primitive_convert): return :output_followed_by_input for transcode_output_followed_by_input. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18608 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * thread.c (rb_thread_priority_set): get rid of C99 feature.nobu2008-08-141-2/+2
| | | | | | | * transcode.c (transcode_search_path): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18606 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (load_transcoder): suppress warning.nobu2008-08-141-1/+3
| | | | | | | * missing/vsnprintf.c (BSD_vfprintf): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18601 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* rdoc updated.akr2008-08-131-5/+12
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18597 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (econv_primitive_convert): add output_byteoffsetakr2008-08-131-9/+36
| | | | | | | argument. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18591 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (rb_trans_open): constified.nobu2008-08-131-1/+3
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18590 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (econv_inspect): new method.akr2008-08-131-0/+18
| | | | | | | + git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18578 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_restartable0): several local variablesakr2008-08-131-9/+6
| | | | | | | removed. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18576 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): add fields: writebuf_off,akr2008-08-131-34/+81
| | | | | | | | | | | | | | writebuf_len and writebuf. (TRANSCODING_WRITEBUF): new macro. * transcode.c (transcode_restartable0): output until output buffer is really full. (rb_transcoding_open_by_transcoder): initialize writebuf_len, writebuf_off and writebuf. (rb_transcoding_close): finalize writebuf. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18573 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add rdoc.akr2008-08-131-0/+88
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18569 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (econv_init): make flags argument optional.akr2008-08-131-6/+18
| | | | | | | (econv_primitive_convert): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans (rb_crlf_newline): new transcoder.akr2008-08-131-1/+13
| | | | | | | | | | | | | | (rb_cr_newline): new transcoder. * transcode.c (trans_open_i): one more exra room for input newline converter. (rb_trans_open): crlf newline and cr newline implemented. (Init_transcode): Encoding::Converter::CRLF_NEWLINE and Encoding::Converter::LF_NEWLINE defined. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18557 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans: new file.akr2008-08-131-21/+46
| | | | | | | | | | | | | | | | | | | | * transcode_data.h (rb_trans_t): add last_tc field. * transcode.c (UNIVERSAL_NEWLINE): defined. (CRLF_NEWLINE): defined. (CR_NEWLINE): defined. (rb_trans_open_by_transcoder_entries): initialize last_tc. (trans_open_i): allocate one more room for newline converter. (rb_trans_open): universal newline implemented. (more_output_buffer): take max_output argument instead ts. (output_replacement_character): take tc argument instead of ts. (transcode_loop): use last_tc field. (econv_init): add flags argument for rb_trans_open. (Init_transcode): Encoding::Converter::UNIVERSAL_NEWLINE defined. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18556 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_search_path): return the length of converterakr2008-08-131-59/+93
| | | | | | | | | | | | | | | | | path. (load_transcoder_entry): renamed from load_transcoder. (load_transcoder): new function for loding transcoder by encoding names. (rb_transcoding_open_by_transcoder): extracted from rb_transcoding_open. (rb_transcoding_open): use load_transcoder and rb_transcoding_open_by_transcoder. (rb_trans_open_by_transcoder_entries): new function. (trans_open_i): construct entries array. (rb_trans_open): use rb_trans_open_by_transcoder_entries. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18551 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (econv_primitive_convert): add output_size argument.akr2008-08-121-5/+10
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18548 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (rb_trans_conv): report last transcode_obuf_full.akr2008-08-121-19/+23
| | | | | | | (econv_max_output): new method Encoding::Converter#max_output. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18547 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (str_transcode): fix error message.naruse2008-08-121-2/+2
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18545 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (rb_cEncodingConverter): new class Encoding::Converter.akr2008-08-121-0/+90
| | | | | | | | | | | | (econv_free): new function. (econv_s_allocate): ditto. (econv_init): ditto. (check_econv): ditto. (econv_primitive_convert): new method. (Init_transcode): define Encoding::Converter. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18544 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (rb_trans_open): free ts before raise.akr2008-08-121-0/+1
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18535 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_trans_elem_t): new field: from and to.akr2008-08-121-14/+24
| | | | | | | | * transcode.c (trans_open_i): just record from and to. (rb_trans_open): load transcodings. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18531 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (trans_open_i): check the result of rb_transcoding_open.akr2008-08-121-1/+5
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (TRANSCODE_ERROR): removed.akr2008-08-121-4/+10
| | | | | | | | | | | * tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid (but unique to ASCII-8BIT) character. * transcode.c (rb_eConversionUndefined): new error. (rb_eInvalidByteSequence): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder): add resetstate_func field forakr2008-08-111-25/+60
| | | | | | | | | | | | | | | | resetting a state of stateful encoding. * enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify finish_eucjp_to_iso2022jp for resetstate_func. * tool/transcode-tblgen.rb: specify NULL for resetstate_func. * transcode.c (output_replacement_character): call resetstate_func before appending the replacement character. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (get_replacement_character): extracted fromakr2008-08-111-20/+26
| | | | | | | output_replacement_character. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18502 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (rb_trans_conv): find second last error.akr2008-08-111-4/+12
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_trans_result_t): new type.akr2008-08-111-127/+247
| | | | | | | | | | | | | | | | | | | | | | | | | | (rb_trans_elem_t): new type. (rb_trans_t): new type. * transcode.c (transcode_dispatch_cb): removed. (transcode_dispatch): removed. (rb_transcoding_result_t): moved to rb_trans_result_t in transcode_data.h. (transcode_restartable0): goto follow_info when FUNsi. (rb_transcoding_open): use get_transcoder_entry. (rb_trans_open): new function. (rb_trans_conv): ditto. (rb_trans_close): ditto. (trans_open_i): ditto. (trans_sweep): ditto. (more_output_buffer): take rb_trans_t instead of rb_transcoding as an argument. (transcode_loop): take from_encoding and to_encoding instead of tr as arguments. use rb_trans_open/rb_trans_conv/rb_trans_close. (str_transcode): don't use transcode_dispatch. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): rename fields.akr2008-08-111-30/+30
| | | | | | | | | | readlen -> recognized_len. feedlen -> readagain_len. * transcode.c: follow the field renaming. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18481 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_restartable0): invalid handling simplified.akr2008-08-101-31/+17
| | | | | | | | (transcode_restartable): use PARTIAL_INPUT for converting buffered input. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18476 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): new field: flags.akr2008-08-101-61/+110
| | | | | | | | | | | | | | * transcode.c (load_transcoder): extracted from transcode_dispatch_cb. (rb_transcoding_result_t): renamed from transcode_result_t. (rb_transcoding_open): new function. (rb_transcoding_convert): ditto. (rb_transcoding_close): ditto. (transcode_loop): use rb_transcoding_open, rb_transcoding_convert and rb_transcoding_close. (str_transcode): don't need rb_transcoding. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c: rename my_transcoding to tc and my_transcoder to tr.akr2008-08-101-96/+96
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18468 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): add feedlen field.akr2008-08-101-8/+52
| | | | | | | | | | | | | | | * transcode.c (transcode_restartable0): renamed from transcode_restartable. save input buffer into feed buffer if next character is started the point before input buffer. for example, "\x00\xd8\x01" then "\x02" in UTF-16LE. \x02 causes invalid and next character is started from \x01. (transcode_restartable): new function to call transcode_restartable0. if feed buffer is not empty, convert it at first. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18467 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_char_start): refactored to remove readlenakr2008-08-101-35/+28
| | | | | | | | argument. (transcode_restartable): refactored to remove readlen variable. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18464 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* r18455 reverted.akr2008-08-091-87/+92
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18462 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_loop): take destination and resize functionakr2008-08-091-14/+17
| | | | | | | | | | | | | | | as parameters. (more_output_buffer): ditto. (str_transcoding_resize): argument changed from rb_transcoding* to VALUE. (str_transcode): call transcode_loop with destination string and its resize function. * transcode_data.h (rb_transcoding): move ruby_string_dest and flush_func to transcode_loop parameters. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18458 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_char_start): arguments changed.akr2008-08-091-92/+87
| | | | | | | | (transcode_restartable): arguments changed to avoid *in_pos points out of buffer by decreasing *in_pos. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18455 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): new fields: next_info and next_byte.akr2008-08-091-10/+23
| | | | | | | | | * transcode.c (transcode_restartable): save/restore next_info and next_byte. sync readlen and in_p when invalid. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_restartable): my_transcoder argument removed.akr2008-08-091-8/+8
| | | | | | | (transcode_loop): my_transcoder argument removed. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18453 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): add fields for restartable akr2008-08-091-65/+334
| | | | | | | | | | | | | | | | | | | | | | transcoding. (rb_transcoder): add max_input field. from_unit_length field is renamed to input_unit_length. * tool/transcode-tblgen.rb: generate max_input field. * enc/trans/iso2022.erb.c: follow rb_transcoder change. * enc/trans/utf_16_32.erb.c: ditto. * transcode.c (PARTIAL_INPUT): new constant. (transcode_char_start): new function. (transcode_result_t): new type. (transcode_restartable): new function. (more_output_buffer): new function. (transcode_loop): use transcode_restartable. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder): from_unit_length field added.akr2008-08-081-13/+16
| | | | | | | | | | | | | | | | | | | | from_utf8 field removed. * tool/transcode-tblgen.rb: generate offsets range. follow rb_transcoder change. * transcode.c (transcode_loop): don't use from_utf8. make invalid region from_unit_length wise. * enc/trans/iso2022.erb.c: follow rb_transcoder and transcode_generate_node change. * enc/trans/utf_16_32.erb.c: follow rb_transcoder and transcode_generate_node change. explicit :invalid map removed. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): new field "stateful".akr2008-08-071-89/+223
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. nute(23:52:53)% head -40 ChangeLog Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org> * transcode_data.h (rb_transcoding): new field "stateful". (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. * enc/trans/utf_16_32.erb.c: follow argument change of FUNso. [ruby-dev:35798] git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * tool/transcode-tblgen.rb: distinguish UNDEF and INVALID.akr2008-08-061-5/+0
| | | | | | | | | [ruby-dev:35709] * transcode.c (transcode_loop): don't need rb_enc_mbclen now. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (str_encode): no need to duplicate first.nobu2008-08-051-14/+25
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (TRANSCODE_ERROR): common transcode failurenobu2008-08-051-2/+2
| | | | | | | | | | exception, would be changed later. * enc/trans/japanese.c (UNSUPPORTED_MODE): unsupported mode transition exception. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18363 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * marshal.c (dump_ensure), process.c (run_exec_dup2),nobu2008-08-041-0/+2
| | | | | | | | | string.c (rb_str_replace), transcode.c (transcode_dispatch): fixed memory leaks. based on patches from shinichiro.h <shinichiro.hamaji AT gmail.com> at [ruby-dev:35751]. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18341 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_loop): undefined character is replaced withnaruse2008-08-011-0/+5
| | | | | | only one character. [ruby-dev:35709] git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18312 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (output_replacement_character):naruse2008-07-311-18/+25
| | | | | | | | | | | | | | rename from _get_replacement_character. * transcode.c (output_replacement_character): fix replacement on UTF-32{BE,LE}. [ruby-dev:35705] * transcode.c (transcode_loop): ditto. * test/ruby/test_transcode.rb (test_invalid_replace): add for above. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (get_replacement_character): use U+FFFD as replacementnaruse2008-07-311-4/+4
| | | | | | | | | | | | character when convert to Unicode. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): rename from test_public_review_issue_121. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): enable option2. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18294 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (get_replacement_character): fix: invalid byte sequencenaruse2008-07-311-4/+4
| | | | | | is always replaced "\x00?". git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18293 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (transcode_loop): constified.nobu2008-07-141-9/+9
| | | | | | | | | | | | * transcode.c (str_transcode): rb_str_set_len() sets a delimiter. * transcode_data.h (rb_transcoder): constified preprocessor and postprocessor input. * enc/trans/japanese.c: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18066 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit (git 2.34.1) at 2026-01-10 14:27:08 +0000