summaryrefslogtreecommitdiffstats
path: root/enc
Commit message (Collapse)AuthorAgeFilesLines
* fix typos.akr2008-09-161-7/+7
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/oniguruma.h (OnigEncodingTypeST): add precise_retakr2008-09-1615-84/+745
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | argument for mbc_to_code. (ONIGENC_MBC_TO_CODE): provide NULL for precise_ret. (ONIGENC_MBC_PRECISE_CODEPOINT): defined. * include/ruby/encoding.h (rb_enc_mbc_precise_codepoint): defined. * regenc.h (onigenc_single_byte_mbc_to_code): precise_ret argument added. (onigenc_mbn_mbc_to_code): ditto. * regenc.c (onigenc_single_byte_mbc_to_code): precise_ret argument added. (onigenc_mbn_mbc_to_code): ditto. * string.c (count_utf8_lead_bytes_with_word): removed. (str_utf8_nth): removed. (str_utf8_offset): removed. (str_strlen): UTF-8 codepoint oriented optimization removed. (rb_str_substr): ditto. (enc_succ_char): use rb_enc_mbc_precise_codepoint. (enc_pred_char): ditto. (rb_str_succ): ditto. * encoding.c (rb_enc_ascget): check length with rb_enc_mbc_precise_codepoint. (rb_enc_codepoint): use rb_enc_mbc_precise_codepoint. * regexec.c (string_cmp_ic): add text_end argument. (match_at): check end of character after exact string matches. * enc/utf_8.c (graphme_table): defined for extended graphme cluster boundary. (grapheme_cmp): defined. (get_grapheme_properties): defined. (grapheme_boundary_p): defined. (MAX_BYTES_LENGTH): defined. (comb_char_enc_len): defined. (mbc_to_code0): extracted from mbc_to_code. (mbc_to_code): use mbc_to_code0. (left_adjust_combchar_head): defined. (utf_8): use a extended graphme cluster as a unit. * enc/unicode.c (onigenc_unicode_mbc_case_fold): use ONIGENC_MBC_PRECISE_CODEPOINT to extract codepoints. (onigenc_unicode_get_case_fold_codes_by_str): ditto. * enc/euc_jp.c (mbc_to_code): follow mbc_to_code field change. use onigenc_mbn_mbc_to_code. * enc/shift_jis.c (mbc_to_code): ditto. * enc/emacs_mule.c (mbc_to_code): ditto. * enc/gbk.c (gbk_mbc_to_code): follow mbc_to_code field and onigenc_mbn_mbc_to_code change. * enc/cp949.c (cp949_mbc_to_code): ditto. * enc/big5.c (big5_mbc_to_code): ditto. * enc/euc_tw.c (euctw_mbc_to_code): ditto. * enc/euc_kr.c (euckr_mbc_to_code): ditto. * enc/gb18030.c (gb18030_mbc_to_code): ditto. * enc/utf_32be.c (utf32be_mbc_to_code): follow mbc_to_code field change. * enc/utf_16be.c (utf16be_mbc_to_code): ditto. * enc/utf_32le.c (utf32le_mbc_to_code): ditto. * enc/utf_16le.c (utf16le_mbc_to_code): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19389 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder): resetsize_func and resetstate_funcakr2008-09-151-1/+1
| | | | | | | | | | also returns ssize_t. * enc/trans/iso2022.trans: follow the type change. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19354 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h: return output functions ssize_t.akr2008-09-145-19/+19
| | | | | | | | | | | | | | | | | | | * transcode.c (transcode_restartable0): don't need to cast the result of output functions. * enc/trans/newline.trans: follow the type change. * enc/trans/escape.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19351 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h: output function takes output buffer size.akr2008-09-145-19/+19
| | | | | | | | | | | | | | | | | | * transcode.c: give output buffer size for output functions. * enc/trans/newline.trans: follow the type change. * enc/trans/escape.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19350 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/oniguruma.h (OnigEncodingTypeST): add end argument forakr2008-09-1314-21/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | left_adjust_char_head. (ONIGENC_LEFT_ADJUST_CHAR_HEAD): add end argument. (onigenc_get_left_adjust_char_head): ditto. * include/ruby/encoding.h (rb_enc_left_char_head): add end argument. * regenc.h (onigenc_single_byte_left_adjust_char_head): ditto. * regenc.c (onigenc_get_right_adjust_char_head): follow the interface change. (onigenc_get_right_adjust_char_head_with_prev): ditto. (onigenc_get_prev_char_head): ditto. (onigenc_step_back): ditto. (onigenc_get_left_adjust_char_head): ditto. (onigenc_single_byte_code_to_mbc): ditto. * re.c: ditto. * string.c: ditto. * io.c: ditto. * regexec.c: ditto. * enc/euc_jp.c: ditto. * enc/cp949.c: ditto. * enc/shift_jis.c: ditto. * enc/gbk.c: ditto. * enc/big5.c: ditto. * enc/euc_tw.c: ditto. * enc/euc_kr.c: ditto. * enc/emacs_mule.c: ditto. * enc/gb18030.c: ditto. * enc/utf_8.c: ditto. * enc/utf_16le.c: ditto. * enc/utf_16be.c: ditto. * enc/utf_32le.c: ditto. * enc/utf_32be.c: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19334 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans: transcoder name renamed to use underscore.akr2008-09-121-4/+4
| | | | | | | | * transcode.c: follow the renaming. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/oniguruma.h (OnigCodePoint): unsigned long to unsigned int.naruse2008-09-112-6/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * include/ruby/encoding.h (rb_enc_codepoint): ditto. * encoding.c (rb_enc_codepoint): signed int to unsigned int. * encoding.c (rb_enc_ascget): ditto. * string.c (rb_str_casecmp): ditto. * string.c (enc_succ_alnum_char): ditto. * string.c (rb_str_inspect): ditto. * string.c (rb_str_upcase_bang): ditto. * string.c (rb_str_downcase_bang): ditto. * string.c (rb_str_capitalize_bang): ditto. * string.c (rb_str_swapcase_bang): ditto. * string.c (struct tr): ditto. * string.c (trnext): ditto. * string.c (tr_trans): ditto. * string.c (tr_setup_table): ditto. * string.c (tr_find): ditto. * string.c (rb_str_delete_bang): ditto. * string.c (rb_str_squeeze_bang): ditto. * string.c (rb_str_count): ditto. * string.c (rb_str_split_m): ditto. * string.c (rb_str_each_line): ditto. * string.c (rb_str_lstrip_bang): ditto. * string.c (rb_str_rstrip_bang): ditto. * string.c (rb_str_intern): ditto. * dir.c (char_casecmp): ditto. * sprintf.c (rb_str_format): ditto. * enc/emacs_mule.c (mbc_to_code): to be 32bit clean. * enc/emacs_mule.c (code_to_mbc): ditto. * enc/gb18030.c (mbc_to_code): ditto. * enc/gb18030.c (code_to_mbc): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans (rb_universal_newline): swap src_encodingakr2008-09-091-2/+1
| | | | | | | | | | | | and dst_encoding. * transcode.c (rb_econv_decorate_at): call get_transcoder_entry only once. (rb_econv_binmode): follow universal_newline change. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19276 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (rb_econv_asciicompat_encoding): renamedakr2008-09-085-18/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | from rb_econv_stateless_encoding to apply stateless ASCII incompatible encodings such as UTF-16BE. * io.c (make_writeconv): use rb_econv_asciicompat_encoding. * transcode_data.h (rb_transcoder_asciicompat_type_t): renamed from rb_transcoder_stateful_type_t. (rb_transcoder): use rb_transcoder_asciicompat_type_t. * transcode.c: follow the type change. (asciicompat_encoding_i): renamed from stateless_encoding_i. (rb_econv_asciicompat_encoding): renamed from rb_econv_stateless_encoding. (econv_s_asciicompat_encoding): method renamed. * tool/transcode-tblgen.rb: follow the type change. * enc/trans/utf_16_32.trans: follow the type change. rb_from_UTF_16BE to UTF-8 is asciicompat_decoder. rb_from_UTF_16LE to UTF-8 is asciicompat_decoder. rb_from_UTF_32BE to UTF-8 is asciicompat_decoder. rb_from_UTF_32LE to UTF-8 is asciicompat_decoder. UTF-8 to rb_to_UTF_16BE is asciicompat_encoder. UTF-8 to rb_to_UTF_16LE is asciicompat_encoder. UTF-8 to rb_to_UTF_32BE is asciicompat_encoder. UTF-8 to rb_to_UTF_32LE is asciicompat_encoder. * enc/trans/newline.trans: follow the type change. universal newline decoder is asciicompat_converter. * enc/trans/escape.trans: follow the type change. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19249 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/iso2022.trans: upcase to iso-2022-jp.akr2008-09-072-5/+5
| | | | | | | | * enc/emacs_mule.c: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19221 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/iso2022.trans: stateless-iso-2022-jp is defined to avoidakr2008-09-072-40/+123
| | | | | | | | | | | undefined conversion error between iso-2022-jp and the corresponding stateless encoding. * enc/emacs_mule.c: replicate emacs-mule as stateless-iso-2022-jp. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19220 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans (hexstr): renamed from str1.akr2008-09-071-9/+9
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19218 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans: use transcode_tblgen.akr2008-09-071-58/+20
| | | | | | | | * tool/transcode-tblgen.rb: generate an empty line after str1. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19217 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (STR1): defined for a string up to 255 bytes.akr2008-09-071-51/+15
| | | | | | | | | | | | | | | | (STR1_BYTEINDEX): defined. (makeSTR1): defined. * tool/transcode-tblgen.rb: generate STR1. * transcode.c (transcode_restartable0): interpret STR1. * enc/trans/escape.trans (fun_so_escape_xml_chref): removed. STR1 is used instead. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (ECONV_XML_ATTR_CONTENT_ENCODER): defined.akr2008-09-071-24/+35
| | | | | | | | | | | | | | | | | | | | (ECONV_STATEFUL_ENCODER_MASK): defined. (ECONV_XML_ATTR_QUOTE_ENCODER): defined. (ECONV_XML_ATTR_ENCODER): removed. * enc/trans/escape.trans (rb_escape_xml_attr_content): defined. (rb_escape_xml_attr_quote): defined. (rb_escape_xml_attr): removed. * io.c (NEED_WRITECONV): writeconv is required if supplemental converter is used. (make_writeconv): apply stateful encoder in writeconv. * transcode.c: follow the constant change. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19209 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (ECONV_XML_TEXT_ENCODER): renamed fromakr2008-09-061-25/+25
| | | | | | | | | | | | | ECONV_HTML_TEXT_ENCODER. (ECONV_XML_ATTR_ENCODER): renamed from ECONV_HTML_ATTR_ENCODER. * enc/trans/escape.trans: follow the renaming. * transcode.c: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19191 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans (fun_so_escape_html_attr): fix return type.akr2008-09-061-1/+1
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19177 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans (escape_html_attr_init): new function.akr2008-09-061-9/+57
| | | | | | | | | | (fun_so_escape_html_attr): new function. (escape_html_attr_finish): new function. (rb_escape_html_attr): use them to quote the converted result. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19173 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/escape.trans: new file.akr2008-09-051-0/+109
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19165 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans (universal_newline_finish): new function.akr2008-09-051-2/+13
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19152 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/newline.trans: record newline types met in universalakr2008-09-041-12/+27
| | | | | | | newline decoder. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19136 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): remove stateful field.akr2008-09-034-45/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | add state field. (TRANSCODING_STATE): defined. (rb_transcoder): add fields: state_size, state_init_func, state_fini_func. change rb_transcoding* argument to void*. * transcode.c (transcode_restartable0): use TRANSCODING_STATE for first arguments of transcoder functions. (rb_transcoding_open_by_transcoder): initialize state field. (rb_transcoding_close): finalize state field. * tool/transcode-tblgen.rb: provide state size/init/fini. * enc/trans/newline.trans (universal_newline_init): defined. (fun_so_universal_newline): take void* as a state pointer. (rb_universal_newline): provide state size/init/fini. (rb_crlf_newline): ditto. (rb_cr_newline): ditto. * enc/trans/iso2022.trans (iso2022jp_init): defined. (fun_si_iso2022jp_to_eucjp): take void* as a state pointer. (fun_so_iso2022jp_to_eucjp): ditto. (fun_so_eucjp_to_iso2022jp): ditto. (iso2022jp_reset_sequence_size): ditto. (finish_eucjp_to_iso2022jp): ditto. (rb_ISO_2022_JP_to_EUC_JP): provide state size/init/fini. (rb_EUC_JP_to_ISO_2022_JP): ditto. * enc/trans/utf_16_32.trans (fun_so_from_utf_16be): take void* as a state pointer. (fun_so_to_utf_16be): ditto. (fun_so_from_utf_16le): ditto. (fun_so_to_utf_16le): ditto. (fun_so_from_utf_32be): ditto. (fun_so_to_utf_32be): ditto. (fun_so_from_utf_32le): ditto. (fun_so_to_utf_32le): ditto. (rb_from_UTF_16BE): provide state size/init/fini. (rb_to_UTF_16BE): ditto. (rb_from_UTF_16LE): ditto. (rb_to_UTF_16LE): ditto. (rb_from_UTF_32BE): ditto. (rb_to_UTF_32BE): ditto. (rb_from_UTF_32LE): ditto. (rb_to_UTF_32LE): ditto. * enc/trans/japanese.trans (fun_so_eucjp2sjis): take void* as a state pointer. (fun_so_sjis2eucjp): ditto. (rb_eucjp2sjis): provide state size/init/fini. (rb_sjis2eucjp): provide state size/init/fini. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19096 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/japanese.trans: new file.akr2008-09-031-0/+96
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: check $(srcdir)/enc/trans beforenaruse2008-09-031-2/+3
| | | | | | | | enc/trans. * enc/trans/make_transdb.rb: keep names_t. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19081 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: check foo.c only if foo.trans exists.akr2008-09-021-2/+15
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19068 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: error message improved.akr2008-09-021-4/+5
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19067 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* revert last commit.akr2008-09-021-5/+4
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19066 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/make_transdb.rb: error message improved.akr2008-09-021-4/+5
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19065 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/utf_16_32.trans (from_UTF_8): rename from to_UTF_16BEusa2008-09-021-15/+15
| | | | | | | | because it was not collect. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19063 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/emacs_mule.c (svn:executable): dropped executable bit.yugui2008-09-022-0/+0
| | | | | | | | * enc/make_encdb.rb (svn:executable): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19062 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/japanese_euc.trans: splitted from japanese.trans to avoidakr2008-09-022-25/+35
| | | | | | | | | | compiler limitation. reported by usa. * enc/trans/japanese_sjis.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19058 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * tool/transcode-tblgen.rb: define TRANSCODE_TABLE_INFO in generatedakr2008-09-013-13/+13
| | | | | | | | | | | | | | code. use it in rb_transcoder. * enc/trans/newline.trans: use TRANSCODE_TABLE_INFO. * enc/trans/iso2022.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * tool/transcode-tblgen.rb: record offsets array as index ofakr2008-09-013-0/+13
| | | | | | | | | | | | | | | | | | | | | byte_array to avoid relocation. * transcode.c (transcode_restartable0): add byte_array to get offsets array. * transcode_data.h (BYTE_LOOKUP_BASE): change return type to uintptr_t. (rb_transcoder): add fields: byte_array, word_array and word_size. * enc/trans/newline.trans: follow rb_transcoder change. * enc/trans/iso2022.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (BYTE_LOOKUP): change to uintptr_t array.akr2008-09-013-22/+22
| | | | | | | | | | | | | | | | | | | | | (BYTE_LOOKUP_BASE): follow the type change. (BYTE_LOOKUP_INFO): ditto. (PType): ditto. (rb_transcoding): ditto. * tool/transcode-tblgen.rb: follow the type change. * transcode.c: ditto. * enc/trans/newline.trans: ditto. * enc/trans/iso2022.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/euc_jp.c (euc-jp-ms): euc-jp-ms is not EUC-JP not an alias ofnaruse2008-09-012-3/+3
| | | | | | | | | | eucJP-ms. * enc/trans/japanese.trans (eucJP-ms): eucJP-ms is the correct name of the encoding in Ruby. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19021 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/japanese.trans: fix mapping priority.naruse2008-09-011-1/+1
| | | | | | | IBM extended is prior than NEC selected IBM. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19019 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/depend: transdb.c may not present.nobu2008-09-011-0/+1
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19015 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/japanese.trans: fix Ruby 1.8 compatibility.naruse2008-09-011-56/+55
| | | | | | | * enc/trans/japanese.trans: fix mapping priority. [ruby-dev:36068] git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19014 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * tool/transcode-tblgen.rb (transcode_generated_code): defined forakr2008-08-316-119/+116
| | | | | | | | | | | | | | | | | | | | | | generating table at once. (transcode_tblgen): returns an empty string. (transcode_generate_node): ditto. * enc/trans/newline.trans: use transcode_generated_code. * enc/trans/iso2022.trans: ditto. * enc/trans/single_byte.trans: ditto. * enc/trans/utf_16_32.trans: ditto. * enc/trans/japanese.trans: ditto. * enc/trans/korean.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19006 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/eucjp-tbl.rb: replace by previous Citrus maps.naruse2008-08-312-29606/+0
| | | | | | | * enc/trans/sjis-tble.rb: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19004 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * tool/transcode-tblgen.rb: add table generator from Citrus maps.naruse2008-08-3125-13/+62273
| | | | | | | | | | | | | | * enc/trans/japanese.trans: use Citrus maps. * enc/trans/CP: add maps from Citrus. * enc/trans/JIS: ditto. * test/ruby/test_transcode.rb: Shift_JIS and EUC-JP doesn't support IBM extended characters. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@19003 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/single_byte.trans (us_ascii_map): don't define 8bit bytes.akr2008-08-311-1/+1
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18982 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/emacs_mule.c: fix ctype.naruse2008-08-241-105/+9
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18827 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * configure.in (BUILTIN_TRANSSRCS): defined.akr2008-08-243-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | (BUILTIN_TRANSOBJS): defined. * enc/Makefile.in (BUILTIN_TRANSES): defined. * enc/make_encmake.rb (BUILTIN_TRANSES): defined. * enc/depend: don't generate rules for builtin transcoders. * common.mk (COMMONOBJS): add BUILTIN_TRANSOBJS. (enc.mk): pass BUILTIN_TRANSOBJS. (newline.c): new rule. (newline.$(OBJEXT)): new ru.e (srcs): newline.c added. * Makefile.in (BUILTIN_TRANSSRCS): defined. (BUILTIN_TRANSOBJS): defined. * transcode.c (Init_transcode): call Init_newline. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18826 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/emacs_mule.c: support Emacs/Mule internal encoding.naruse2008-08-241-0/+436
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18800 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/io.h (FMODE_TEXTMODE): defined.akr2008-08-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * include/ruby/encoding.h (rb_econv_t): new field: flags. (rb_econv_binmode): declared. * io.c (io_unread): text mode hack removed. (NEED_NEWLINE_DECODER): defined. (NEED_NEWLINE_ENCODER): defined. (NEED_READCONV): defined. (NEED_WRITECONV): defined. (TEXTMODE_NEWLINE_ENCODER): defined for windows. (make_writeconv): setup converter with TEXTMODE_NEWLINE_ENCODER for text mode. (io_fwrite): use NEED_WRITECONV. character code conversion is disabled if fptr->writeconv_stateless is nil. (make_readconv): setup converter with ECONV_UNIVERSAL_NEWLINE_DECODER for text mode. (read_all): use NEED_READCONV. (appendline): use NEED_READCONV. (rb_io_getline_1): use NEED_READCONV. (io_getc): use NEED_READCONV. (rb_io_ungetc): use NEED_READCONV. (rb_io_binmode): OS-level text mode test removed. call rb_econv_binmode. (rb_io_binmode_m): call rb_io_binmode_m with write_io as well. (rb_io_flags_mode): return mode string including "t". (rb_io_mode_flags): detect "t" for text mode. (rb_sysopen): always specify O_BINARY. * transcode.c (rb_econv_open_by_transcoder_entries): initialize flags. (rb_econv_open): if source and destination encoding is both empty string, open newline converter. last_tc will be NULL in this case. (rb_econv_encoding_to_insert_output): last_tc may be NULL now. (rb_econv_string): ditto. (output_replacement_character): ditto. (transcode_loop): ditto. (econv_init): ditto. (econv_inspect): ditto. (rb_econv_binmode): new function. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (rb_econv_t): add fields: in_buf_start,akr2008-08-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in_data_start, in_data_end, in_buf_end and last_trans_index. (rb_econv_output): removed. (rb_econv_insert_output): declared. (rb_econv_encoding_to_insert_output): declared. * enc/trans/newline.trans (rb_universal_newline): stateful_type changed. * transcode.c (transcode_restartable0): initialize inchar_start, tc->recognized_len and next_table at beginning of the loop. (rb_econv_open_by_transcoder_entries): initialize new fields. (rb_econv_open): setup last_trans_index. (trans_sweep): last out_buf_start can be non-NULL now. (rb_econv_convert): check last out_buf_start and in_buf_start at first. (rb_econv_output_with_destination_encoding): removed. (econv_just_convert): removed. (rb_econv_output): removed. (econv_primitive_output): method removed. (rb_econv_encoding_to_insert_output): new function. (allocate_converted_string): new function. (rb_econv_insert_output): new function. (econv_primitive_insert_output): new method. (output_replacement_character): use rb_econv_insert_output. unused arguments removed. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18654 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder_stateful_type_t): defined.akr2008-08-153-0/+13
| | | | | | | | | | | | | | | | | (rb_transcoder): add field: stateful_type. * tool/transcode-tblgen.rb: generate stateful_type field as stateless_converter. * enc/trans/iso2022.trans: follow rb_transcoder change. * enc/trans/newline.trans: ditto. * enc/trans/utf_16_32.trans: ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18650 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * include/ruby/encoding.h (rb_econv_output): declared.akr2008-08-141-4/+13
| | | | | | | | | | | | | | | | | | * transcode_data.h (rb_transcoder): add resetsize_func field. * enc/trans/iso2022.trans (iso2022jp_reset_sequence_size): defined. (rb_EUC_JP_to_ISO_2022_JP): provede resetsize_func. * tool/transcode-tblgen.rb: set NULL for resetsize_func. * transcode.c (rb_econv_output): new function for inserting output. (output_replacement_character): use rb_econv_output. (transcode_loop): check return value of output_replacement_character. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18628 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit (git 2.34.1) at 2024-11-12 05:08:45 +0000