summaryrefslogtreecommitdiffstats
path: root/test/ruby/test_transcode.rb
Commit message (Collapse)AuthorAgeFilesLines
* * test/ruby/test_transcode.rb: test_shift_jis:duerst2008-08-221-21/+21
| | | | | | | | fixed comment strings (see r18291) git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18772 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (TRANSCODE_ERROR): removed.akr2008-08-121-30/+29
| | | | | | | | | | | * tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid (but unique to ASCII-8BIT) character. * transcode.c (rb_eConversionUndefined): new error. (rb_eInvalidByteSequence): ditto. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add a test.akr2008-08-111-0/+11
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18504 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder): add resetstate_func field forakr2008-08-111-0/+3
| | | | | | | | | | | | | | | | resetting a state of stateful encoding. * enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify finish_eucjp_to_iso2022jp for resetstate_func. * tool/transcode-tblgen.rb: specify NULL for resetstate_func. * transcode.c (output_replacement_character): call resetstate_func before appending the replacement character. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (rb_trans_conv): find second last error.akr2008-08-111-0/+5
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_trans_result_t): new type.akr2008-08-111-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | (rb_trans_elem_t): new type. (rb_trans_t): new type. * transcode.c (transcode_dispatch_cb): removed. (transcode_dispatch): removed. (rb_transcoding_result_t): moved to rb_trans_result_t in transcode_data.h. (transcode_restartable0): goto follow_info when FUNsi. (rb_transcoding_open): use get_transcoder_entry. (rb_trans_open): new function. (rb_trans_conv): ditto. (rb_trans_close): ditto. (trans_open_i): ditto. (trans_sweep): ditto. (more_output_buffer): take rb_trans_t instead of rb_transcoding as an argument. (transcode_loop): take from_encoding and to_encoding instead of tr as arguments. use rb_trans_open/rb_trans_conv/rb_trans_close. (str_transcode): don't use transcode_dispatch. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add several tests for UTF-32LE.akr2008-08-081-0/+7
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoder): from_unit_length field added.akr2008-08-081-1/+23
| | | | | | | | | | | | | | | | | | | | from_utf8 field removed. * tool/transcode-tblgen.rb: generate offsets range. follow rb_transcoder change. * transcode.c (transcode_loop): don't use from_utf8. make invalid region from_unit_length wise. * enc/trans/iso2022.erb.c: follow rb_transcoder and transcode_generate_node change. * enc/trans/utf_16_32.erb.c: follow rb_transcoder and transcode_generate_node change. explicit :invalid map removed. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode_data.h (rb_transcoding): new field "stateful".akr2008-08-071-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. nute(23:52:53)% head -40 ChangeLog Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org> * transcode_data.h (rb_transcoding): new field "stateful". (rb_transcoder): preprocessor and postprocessor field removed. change arguments of func_ii, func_si, func_io and func_so. new field "finish_func". * tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio generatable. * transcode.c (transcoder_lib_table): removed. (transcoder_table): change structure. (transcoder_key): removed because the above structure change. (make_transcoder_entry): new function. (get_transcoder_entry): ditto. (rb_register_transcoder): follow the structure change. (declare_transcoder): ditto. (transcode_search_path): new function for breadth first search to find a list of converters. (transcode_search_path_i): new function. (transcode_dispatch_cb): ditto. (transcode_dispatch): use transcode_search_path. (transcode_loop): follow the argument change. (str_transcode): preprocessor and postprocessor stuff removed. * enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion re-implemented. * enc/trans/japanese.erb.c: ISO-2022-JP stuff removed. * enc/trans/utf_16_32.erb.c: follow argument change of FUNso. [ruby-dev:35798] git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add tests for [ruby-dev:35726] and [ruby-dev:35709].akr2008-08-061-0/+6
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (output_replacement_character):naruse2008-07-311-0/+15
| | | | | | | | | | | | | | rename from _get_replacement_character. * transcode.c (output_replacement_character): fix replacement on UTF-32{BE,LE}. [ruby-dev:35705] * transcode.c (transcode_loop): ditto. * test/ruby/test_transcode.rb (test_invalid_replace): add for above. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121):naruse2008-07-311-2/+2
| | | | | | fix option1 and 3. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (get_replacement_character): use U+FFFD as replacementnaruse2008-07-311-7/+4
| | | | | | | | | | | | character when convert to Unicode. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): rename from test_public_review_issue_121. * test/ruby/test_transcode.rb (test_unicode_public_review_issue_121): enable option2. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18294 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/ruby/test_transcode.rb: added test_shift_jisduerst2008-07-311-0/+56
| | | | | | | | | (contributed by Yoshihiro Kambayashi) and test_public_review_issue_121 (see http://www.unicode.org/review/pr-121.html) git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18291 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * test/ruby/test_transcode.rb: refactoring/cleanup ofduerst2008-07-251-7/+8
| | | | | | | | test_iso_2022_jp(_1) git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18203 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * test/ruby/test_transcode.rb: added two commentsduerst2008-07-221-2/+2
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@18160 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * test/ruby/test_transcode.rb: add tests for iso-2022-jp.mame2008-06-041-0/+15
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@16821 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Sun Mar 16 18:07:07 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2008-03-161-1/+5
| | | | | | | | | | | * enc/trans/utf_16_32.c: bug fix (some invalid UTF-8 sequences were legal) * test/ruby/test_transcode.rb: test for above bug git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15786 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Web Mar 5 17:43:43 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2008-03-051-1/+6
| | | | | | | | | | | | * transcode.c (transcode_loop): Adjusted detection of invalid (ill-formed) UTF-8 sequences. Fixing potential security issue, see http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes. * test/ruby/test_transcode.rb: Added two tests for above fix. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15692 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Thu Feb 21 17:15:15 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2008-02-211-2/+5
| | | | | | | | | | | | | | | * transcode.c: Added basic support for passing options to String#encode via a hash. Currently only one option, with one value, is supported: invalid: :ignore (dropping invalid byte sequences instead of producing an error). Option naming is not yet stable! * test/ruby/test_transcode.rb: Added a single test for invalid: :ignore option. Not more tests because most data does not yet distinguish between INVALID and UNKNOWN. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15565 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * enc/trans/korean.c: add support for CP949 by Park Ji-In. [ruby-dev:33626]naruse2008-02-071-1/+5
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15393 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Mon Jan 21 19:42:42 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2008-01-211-0/+59
| | | | | | | | | * transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb: added UTF-32BE and UTF-32LE conversions. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15156 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Sun Jan 20 20:00:20 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2008-01-201-45/+50
| | | | | | | | | | | * transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb: added UTF-16LE conversions. * fixed changelog for last commit git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15144 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Sun Jan 20 15:08:08 2008 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2008-01-201-3/+52
| | | | | | | | | | | | | | | * enc/trans/utf_16_32.c: new file, currently implementing UTF-16BE conversions only. * test/ruby/test_transcode.rb: Added tests for UTF-16BE; made check_both_ways() use force_encoding differently. * transcode_data.h, transcode.c: Support for more conversion functions. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@15142 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Fri Dec 28 01:55:04 2007 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2007-12-281-0/+3
| | | | | | | | | | | | | | | | | | | | | * transcode.c (transcode_dispatch): reverted some of the changes in r14746. * transcode.c, enc/trans/single_byte.c: Added conversions to/from US-ASCII and ASCII-8BIT (using data tables). * enc/trans/single_byte.c: Some spacing/ordering changes due to automatic data file generation. * transcode_data.h, transcode.c: Preliminary code for using micro-conversion functions. * test/ruby/test_transcode.rb: Added some tests for US-ASCII and ASCII-8BIT conversions. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14766 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Sat Dec 22 15:54:54 2007 Martin Duerst <duerst@it.aoyama.ac.jp>duerst2007-12-221-32/+46
| | | | | | | | | * test/ruby/test_transcode.rb: Added simple tests for EUC-JP and Shift_JIS and tests for ASCII-only range git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14486 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* for undefined conversions.matz2007-12-171-0/+1
| | | | | | | | | | | | * transcode_data_iso_8859.c: Changed from character constants ('\xC2') to integer contants (0xC2) for shorter files and better readability; eliminated duplicated tables; changed from -1 offset to actual UNDEF entry (not yet distinguishing UNDEF and ILLEGAL correctly). * test/ruby/test_transcode.rb: added a test for UNDEF conversion. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14251 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c (str_transcode, transcode_dispatch): added two-stepmatz2007-12-151-35/+69
| | | | | | | | | | | | | | * trancode.c: some minor formatting fixes * transcode_data.h, transcode_data_iso_8859.c: Shortened extremely frequently used macros to shorten file length. * test/ruby/test_transcode.rb: Fixed name of test class; added setup method to ensure all necessary encodings exist; split tests into more test methods; added tests; fixed ordering of arguments in assert_equal to have expected result first. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14236 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * test/ruby/test_transcode.rb: added tests from Martin Duerst <duerstnobu2007-12-111-28/+59
| | | | | | | AT it.aoyama.ac.jp>. [ruby-dev:32532] git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14192 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode*.[ch], test/ruby/test_transcode.rb: set properties.nobu2007-12-101-44/+44
| | | | git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14175 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* * transcode.c: new file to provide encoding conversion features.matz2007-12-101-0/+44
code contributed by Martin Duerst. git-svn-id: http://svn.ruby-lang.org/repos/ruby/trunk@14172 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit (git 2.34.1) at 2026-01-05 19:07:45 +0000