Issue 81366

Summary: New transliteration modules for Japanese
Product: Internationalization Reporter: bluedwarf <bluedwarf>
Component: i18npoolAssignee: stefan.baltzer
Status: CLOSED FIXED QA Contact: issues@l10n <issues>
Severity: Trivial    
Priority: P3 CC: issues
Version: 680m223   
Target Milestone: ---   
Hardware: All   
OS: All   
Issue Type: PATCH Latest Confirmation in: ---
Developer Difficulty: ---
Issue Depends on:    
Issue Blocks: 80766, 81383    
Attachments:
Description Flags
Proposed patch for i18npool module
none
Proposed patch for i18nutil module
none
Test cases for new transliteration modules. none

Description bluedwarf 2007-09-07 14:30:48 UTC
I created new transliteration modules for mainly Japanese.

They are
 - FULLWIDTHKATAKANA_HALFWIDTHKATAKANA
   transliteration from fullwidth katakana characters to halfwidth katakana
characters
 - HALFWIDTHKATAKANA_FULLWIDTHKATAKANA
   transliteration from halfwidth katakana characters to fullwidth katakana
characters
 - FULLWIDTH_HALFWIDTH_LIKE_ASC
   transliteration from fullwidth characters to halfwidth characters like ASC
function
 - HALFWIDTH_FULLWIDTH_LIKE_JIS
   transliteration from halfwidth characters to fullwidth characters like JIS
function

The first two transliteration modules will be used for PHONETIC function. (i80766)
The last two transliteration modules will be used for ASC and JIS function.
(i78727 => need to open a new issue)

These patches also include some fixes on decomposition and composition related
to (semi-)voiced sound mark. These fixes replace COMBINING KATAKANA-HIRAGANA
(SEMI-)VOICED SOUND MARK with KATAKANA-HIRAGANA (SEMI-)VOICED SOUND MARK.
COMBINING characters are not supported in most cases. By default, OOo 2.2.1 on
Windows and Debian etch shows undesirable white squares instead of (semi-)voiced
sound mark.

Note that this changes the operation of FULLWIDTH_HALFWIDTH and
HALFWIDTH_FULLWIDTH transliteration modules.
Comment 1 bluedwarf 2007-09-07 14:35:21 UTC
Created attachment 48050 [details]
Proposed patch for i18npool module
Comment 2 bluedwarf 2007-09-07 14:35:58 UTC
Created attachment 48051 [details]
Proposed patch for i18nutil module
Comment 3 bluedwarf 2007-09-07 14:43:00 UTC
I'm sorry but the proposed patches were created from my personal CVS repository.
The original version is SRC680_m223.
Comment 4 bluedwarf 2007-09-07 14:55:04 UTC
Created attachment 48052 [details]
Test cases for new transliteration modules.
Comment 5 bluedwarf 2007-09-07 15:09:42 UTC
The attached Calc file is the test case document. It has 2 sheets and macros.

You can verify that the modification has compatibility of HALFWIDTH_FULLWIDTH
and FULLWIDTH_HALFWIDTH modules with version 2.2.1. You can also verify that the
performance of new FULLWIDTHKATAKANA_HALFWIDTHKATAKANA and
HALFWIDTHKATAKANA_FULLWIDTHKATAKANA modules is up to your expectations.

As to FULLWIDTH_HALFWIDTH_LIKE_ASC and HALFWIDTH_FULLWIDTH_LIKE_JIS, I will open
a new issue with another patch, which changes internal implementation of ASC and
JIS function to use the two new transliteration modules, in a few days. Please
wait the patch and you are going to be able to verify that these modules work
correctly with the test case attached to i78727.
Comment 6 pavel 2007-09-07 19:19:24 UTC
Eike: is it yours? Or Karl's?
Comment 7 bluedwarf 2007-09-10 11:51:07 UTC
Additional comments.

The main purpose of these patches is, needless to say, to provide various
transliteration modules for users and developers. Not only that, this patch
provides an OneToOne mapping table shared by some similar transliteration
modules so that it can cuts off memory usage for various mapping tables.

However, this modification still has some minor problems below:
 - many redundant lines in i18npool (I will fix this problem soon by C macro.)
 - requires systematic management of exceptional conversion (see lines labeled
with "dirty hack!" in my i18nopol patch)
 - each instance holds their own index for fast search
Comment 8 ooo 2007-10-09 12:32:39 UTC
I was on vacation the last weeks, hence the long silence.

As Karl is familiar with implementation of the transliteration mappings he might
also want to take a look at the patch. To me it looks viable.

@khong: I set the target to "not determined"; if there are no problems we could
as well add this to OOo2.4, please retarget then and add this issue to one of
your next CWSs.

Thanks
  Eike
Comment 9 karl.hong 2007-12-12 22:59:01 UTC
Apply patch in cws i18n39.
Comment 10 karl.hong 2007-12-13 07:59:48 UTC
ready for QA.
Comment 11 stefan.baltzer 2008-01-17 15:12:31 UTC
Verified in CWS i18n39
Comment 12 bluedwarf 2008-11-24 20:14:29 UTC
I found this change in the latest product release.
Closing.