[RFC] iconvdata/, localedata/: Fix TSCII and document tests.

Message ID 103199fc-7c6c-93ce-4702-2689b06b67b7@redhat.com
State New
Headers show
Series
  • [RFC] iconvdata/, localedata/: Fix TSCII and document tests.
Related show

Commit Message

Adhemerval Zanella via Libc-alpha June 21, 2020, 2:11 p.m.
This is a general cleanup of the testing infrastructure around the
iconv tests. I'm posting it now to get any feedback about general
direction with the cleanup. Should I pass arguments to the script
instead of environment variables?

I have one test failure in test-tables which I need to fix which
has to do with the regexps there.

8< --- 8< --- 8<
The TSCII charmap was completely broken and is fixed by the removal
of the many-to-one mappings, not needed WIDTH specification,
invalid identical encodings, and wrong mb_cur_max.  The %IRREVERSIBLE%
markup is documented, and that documentation is extended to all
locales that have such markup.

TSCII was not explicitly being excluded from testing, but it should
have been, and this led me to cleanup run-iconv-test.sh for TSCII.

The iconvdata/run-iconv-test.sh had two implicit inputs, TESTS and
TESTS2, the latter which was not included in the dependencies, along
with iconv-test.yyy which was not included in generated.  We rename
the tests to iconv-test1.in and iconv-test2.in because that makes
it clearer they are a part of the iconv-test special test, and we
pass the test data names via env vars.  We add iconv-test2.in to the
dependencies, and iconv-test.yyy to generated to ensure the test is
rebuilt if the test data changes and that iconv-test.yyy is removed.
Further to this we extensively comment run-iconv-test.sh and explain
what each step of the test is doing.  We remove the cryptic grep for
excluding charmaps that have data that cannot be handled by iconv
with charmaps as input and we explicitly name those charmaps which
cannot be tested in "Test 1c: Convert using charmaps." and why.
We incidentally fix the BIG5-HKSCS <-> BIG5HKSCS alias problem even
if we don't test it, we might in the future, and not it will work.

In reviewing iconvdata/tst-table-charmap.sh it was shown that the
%IRREVERSIBLE% markup in the charmaps is used both by the table
generation for testing *and* the iconv table generation.  Thus as
part of the additional documentation effort we add comments to all
of the %IRREVERSIBLE% markup to note where it is used for testing
or where it is used for iconv table generation.

For BIG5-HKSCS we add additional comments for HKSCS-2016, which was
quite helpful during review.
---
 iconvdata/Makefile                   |  10 +-
 iconvdata/{TESTS => iconv-test1.in}  |   0
 iconvdata/{TESTS2 => iconv-test2.in} |   0
 iconvdata/run-iconv-test.sh          | 123 +++++++-
 localedata/charmaps/BIG5             |   7 +
 localedata/charmaps/BIG5-HKSCS       |  32 +++
 localedata/charmaps/EUC-JP-MS        |   7 +
 localedata/charmaps/EUC-TW           |   1 +
 localedata/charmaps/IBM1132          |   1 +
 localedata/charmaps/IBM1133          |   1 +
 localedata/charmaps/IBM1160          |   4 +
 localedata/charmaps/IBM1161          |   2 +
 localedata/charmaps/TSCII            | 409 ++++++++++++++-------------
 localedata/charmaps/WINDOWS-31J      |   7 +-
 14 files changed, 394 insertions(+), 210 deletions(-)
 rename iconvdata/{TESTS => iconv-test1.in} (100%)
 rename iconvdata/{TESTS2 => iconv-test2.in} (100%)

-- 
2.26.2

Comments

Adhemerval Zanella via Libc-alpha June 22, 2020, 3:41 p.m. | #1
* Carlos O'Donell via Libc-alpha:

> diff --git a/localedata/charmaps/TSCII b/localedata/charmaps/TSCII

> index 9646f326cb..3d9ae1fb5e 100644

> --- a/localedata/charmaps/TSCII

> +++ b/localedata/charmaps/TSCII

> @@ -2,8 +2,26 @@

>  <comment_char> %

>  <escape_char> /

>  <mb_cur_min> 1

> -<mb_cur_max> 1

> -% based on TSCII version 1.7

> +<mb_cur_max> 3

> +

> +% Tamil Script Code for Information Interchange

> +%

> +% Based on TSCII version 1.7

> +%

> +% The lower 128 code points are ASCII, but the upper code points are

> +% TSCII characters that often map to multiple Unicode code points.  The

> +% one-to-many mapping means that much of the character map is commented

> +% out since we don't support many-to-one mappings in POSIX-compatible

> +% character maps.  There are 179 such mappings where one encoded TSCII

> +% character is mapped to more than one Unicode code point.

> +%

> +% Note that iconv is capable of and supports such conversions, but iconv

> +% when run with character maps as from-encoding or to-encoding is unable

> +% to support such conversions.

> +%

> +% For conversion reference:

> +% https://www.unicode.org/notes/tn15/Tscii2Unicode2.pdf


Does this mean that after this change, glibc will no longer perform
proper multi-byte to wide string conversion for single-byte characters
such as 0x8c?  Or is the charmap file just for reference purposes, and
conversion of 0x8c to U+0B95 U+0BCD U+0BB7 U+0BCD works as before?

Thanks,
Florian
Andreas Schwab June 22, 2020, 3:49 p.m. | #2
On Jun 21 2020, Carlos O'Donell via Libc-alpha wrote:

> +% not include them. Note that iconv is capable of and supports such

> +% conversions, but iconv when run with character maps as from-encoding or

> +% to-encoding is unable to support such conversions.

> +%

> +% Note that iconv is capable of and supports such conversions, but iconv

> +% when run with character maps as from-encoding or to-encoding is unable

> +% to support such conversions.


s/(.*)\1/\1/

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Adhemerval Zanella via Libc-alpha June 22, 2020, 5:10 p.m. | #3
On 6/22/20 11:41 AM, Florian Weimer wrote:
> * Carlos O'Donell via Libc-alpha:

> 

>> diff --git a/localedata/charmaps/TSCII b/localedata/charmaps/TSCII

>> index 9646f326cb..3d9ae1fb5e 100644

>> --- a/localedata/charmaps/TSCII

>> +++ b/localedata/charmaps/TSCII

>> @@ -2,8 +2,26 @@

>>  <comment_char> %

>>  <escape_char> /

>>  <mb_cur_min> 1

>> -<mb_cur_max> 1

>> -% based on TSCII version 1.7

>> +<mb_cur_max> 3

>> +

>> +% Tamil Script Code for Information Interchange

>> +%

>> +% Based on TSCII version 1.7

>> +%

>> +% The lower 128 code points are ASCII, but the upper code points are

>> +% TSCII characters that often map to multiple Unicode code points.  The

>> +% one-to-many mapping means that much of the character map is commented

>> +% out since we don't support many-to-one mappings in POSIX-compatible

>> +% character maps.  There are 179 such mappings where one encoded TSCII

>> +% character is mapped to more than one Unicode code point.

>> +%

>> +% Note that iconv is capable of and supports such conversions, but iconv

>> +% when run with character maps as from-encoding or to-encoding is unable

>> +% to support such conversions.

>> +%

>> +% For conversion reference:

>> +% https://www.unicode.org/notes/tn15/Tscii2Unicode2.pdf

> 

> Does this mean that after this change, glibc will no longer perform

> proper multi-byte to wide string conversion for single-byte characters

> such as 0x8c?  Or is the charmap file just for reference purposes, and

> conversion of 0x8c to U+0B95 U+0BCD U+0BB7 U+0BCD works as before?


We do not support what is written in the TSCII character map today,
and the character map is completely broken and will not compile.

The parser cannot handle multiple internal code units mappings to the
encoded multi-byte stream. There is no such support in glibc. It seems
like TSCII's charmap was committed blind with no real testing, and only
provides input for reference to the iconv converter.

For example the following is invalid and the parser won't parse it:
<U0B95><U0BCD><U0BB7><U0BCD> /x8c         TAMIL GLYPH KSH

I don't know why it was ever committed in the first place. Probably
just to record the fact that this exists.

The current regexp in the test will exclude TSCII from testing because
of this line, and that means it has bit rotted and is useless.

My goal is to make TSCII actually a usable charmap if you needed it,
and have the testing include a few more comments to help future readers.

Conversion of 0x8c is handled by special case in iconv and is correct.

e.g.
iconvdata/tscii.c:
476         else if (ch == 0x8c)                                                  \
477           {                                                                   \
478             /* Output <U0B95><U0BCD><U0BB7><U0BCD>, if we have room for       \
479                four characters.  */                                           \

In summary:
- TSCII is completely broken today, and this patch makes it usable.
- TSCII can now be used in a locale, but is missing a lot of entries
  because of the POSIX charmap limitations.
- Conversions for TSCII are correctly handled only by iconv.

Does that answer your question?

-- 
Cheers,
Carlos.
Adhemerval Zanella via Libc-alpha June 22, 2020, 5:14 p.m. | #4
On 6/22/20 11:49 AM, Andreas Schwab wrote:
> On Jun 21 2020, Carlos O'Donell via Libc-alpha wrote:

> 

>> +% not include them. Note that iconv is capable of and supports such

>> +% conversions, but iconv when run with character maps as from-encoding or

>> +% to-encoding is unable to support such conversions.

>> +%

>> +% Note that iconv is capable of and supports such conversions, but iconv

>> +% when run with character maps as from-encoding or to-encoding is unable

>> +% to support such conversions.

> 

> s/(.*)\1/\1/


Thanks will remove the duplicate.

-- 
Cheers,
Carlos.
Adhemerval Zanella via Libc-alpha June 24, 2020, 2:24 p.m. | #5
* Carlos O'Donell:

> We do not support what is written in the TSCII character map today,

> and the character map is completely broken and will not compile.


But will a compiled TSCII locale still work, even if the charmap file is
not parsed correctly by localdef?  As long as the locale definition does
reference any of the problematic characters?

Thanks,
Florian
Adhemerval Zanella via Libc-alpha June 24, 2020, 5:01 p.m. | #6
On 6/24/20 10:24 AM, Florian Weimer wrote:
> * Carlos O'Donell:

> 

>> We do not support what is written in the TSCII character map today,

>> and the character map is completely broken and will not compile.

> 

> But will a compiled TSCII locale still work, even if the charmap file is

> not parsed correctly by localdef?  As long as the locale definition does

> reference any of the problematic characters?


Yes, a TSCII locale, say C.TSCII will still work, but you get errors
(though those errors don't actually get propagated because we use error_at_line
for linereader errors instead of record_error_at_line which propagates these
errors, and that's a bug in linereader.h lr_error()).

Such a locale compilation generates the following warnings:

$ localedef -f TSCII -i ./localedata/locales/C /home/carlos/tmp/C.TSCII
/usr/share/i18n/charmaps/TSCII:139: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:141: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:142: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:144: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:145: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:148: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:149: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:150: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:151: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:152: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:153: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:154: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:155: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:156: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:157: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:170: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:171: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:172: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:173: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:183: too many bytes in character encoding
/usr/share/i18n/charmaps/TSCII:184: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:185: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:186: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:187: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:188: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:189: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:190: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:191: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:192: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:193: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:194: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:195: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:196: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:197: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:198: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:199: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:200: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:201: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:202: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:203: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:204: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:205: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:206: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:207: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:208: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:209: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:210: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:211: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:212: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:213: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:214: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:215: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:216: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:217: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:218: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:219: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:221: too many bytes in character encoding
/usr/share/i18n/charmaps/TSCII:222: too many bytes in character encoding
/usr/share/i18n/charmaps/TSCII:223: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:224: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:225: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:226: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:227: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:228: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:229: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:230: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:231: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:232: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:233: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:234: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:235: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:236: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:237: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:238: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:239: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:240: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:241: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:242: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:243: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:244: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:245: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:246: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:247: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:248: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:249: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:250: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:251: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:252: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:253: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:254: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:255: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:256: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:257: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:258: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:259: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:260: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:261: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:262: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:263: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:264: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:265: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:266: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:267: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:268: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:269: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:270: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:271: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:272: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:273: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:274: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:275: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:276: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:278: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:279: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:280: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:281: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:282: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:283: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:284: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:285: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:286: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:287: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:288: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:289: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:290: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:291: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:292: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:293: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:294: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:295: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:329: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:330: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:331: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:332: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:333: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:334: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:335: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:336: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:337: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:338: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:339: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:340: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:341: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:342: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:343: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:344: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:345: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:346: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:347: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:348: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:349: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:350: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:351: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:352: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:353: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:354: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:355: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:356: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:357: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:358: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:359: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:360: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:361: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:362: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:363: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:364: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:365: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:366: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:367: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:368: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:369: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:370: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:371: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:372: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:373: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:374: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:375: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:376: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:377: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:378: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:379: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:380: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:385: unknown character `U00000B82'
/usr/share/i18n/charmaps/TSCII:387: unknown character `U00000BCD'

My patch fixes 179 invalid entries.

Above you can see 184 errors, 2 of which are from invalid WIDTH specifiers (the two at the end),
and 3 are from invalid encodings (mb_cur_max is only 1, and should be 3), leaving 179 invalid entries
all of which are now fixed.

The charmap definition is erroneous and those entries are being ignored, but since the C locale
doesn't use any of those entries it is overall "safe" from that perspective.

Note: The unknown character come from WIDTH entries for characters not defined in the charmap.

Note: There *should* be "too many bytes in character encoding" errors for many more entries, but
      the charmap.c processing is restricted to those entries that successfully parsed (don't
      have multiple 'from' code points which are invalid).

With the fixed TSCII we get:
$ localedef -f /home/carlos/src/glibc/localedata/charmaps/TSCII -i ./localedata/locales/C /home/carlos/tmp/C.TSCII
$ echo $?
0

We've fixed all the lines with invalid format and instead turned them into comments regarding
what the charmap *should* have.

Compiling C.TSCII before and after the TSCII changes yields identical binary locales with the
exception of LC_CTYPE which is likely different because maybe we pikcup an extra entry via
i18n_ctype's extended list of LC_CTYPE information. I haven't reviewed the exact details.

Does that answer your question?

-- 
Cheers,
Carlos.

Patch

diff --git a/iconvdata/Makefile b/iconvdata/Makefile
index 4ec2741cdc..6a02f227e0 100644
--- a/iconvdata/Makefile
+++ b/iconvdata/Makefile
@@ -178,8 +178,9 @@  generated-modules := $(gen-8bit-modules) $(gen-8bit-gap-modules) \
 		     $(gen-special-modules)
 
 generated += $(generated-modules:=.h) $(generated-modules:=.stmp) \
-	     iconv-test.out iconv-rules tst-loading.mtrace	 \
-	     mtrace-tst-loading.out tst-tables.out iconv-test.xxx
+	     iconv-test.out iconv-rules tst-loading.mtrace	  \
+	     mtrace-tst-loading.out tst-tables.out iconv-test.xxx \
+	     iconv-test.yyy
 ifdef objpfx
 generated += gconv-modules
 endif
@@ -324,8 +325,11 @@  $(objpfx)bug-iconv12.out: $(objpfx)gconv-modules \
 
 $(objpfx)iconv-test.out: run-iconv-test.sh $(objpfx)gconv-modules \
 			 $(addprefix $(objpfx),$(modules.so)) \
-			 $(common-objdir)/iconv/iconv_prog TESTS
+			 $(common-objdir)/iconv/iconv_prog iconv-test1.in \
+			 iconv-test2.in
 	iconv_modules="$(modules)" \
+	iconv_test1="iconv-test1.in" \
+	iconv_test2="iconv-test2.in" \
 	$(SHELL) $< $(common-objdir) '$(test-wrapper-env)' \
 		 '$(run-program-env)' > $@; \
 	$(evaluate-test)
diff --git a/iconvdata/TESTS b/iconvdata/iconv-test1.in
similarity index 100%
rename from iconvdata/TESTS
rename to iconvdata/iconv-test1.in
diff --git a/iconvdata/TESTS2 b/iconvdata/iconv-test2.in
similarity index 100%
rename from iconvdata/TESTS2
rename to iconvdata/iconv-test2.in
diff --git a/iconvdata/run-iconv-test.sh b/iconvdata/run-iconv-test.sh
index 56b6630a6d..e337146af7 100755
--- a/iconvdata/run-iconv-test.sh
+++ b/iconvdata/run-iconv-test.sh
@@ -23,6 +23,10 @@  set -e
 codir=$1
 test_wrapper_env="$2"
 run_program_env="$3"
+# Additionally we get as input the following environment variables:
+# iconv_modules: The list of modules to test.
+# iconv_test1: The input data for the first test.
+# iconv_test2: The input data for the second test.
 
 # We use always the same temporary file.
 temp1=$codir/iconvdata/iconv-test.xxx
@@ -45,9 +49,10 @@  else
   ac_n= ac_c='\c' ac_t=
 fi
 
-# We read the file named TESTS.  All non-empty lines not starting with
+# We read the file named ICONV_TEST1.  All non-empty lines not starting with
 # `#' are interpreted as commands.
 failed=0
+echo "Reading $iconv_tests and running tests:"
 while read from to subset targets; do
   # Ignore empty and comment lines.
   if test -z "$subset" || test "$from" = '#'; then continue; fi
@@ -57,91 +62,179 @@  while read from to subset targets; do
 
   if test -n "$targets"; then
     for t in $targets; do
+      # Test 1a: Convert data using iconv internally.
       if test -f testdata/$from; then
 	echo $ac_n "   test data: $from -> $t $ac_c"
+
+	# Convert the test data using iconv DSOs.
 	$PROG -f $from -t $t testdata/$from < /dev/null > $temp1 ||
 	  { if test $? -gt 128; then exit 1; fi
 	    echo "FAILED"; failed=1; continue; }
+
+	# Conversion succeeded.
 	echo $ac_n "OK$ac_c"
+
+	# Compare converted data to expected data.
 	if test -s testdata/$from..$t; then
 	  LC_ALL=C cmp $temp1 testdata/$from..$t > /dev/null 2>&1 ||
 	    { echo "/FAILED"; failed=1; continue; }
 	  echo $ac_n "/OK$ac_c"
 	fi
+
+	# Conversion *and* comparison succeeded.
 	echo $ac_n " -> $from $ac_c"
+
+	# Convert in the opposite direction.
 	$PROG -f $t -t $to -o $temp2 $temp1 < /dev/null ||
 	  { if test $? -gt 128; then exit 1; fi
 	    echo "FAILED"; failed=1; continue; }
+
+	# Conversion succeeded.
 	echo $ac_n "OK$ac_c"
+
+	# Compare converted data to expected data.
 	test -s $temp1 &&
 	LC_ALL=C cmp testdata/$from $temp2 > /dev/null 2>&1 ||
 	  { echo "/FAILED"; failed=1; continue; }
+
+	# Conversion *and* comparison succeeded in both directions.
 	echo "/OK"
+	# Cleanup test.
 	rm -f $temp1 $temp2
+
       fi
 
-      # Now test some bigger text, entirely in ASCII.  If ASCII is no subset
+      # Test 1b: Convert "The Art of War" (larger text) [ASCII subset]
+      #
+      # Now test some bigger text, entirely in ASCII.  If ASCII is not a subset
       # of the coded character set we convert the text to this coded character
       # set.  Otherwise we convert to all the TARGETS.
       if test $subset = Y; then
 	echo $ac_n "      suntzu: $from -> $t -> $to $ac_c"
+
+	# Convert test data to target and back.
 	$PROG -f $from -t $t testdata/suntzus < /dev/null |
 	$PROG -f $t -t $to > $temp1 ||
 	  { if test $? -gt 128; then exit 1; fi
 	    echo "FAILED"; failed=1; continue; }
+
+	# Conversion succeeded.
 	echo $ac_n "OK$ac_c"
+
+	# Compare against expected data.
 	LC_ALL=C cmp testdata/suntzus $temp1 ||
 	  { echo "/FAILED"; failed=1; continue; }
+
+	# Conversion *and* comparison succeeded.
 	echo "/OK"
+	# Cleanup test.
+	rm -f $temp1
       fi
-      rm -f $temp1
-
-      # And tests where iconv(1) has to handle charmaps.
-      if test "$t" = UTF8; then tc=UTF-8; else tc="$t"; fi
-      if test -f ../localedata/charmaps/$from &&
-         test -f ../localedata/charmaps/$tc &&
-	 test -f testdata/$from &&
-	 ! grep '<U....><U....>' ../localedata/charmaps/$from > /dev/null; then
+
+      # Test 1c: Convert using charmaps.
+      #
+      # And tests where iconv(1) has to handle charmaps:
+      # We exclude charmaps that have many-to-one mappings, which today is
+      # TSCII, SHIFT_SJISX0213, BIG5HKSCS, and EUC-JISX0213.  We exclude them
+      # because such mappings make the there-and-back conversion tests
+      # difficult.  All of these charmaps have commented out entries or
+      # valid entries for their many-to-one conversions e.g.
+      # '% <U0B9C><U0BC1> /x83/xa4 TAMIL GLYPH JU' (TSCII)
+      # or '% <U304B><U309A> /x82/xf5' (SHIFT_JISX0213).
+      # We could test them but it would require splitting the data files out
+      # into those that can be handled by charmap conversions versus those
+      # that can only be handled by iconv conversion (the larger set).
+
+      # Some charsets need to be converted to special charmap names.
+      tc="$t";
+      fromc="$from";
+      if test "$t" = UTF8; then tc=UTF-8; fi
+      if test "$from" = BIG5HKSCS; then fromc=BIG5-HKSCS; fi
+
+      # Run the charmap tests:
+      if test -f ../localedata/charmaps/$fromc \
+	 && test -f ../localedata/charmaps/$tc \
+	 && test -f testdata/$from \
+	 && test "$from" != "BIG5HKSCS" \
+	 && test "$from" != "EUC-JISX0213" \
+	 && test "$from" != "SHIFT_JISX0213" \
+	 && test "$from" != "TSCII"; then
+
+	# Identify the the FROM and TO charmaps.
 	echo $ac_n "test charmap: $from -> $t $ac_c"
-	$PROG -f ../localedata/charmaps/$from -t ../localedata/charmaps/$tc \
-	      testdata/$from < /dev/null > $temp1 ||
+
+	# Convert from FROM to TO using the initial data in testdata/FROM
+	# and storing in $temp1.
+	$PROG -f ../localedata/charmaps/$fromc -t ../localedata/charmaps/$tc \
+	  testdata/$from < /dev/null > $temp1 ||
 	  { if test $? -gt 128; then exit 1; fi
 	    echo "FAILED"; failed=1; continue; }
+
+	# The conversion from FROM to TO of the testdata/FROM succeeded.
 	echo $ac_n "OK$ac_c"
+
+	# If testdata/FROM..TO exists then we compare the generated output
+	# in $temp1 to the the expected output in FROM..TO.
 	if test -s testdata/$from..$t; then
 	  LC_ALL=C cmp $temp1 testdata/$from..$t > /dev/null 2>&1 ||
 	    { echo "/FAILED"; failed=1; continue; }
+
+	  # The comparison to expected output succeeded.
 	  echo $ac_n "/OK$ac_c"
 	fi
+
+	# Conversion *and* the comparison succeeded.
 	echo $ac_n " -> $from $ac_c"
+
+	# Run the conversion from TO to FROM (backwards) storing in $temp2.
 	$PROG -t ../localedata/charmaps/$from -f ../localedata/charmaps/$tc \
 	      -o $temp2 $temp1 < /dev/null ||
 	  { if test $? -gt 128; then exit 1; fi
 	    echo "FAILED"; failed=1; continue; }
+
+	# Conversion succeeded.
 	echo $ac_n "OK$ac_c"
+
+	# Compare the expected data with the converted data.
 	test -s $temp1 &&
 	LC_ALL=C cmp testdata/$from $temp2 > /dev/null 2>&1 ||
 	  { echo "/FAILED"; failed=1; continue; }
+
+	# We succeeded in converting to the target charmap and back
+	# again and both directions worked and the data matched.
 	echo "/OK"
+
+	# Cleanup test.
 	rm -f $temp1 $temp2
       fi
     done
   fi
 
+  # Test 1b: Convert "The Art of War" (larger text) [Non-ASCII subset]
   if test "$subset" = N; then
     echo $ac_n "      suntzu: ASCII -> $to -> ASCII $ac_c"
+
+    # Convert only from ASCII to target and back to ASCII.
     $PROG -f ASCII -t $to testdata/suntzus < /dev/null |
     $PROG -f $to -t ASCII > $temp1 ||
       { if test $? -gt 128; then exit 1; fi
 	echo "FAILED"; failed=1; continue; }
+
+    # Conversion succeeded.
     echo $ac_n "OK$ac_c"
+
+    # Compare against expected ASCII data.
     LC_ALL=C cmp testdata/suntzus $temp1 ||
       { echo "/FAILED"; failed=1; continue; }
+
+    # Conversion *and* comparison succeeded.
     echo "/OK"
+    # Cleanup test.
+    rm -f $temp1
   fi
-done < TESTS
+done < "$iconv_test1"
 
-# We read the file named TESTS2.  All non-empty lines not starting with
+# We read the file named ICONV_TEST2.  All non-empty lines not starting with
 # `#' are interpreted as commands.
 while read utf8 from filename; do
   # Ignore empty and comment lines.
@@ -182,7 +275,7 @@  while read utf8 from filename; do
     { echo "/FAILED"; failed=1; continue; }
   echo "OK"
 
-done < TESTS2
+done < "$iconv_test2"
 
 # Check for crashes in decoders.
 printf '\016\377\377\377\377\377\377\377' > $temp1
diff --git a/localedata/charmaps/BIG5 b/localedata/charmaps/BIG5
index 50f5f16cd2..b552e43763 100644
--- a/localedata/charmaps/BIG5
+++ b/localedata/charmaps/BIG5
@@ -19,6 +19,9 @@ 
 %     /xa2/xcc, /xa2/xce, /xf9/xe9, /xf9/xea, /xf9/xeb,
 %     /xf9/xf9, /xf9/xfa, /xf9/xfb, /xf9/xfc, /xf9/xfd
 %
+% The %IRREVERSIBLE% markup is used to generate the data used by iconv.
+% Please do not remove the %IRREVERSIBLE% markup.
+%
 % alias BIG5-CP950
 
 CHARMAP
@@ -417,8 +420,10 @@  CHARMAP
 <U3027>		/xa2/xc9	HANGZHOU NUMERAL SEVEN
 <U3028>		/xa2/xca	HANGZHOU NUMERAL EIGHT
 <U3029>		/xa2/xcb	HANGZHOU NUMERAL NINE
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U5341>		/xa2/xcc	<CJK>
 <U5344>		/xa2/xcd	<CJK>
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U5345>		/xa2/xce	<CJK>
 <UFF21>		/xa2/xcf	FULLWIDTH LATIN CAPITAL LETTER A
 <UFF22>		/xa2/xd0	FULLWIDTH LATIN CAPITAL LETTER B
@@ -14050,6 +14055,7 @@  CHARMAP
 <U2552>		/xf9/xe6	BOX DRAWINGS DOWN SINGLE AND RIGHT DOUBLE
 <U2564>		/xf9/xe7	BOX DRAWINGS DOWN SINGLE AND HORIZONTAL DOUBLE
 <U2555>		/xf9/xe8	BOX DRAWINGS DOWN SINGLE AND LEFT DOUBLE
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U255E>		/xf9/xe9	BOX DRAWINGS VERTICAL SINGLE AND RIGHT DOUBLE
 %IRREVERSIBLE%<U256A>		/xf9/xea	BOX DRAWINGS VERTICAL SINGLE AND HORIZONTAL DOUBLE
 %IRREVERSIBLE%<U2561>		/xf9/xeb	BOX DRAWINGS VERTICAL SINGLE AND LEFT DOUBLE
@@ -14066,6 +14072,7 @@  CHARMAP
 <U2568>		/xf9/xf6	BOX DRAWINGS UP DOUBLE AND HORIZONTAL SINGLE
 <U255C>		/xf9/xf7	BOX DRAWINGS UP DOUBLE AND LEFT SINGLE
 <U2551>		/xf9/xf8	BOX DRAWINGS DOUBLE VERTICAL
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2550>		/xf9/xf9	BOX DRAWINGS DOUBLE HORIZONTAL
 %IRREVERSIBLE%<U256D>		/xf9/xfa	BOX DRAWINGS LIGHT ARC DOWN AND RIGHT
 %IRREVERSIBLE%<U256E>		/xf9/xfb	BOX DRAWINGS LIGHT ARC DOWN AND LEFT
diff --git a/localedata/charmaps/BIG5-HKSCS b/localedata/charmaps/BIG5-HKSCS
index 0735efc5c8..3489654fb2 100644
--- a/localedata/charmaps/BIG5-HKSCS
+++ b/localedata/charmaps/BIG5-HKSCS
@@ -10,6 +10,12 @@ 
 % Last updated from the HKSCS-2008 standard
 % http://www.ogcio.gov.hk/en/business/tech_promotion/ccli/terms/doc/e_hkscs_2008.pdf
 %
+% The latest available standard is HKSCS-2016 which has not yet been applied
+% https://www.ogcio.gov.hk/en/our_work/business/tech_promotion/ccli/terms/doc/e_hkscs_2016.pdf
+%
+% The %IRREVERSIBLE% markup is used to generate the data used by iconv.
+% Please do not remove the %IRREVERSIBLE% markup.
+%
 
 CHARMAP
 <U0000>     /x00         NULL
@@ -300,8 +306,31 @@  CHARMAP
 <U00D3>     /x88/x5f     LATIN CAPITAL LETTER O WITH ACUTE
 <U01D1>     /x88/x60     LATIN CAPITAL LETTER O WITH CARON
 <U00D2>     /x88/x61     LATIN CAPITAL LETTER O WITH GRAVE
+%
+% BIG5-HKSCS has 4 mappings that cannot be represented in a normal
+% character map. For BIG5-HKSCS 2008 (Amd. 1 to Amd. 6) they are:
+% /x88/x62 => <U00CA><U0304>
+% /x88/x64 => <U00CA><U030C>
+% /x88/xa3 => <U00EA><U0304>
+% /x88/xa5 => <U00EA><U030C>
+%
+% These mappings are noted in BIG5-HKSCS 2016 standard as documented in the
+% initial comments.
+%
+% These mappings cannot be represented in a traditional POSIX character map
+% which has no way to indicate that a sequence of multi-byte characters
+% generates multiple wide characters.  Therefore we document them here but do
+% not include them. Note that iconv is capable of and supports such
+% conversions, but iconv when run with character maps as from-encoding or
+% to-encoding is unable to support such conversions.
+%
+% Note that iconv is capable of and supports such conversions, but iconv
+% when run with character maps as from-encoding or to-encoding is unable
+% to support such conversions.
+
 %<U00CA><U0304>     /x88/x62
 <U1EBE>     /x88/x63     LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND ACUTE
+% See note above.
 %<U00CA><U030C>     /x88/x64
 <U1EC0>     /x88/x65     LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND GRAVE
 <U00CA>     /x88/x66     LATIN CAPITAL LETTER E WITH CIRCUMFLEX
@@ -331,8 +360,10 @@  CHARMAP
 <U01DA>     /x88/x7e     LATIN SMALL LETTER U WITH DIAERESIS AND CARON
 <U01DC>     /x88/xa1     LATIN SMALL LETTER U WITH DIAERESIS AND GRAVE
 <U00FC>     /x88/xa2     LATIN SMALL LETTER U WITH DIAERESIS
+% See note above.
 %<U00EA><U0304>     /x88/xa3
 <U1EBF>     /x88/xa4     LATIN SMALL LETTER E WITH CIRCUMFLEX AND ACUTE
+% See note above.
 %<U00EA><U030C>     /x88/xa5
 <U1EC1>     /x88/xa6     LATIN SMALL LETTER E WITH CIRCUMFLEX AND GRAVE
 <U00EA>     /x88/xa7     LATIN SMALL LETTER E WITH CIRCUMFLEX
@@ -4201,6 +4232,7 @@  CHARMAP
 <U2510>     /xa2/x7b     BOX DRAWINGS LIGHT DOWN AND LEFT
 <U2514>     /xa2/x7c     BOX DRAWINGS LIGHT UP AND RIGHT
 <U2518>     /xa2/x7d     BOX DRAWINGS LIGHT UP AND LEFT
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U256D>     /xa2/x7e     BOX DRAWINGS LIGHT ARC DOWN AND RIGHT
 %IRREVERSIBLE%<U256E>     /xa2/xa1     BOX DRAWINGS LIGHT ARC DOWN AND LEFT
 %IRREVERSIBLE%<U2570>     /xa2/xa2     BOX DRAWINGS LIGHT ARC UP AND RIGHT
diff --git a/localedata/charmaps/EUC-JP-MS b/localedata/charmaps/EUC-JP-MS
index 6b1c9e4733..9fd1139ea4 100644
--- a/localedata/charmaps/EUC-JP-MS
+++ b/localedata/charmaps/EUC-JP-MS
@@ -839,16 +839,19 @@  CHARMAP
 <U337E>     /xad/xed     SQUARE ERA NAME MEIZI
 <U337D>     /xad/xee     SQUARE ERA NAME TAISYOU
 <U337C>     /xad/xef     SQUARE ERA NAME SYOUWA
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2252>     /xad/xf0     APPROXIMATELY EQUAL TO OR THE IMAGE OF
 %IRREVERSIBLE%<U2261>     /xad/xf1     IDENTICAL TO
 %IRREVERSIBLE%<U222B>     /xad/xf2     INTEGRAL
 <U222E>     /xad/xf3     CONTOUR INTEGRAL
 <U2211>     /xad/xf4     N-ARY SUMMATION
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U221A>     /xad/xf5     SQUARE ROOT
 %IRREVERSIBLE%<U22A5>     /xad/xf6     UP TACK
 %IRREVERSIBLE%<U2220>     /xad/xf7     ANGLE
 <U221F>     /xad/xf8     RIGHT ANGLE
 <U22BF>     /xad/xf9     RIGHT TRIANGLE
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2235>     /xad/xfa     BECAUSE
 %IRREVERSIBLE%<U2229>     /xad/xfb     INTERSECTION
 %IRREVERSIBLE%<U222A>     /xad/xfc     UNION
@@ -8163,6 +8166,7 @@  CHARMAP
 <U00AF>     /x8f/xa2/xb4     MACRON
 <U02DB>     /x8f/xa2/xb5     OGONEK
 <U02DA>     /x8f/xa2/xb6     RING ABOVE
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<UFF5E>     /x8f/xa2/xb7     FULLWIDTH TILDE
 <U0384>     /x8f/xa2/xb8     GREEK TONOS
 <U0385>     /x8f/xa2/xb9     GREEK DIALYTIKA TONOS
@@ -8175,6 +8179,7 @@  CHARMAP
 <U00AE>     /x8f/xa2/xee     REGISTERED SIGN
 <U2122>     /x8f/xa2/xef     TRADE MARK SIGN
 <U00A4>     /x8f/xa2/xf0     CURRENCY SIGN
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2116>     /x8f/xa2/xf1     NUMERO SIGN
 <U0386>     /x8f/xa6/xe1     GREEK CAPITAL LETTER ALPHA WITH TONOS
 <U0388>     /x8f/xa6/xe2     GREEK CAPITAL LETTER EPSILON WITH TONOS
@@ -14232,6 +14237,7 @@  CHARMAP
 <U2177>     /x8f/xf3/xfa     SMALL ROMAN NUMERAL EIGHT
 <U2178>     /x8f/xf3/xfb     SMALL ROMAN NUMERAL NINE
 <U2179>     /x8f/xf3/xfc     SMALL ROMAN NUMERAL TEN
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2160>     /x8f/xf3/xfd     ROMAN NUMERAL ONE
 %IRREVERSIBLE%<U2161>     /x8f/xf3/xfe     ROMAN NUMERAL TWO
 %IRREVERSIBLE%<U2162>     /x8f/xf4/xa1     ROMAN NUMERAL THREE
@@ -14244,6 +14250,7 @@  CHARMAP
 %IRREVERSIBLE%<U2169>     /x8f/xf4/xa8     ROMAN NUMERAL TEN
 <UFF07>     /x8f/xf4/xa9     FULLWIDTH APOSTROPHE
 <UFF02>     /x8f/xf4/xaa     FULLWIDTH QUOTATION MARK
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U3231>     /x8f/xf4/xab     PARENTHESIZED IDEOGRAPH STOCK
 %IRREVERSIBLE%<U2116>     /x8f/xf4/xac     NUMERO SIGN
 %IRREVERSIBLE%<U2121>     /x8f/xf4/xad     TELEPHONE SIGN
diff --git a/localedata/charmaps/EUC-TW b/localedata/charmaps/EUC-TW
index c9c9cdd82a..a3adf38347 100644
--- a/localedata/charmaps/EUC-TW
+++ b/localedata/charmaps/EUC-TW
@@ -6009,6 +6009,7 @@  CHARMAP
 %
 % CNS 11643-1992 Plane 1 again
 %
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U3000>     /x8e/xa1/xa1/xa1 IDEOGRAPHIC SPACE
 %IRREVERSIBLE%<UFF0C>     /x8e/xa1/xa1/xa2 FULLWIDTH COMMA
 %IRREVERSIBLE%<U3001>     /x8e/xa1/xa1/xa3 IDEOGRAPHIC COMMA
diff --git a/localedata/charmaps/IBM1132 b/localedata/charmaps/IBM1132
index 948d5a4416..6154070aef 100644
--- a/localedata/charmaps/IBM1132
+++ b/localedata/charmaps/IBM1132
@@ -115,6 +115,7 @@  CHARMAP
 <U005F>     /x6d         LOW LINE
 <U003E>     /x6e         GREATER-THAN SIGN
 <U003F>     /x6f         QUESTION MARK
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U006B>     /x70         LATIN SMALL LETTER K
 <U0EA3>     /x72         LAO LETTER LO LING
 <U0EA5>     /x73         LAO LETTER LO LOOT
diff --git a/localedata/charmaps/IBM1133 b/localedata/charmaps/IBM1133
index a4848439db..cbe09214c3 100644
--- a/localedata/charmaps/IBM1133
+++ b/localedata/charmaps/IBM1133
@@ -219,6 +219,7 @@  CHARMAP
 <U0EC6>     /xdb         LAO KO LA
 <U0EDC>     /xdd         LAO HO NO
 <U0EDD>     /xde         LAO HO MO
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U006B>     /xdf         LATIN SMALL LETTER K
 <U0ED0>     /xf0         LAO DIGIT ZERO
 <U0ED1>     /xf1         LAO DIGIT ONE
diff --git a/localedata/charmaps/IBM1160 b/localedata/charmaps/IBM1160
index 646b921133..cce874f921 100644
--- a/localedata/charmaps/IBM1160
+++ b/localedata/charmaps/IBM1160
@@ -85,6 +85,7 @@  CHARMAP
 <U002B>     /x4e         PLUS SIGN
 <U007C>     /x4f         VERTICAL LINE
 <U0026>     /x50         AMPERSAND
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U0E48>     /x51         THAI CHARACTER MAI EK
 <U0E08>     /x52         THAI CHARACTER CHO CHAN
 <U0E09>     /x53         THAI CHARACTER CHO CHING
@@ -206,6 +207,7 @@  CHARMAP
 <U0047>     /xc7         LATIN CAPITAL LETTER G
 <U0048>     /xc8         LATIN CAPITAL LETTER H
 <U0049>     /xc9         LATIN CAPITAL LETTER I
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U0E49>     /xca         THAI CHARACTER MAI THO
 <U0E35>     /xcb         THAI CHARACTER SARA II
 <U0E36>     /xcc         THAI CHARACTER SARA UE
@@ -229,6 +231,7 @@  CHARMAP
 <U0E43>     /xde         THAI CHARACTER SARA AI MAIMUAN
 <U0E44>     /xdf         THAI CHARACTER SARA AI MAIMALAI
 <U005C>     /xe0         REVERSE SOLIDUS
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U0E4A>     /xe1         THAI CHARACTER MAI TRI
 <U0053>     /xe2         LATIN CAPITAL LETTER S
 <U0054>     /xe3         LATIN CAPITAL LETTER T
@@ -257,6 +260,7 @@  CHARMAP
 <U0E4B>     /xfa         THAI CHARACTER MAI CHATTAWA
 <U0E4C>     /xfb         THAI CHARACTER THANTHAKHAT
 <U0E4D>     /xfc         THAI CHARACTER NIKHAHIT
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U0E4B>     /xfd         THAI CHARACTER MAI CHATTAWA
 <U20AC>     /xfe         EURO SIGN
 <U009F>     /xff         <control>
diff --git a/localedata/charmaps/IBM1161 b/localedata/charmaps/IBM1161
index 9340ba141b..52337c6117 100644
--- a/localedata/charmaps/IBM1161
+++ b/localedata/charmaps/IBM1161
@@ -132,6 +132,7 @@  CHARMAP
 <U007D>     /x7d         RIGHT CURLY BRACKET
 <U007E>     /x7e         TILDE
 <U001A>     /x7f         <control>
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U0E48>     /xa0         THAI CHARACTER MAI EK
 <U0E01>     /xa1         THAI CHARACTER KO KAI
 <U0E02>     /xa2         THAI CHARACTER KHO KHAI
@@ -191,6 +192,7 @@  CHARMAP
 <U0E38>     /xd8         THAI CHARACTER SARA U
 <U0E39>     /xd9         THAI CHARACTER SARA UU
 <U0E3A>     /xda         THAI CHARACTER PHINTHU
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U0E49>     /xdb         THAI CHARACTER MAI THO
 %IRREVERSIBLE%<U0E4A>     /xdc         THAI CHARACTER MAI TRI
 %IRREVERSIBLE%<U0E4B>     /xdd         THAI CHARACTER MAI CHATTAWA
diff --git a/localedata/charmaps/TSCII b/localedata/charmaps/TSCII
index 9646f326cb..3d9ae1fb5e 100644
--- a/localedata/charmaps/TSCII
+++ b/localedata/charmaps/TSCII
@@ -2,8 +2,26 @@ 
 <comment_char> %
 <escape_char> /
 <mb_cur_min> 1
-<mb_cur_max> 1
-% based on TSCII version 1.7
+<mb_cur_max> 3
+
+% Tamil Script Code for Information Interchange
+%
+% Based on TSCII version 1.7
+%
+% The lower 128 code points are ASCII, but the upper code points are
+% TSCII characters that often map to multiple Unicode code points.  The
+% one-to-many mapping means that much of the character map is commented
+% out since we don't support many-to-one mappings in POSIX-compatible
+% character maps.  There are 179 such mappings where one encoded TSCII
+% character is mapped to more than one Unicode code point.
+%
+% Note that iconv is capable of and supports such conversions, but iconv
+% when run with character maps as from-encoding or to-encoding is unable
+% to support such conversions.
+%
+% For conversion reference:
+% https://www.unicode.org/notes/tn15/Tscii2Unicode2.pdf
+%
 
 CHARMAP
 <U0000>                      /x00         NULL
@@ -134,27 +152,30 @@  CHARMAP
 <U007D>                      /x7d         RIGHT CURLY BRACKET
 <U007E>                      /x7e         TILDE
 <U007F>                      /x7f         DELETE
-<U0BE6>                      /x80         TAMIL DIGIT ZERO (currently unassigned)
+<U0BE6>                      /x80         TAMIL DIGIT ZERO (Since Unicode 4.1)
 <U0BE7>                      /x81         TAMIL DIGIT ONE
-<U0BB8><U0BCD><U0BB0><U0BC0> /x82         TAMIL GLYPH SRI
+% Note: Prior to Unicode 4.1 the SRI ligature was:
+% <U0BB8><U0BCD><U0BB0><U0BC0>,
+% but since Unicode 4.1 the SRI ligature should start with <U0BB6>.
+% <U0BB6><U0BCD><U0BB0><U0BC0> /x82         TAMIL GLYPH SRI
 <U0B9C>                      /x83         TAMIL LETTER JA
-<U0B9C><U0BC1>               /x83/xa4     TAMIL GLYPH JU
-<U0B9C><U0BC2>               /x83/xa5     TAMIL GLYPH JUU
+% <U0B9C><U0BC1>               /x83/xa4     TAMIL GLYPH JU
+% <U0B9C><U0BC2>               /x83/xa5     TAMIL GLYPH JUU
 <U0BB7>                      /x84         TAMIL LETTER SSA
-<U0BB7><U0BC1>               /x84/xa4     TAMIL GLYPH SSU
-<U0BB7><U0BC2>               /x84/xa5     TAMIL GLYPH SSUU
+% <U0BB7><U0BC1>               /x84/xa4     TAMIL GLYPH SSU
+% <U0BB7><U0BC2>               /x84/xa5     TAMIL GLYPH SSUU
 <U0BB8>                      /x85         TAMIL LETTER SA
 <U0BB9>                      /x86         TAMIL LETTER HA
-<U0B95><U0BCD><U0BB7>        /x87         TAMIL GLYPH KSHA
-<U0B9C><U0BCD>               /x88         TAMIL GLYPH J
-<U0BB7><U0BCD>               /x89         TAMIL GLYPH SS
-<U0BB8><U0BCD>               /x8a         TAMIL GLYPH S
-<U0BB8><U0BC1>               /x8a/xa4     TAMIL GLYPH SU
-<U0BB8><U0BC2>               /x8a/xa5     TAMIL GLYPH SUU
-<U0BB9><U0BCD>               /x8b         TAMIL GLYPH H
-<U0BB9><U0BC1>               /x8b/xa4     TAMIL GLYPH HU
-<U0BB9><U0BC2>               /x8b/xa5     TAMIL GLYPH HUU
-<U0B95><U0BCD><U0BB7><U0BCD> /x8c         TAMIL GLYPH KSH
+% <U0B95><U0BCD><U0BB7>        /x87         TAMIL GLYPH KSHA
+% <U0B9C><U0BCD>               /x88         TAMIL GLYPH J
+% <U0BB7><U0BCD>               /x89         TAMIL GLYPH SS
+% <U0BB8><U0BCD>               /x8a         TAMIL GLYPH S
+% <U0BB8><U0BC1>               /x8a/xa4     TAMIL GLYPH SU
+% <U0BB8><U0BC2>               /x8a/xa5     TAMIL GLYPH SUU
+% <U0BB9><U0BCD>               /x8b         TAMIL GLYPH H
+% <U0BB9><U0BC1>               /x8b/xa4     TAMIL GLYPH HU
+% <U0BB9><U0BC2>               /x8b/xa5     TAMIL GLYPH HUU
+% <U0B95><U0BCD><U0BB7><U0BCD> /x8c         TAMIL GLYPH KSH
 <U0BE8>                      /x8d         TAMIL DIGIT TWO
 <U0BE9>                      /x8e         TAMIL DIGIT THREE
 <U0BEA>                      /x8f         TAMIL DIGIT FOUR
@@ -167,10 +188,10 @@  CHARMAP
 <U0BED>                      /x96         TAMIL DIGIT SEVEN
 <U0BEE>                      /x97         TAMIL DIGIT EIGHT
 <U0BEF>                      /x98         TAMIL DIGIT NINE
-<U0B99><U0BC1>               /x99         TAMIL GLYPH NGU
-<U0B9E><U0BC1>               /x9a         TAMIL GLYPH NYU
-<U0B99><U0BC2>               /x9b         TAMIL GLYPH NGUU
-<U0B9E><U0BC2>               /x9c         TAMIL GLYPH NYUU
+% <U0B99><U0BC1>               /x99         TAMIL GLYPH NGU
+% <U0B9E><U0BC1>               /x9a         TAMIL GLYPH NYU
+% <U0B99><U0BC2>               /x9b         TAMIL GLYPH NGUU
+% <U0B9E><U0BC2>               /x9c         TAMIL GLYPH NYUU
 <U0BF0>                      /x9d         TAMIL NUMBER TEN
 <U0BF1>                      /x9e         TAMIL NUMBER ONE HUNDRED
 <U0BF2>                      /x9f         TAMIL NUMBER ONE THOUSAND
@@ -180,124 +201,136 @@  CHARMAP
 <U0BC1>                      /xa4         TAMIL VOWEL SIGN U
 <U0BC2>                      /xa5         TAMIL VOWEL SIGN UU
 <U0BC6>                      /xa6         TAMIL VOWEL SIGN E
-<U0BCA>                      /xa6/xa1     TAMIL VOWEL SIGN O
-<U0B95><U0BC6>               /xa6/xb8     TAMIL GLYPH KE
-<U0B95><U0BCA>               /xa6/xb8/xa1 TAMIL GLYPH KAI
-<U0B99><U0BC6>               /xa6/xb9     TAMIL GLYPH NGE
-<U0B99><U0BCA>               /xa6/xb9/xa1 TAMIL GLYPH NGAI
-<U0B9A><U0BC6>               /xa6/xba     TAMIL GLYPH CE
-<U0B9A><U0BCA>               /xa6/xba/xa1 TAMIL GLYPH CAI
-<U0B9E><U0BC6>               /xa6/xbb     TAMIL GLYPH NYE
-<U0B9E><U0BCA>               /xa6/xbb/xa1 TAMIL GLYPH NYAI
-<U0B9F><U0BC6>               /xa6/xbc     TAMIL GLYPH TTE
-<U0B9F><U0BCA>               /xa6/xbc/xa1 TAMIL GLYPH TTAI
-<U0BA3><U0BC6>               /xa6/xbd     TAMIL GLYPH NNE
-<U0BA3><U0BCA>               /xa6/xbd/xa1 TAMIL GLYPH NNAI
-<U0BA4><U0BC6>               /xa6/xbe     TAMIL GLYPH TE
-<U0BA4><U0BCA>               /xa6/xbe/xa1 TAMIL GLYPH TAI
-<U0BA8><U0BC6>               /xa6/xbf     TAMIL GLYPH NE
-<U0BA8><U0BCA>               /xa6/xbf/xa1 TAMIL GLYPH NAI
-<U0BAA><U0BC6>               /xa6/xc0     TAMIL GLYPH PE
-<U0BAA><U0BCA>               /xa6/xc0/xa1 TAMIL GLYPH PAI
-<U0BAE><U0BC6>               /xa6/xc1     TAMIL GLYPH ME
-<U0BAE><U0BCA>               /xa6/xc1/xa1 TAMIL GLYPH MAI
-<U0BAF><U0BC6>               /xa6/xc2     TAMIL GLYPH YE
-<U0BAF><U0BCA>               /xa6/xc2/xa1 TAMIL GLYPH YAI
-<U0BB0><U0BC6>               /xa6/xc3     TAMIL GLYPH RE
-<U0BB0><U0BCA>               /xa6/xc3/xa1 TAMIL GLYPH RAI
-<U0BB2><U0BC6>               /xa6/xc4     TAMIL GLYPH LE
-<U0BB2><U0BCA>               /xa6/xc4/xa1 TAMIL GLYPH LAI
-<U0BB5><U0BC6>               /xa6/xc5     TAMIL GLYPH VE
-<U0BB5><U0BCA>               /xa6/xc5/xa1 TAMIL GLYPH VAI
-<U0BB4><U0BC6>               /xa6/xc6     TAMIL GLYPH LLLE
-<U0BB4><U0BCA>               /xa6/xc6/xa1 TAMIL GLYPH LLLAI
-<U0BB3><U0BC6>               /xa6/xc7     TAMIL GLYPH LLE
-<U0BB3><U0BCA>               /xa6/xc7/xa1 TAMIL GLYPH LLAI
-<U0BB1><U0BC6>               /xa6/xc8     TAMIL GLYPH RRE
-<U0BB1><U0BCA>               /xa6/xc8/xa1 TAMIL GLYPH RRAI
-<U0BA9><U0BC6>               /xa6/xc9     TAMIL GLYPH NNNE
-<U0BA9><U0BCA>               /xa6/xc9/xa1 TAMIL GLYPH NNNAI
+% The encoded /xa6/xa1 is <U0BC6><U0BBE> which is the decomposition of
+% <U0BCA> and is encoded exactly the same and collated the same.
+%
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
+%IRREVERSIBLE% <U0BCA>                      /xa6/xa1     TAMIL VOWEL SIGN O
+% <U0B95><U0BC6>               /xa6/xb8     TAMIL GLYPH KE
+% <U0B95><U0BCA>               /xa6/xb8/xa1 TAMIL GLYPH KAI
+% <U0B99><U0BC6>               /xa6/xb9     TAMIL GLYPH NGE
+% <U0B99><U0BCA>               /xa6/xb9/xa1 TAMIL GLYPH NGAI
+% <U0B9A><U0BC6>               /xa6/xba     TAMIL GLYPH CE
+% <U0B9A><U0BCA>               /xa6/xba/xa1 TAMIL GLYPH CAI
+% <U0B9E><U0BC6>               /xa6/xbb     TAMIL GLYPH NYE
+% <U0B9E><U0BCA>               /xa6/xbb/xa1 TAMIL GLYPH NYAI
+% <U0B9F><U0BC6>               /xa6/xbc     TAMIL GLYPH TTE
+% <U0B9F><U0BCA>               /xa6/xbc/xa1 TAMIL GLYPH TTAI
+% <U0BA3><U0BC6>               /xa6/xbd     TAMIL GLYPH NNE
+% <U0BA3><U0BCA>               /xa6/xbd/xa1 TAMIL GLYPH NNAI
+% <U0BA4><U0BC6>               /xa6/xbe     TAMIL GLYPH TE
+% <U0BA4><U0BCA>               /xa6/xbe/xa1 TAMIL GLYPH TAI
+% <U0BA8><U0BC6>               /xa6/xbf     TAMIL GLYPH NE
+% <U0BA8><U0BCA>               /xa6/xbf/xa1 TAMIL GLYPH NAI
+% <U0BAA><U0BC6>               /xa6/xc0     TAMIL GLYPH PE
+% <U0BAA><U0BCA>               /xa6/xc0/xa1 TAMIL GLYPH PAI
+% <U0BAE><U0BC6>               /xa6/xc1     TAMIL GLYPH ME
+% <U0BAE><U0BCA>               /xa6/xc1/xa1 TAMIL GLYPH MAI
+% <U0BAF><U0BC6>               /xa6/xc2     TAMIL GLYPH YE
+% <U0BAF><U0BCA>               /xa6/xc2/xa1 TAMIL GLYPH YAI
+% <U0BB0><U0BC6>               /xa6/xc3     TAMIL GLYPH RE
+% <U0BB0><U0BCA>               /xa6/xc3/xa1 TAMIL GLYPH RAI
+% <U0BB2><U0BC6>               /xa6/xc4     TAMIL GLYPH LE
+% <U0BB2><U0BCA>               /xa6/xc4/xa1 TAMIL GLYPH LAI
+% <U0BB5><U0BC6>               /xa6/xc5     TAMIL GLYPH VE
+% <U0BB5><U0BCA>               /xa6/xc5/xa1 TAMIL GLYPH VAI
+% <U0BB4><U0BC6>               /xa6/xc6     TAMIL GLYPH LLLE
+% <U0BB4><U0BCA>               /xa6/xc6/xa1 TAMIL GLYPH LLLAI
+% <U0BB3><U0BC6>               /xa6/xc7     TAMIL GLYPH LLE
+% <U0BB3><U0BCA>               /xa6/xc7/xa1 TAMIL GLYPH LLAI
+% <U0BB1><U0BC6>               /xa6/xc8     TAMIL GLYPH RRE
+% <U0BB1><U0BCA>               /xa6/xc8/xa1 TAMIL GLYPH RRAI
+% <U0BA9><U0BC6>               /xa6/xc9     TAMIL GLYPH NNNE
+% <U0BA9><U0BCA>               /xa6/xc9/xa1 TAMIL GLYPH NNNAI
 <U0BC7>                      /xa7         TAMIL VOWEL SIGN EE
-<U0BCB>                      /xa7/xa1     TAMIL VOWEL SIGN OO
+% The encoded /xa7/xa1 is <U0BC7><U0BBE> which is the decomposition of
+% <U0BCB> and is encoded exactly the same and collated the same.
+%
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
+%IRREVERSIBLE% <U0BCB>                      /xa7/xa1     TAMIL VOWEL SIGN OO
 <U0BCC>                      /xa7/xaa     TAMIL VOWEL SIGN AU
-<U0B95><U0BC7>               /xa7/xb8     TAMIL GLYPH KEE
-<U0B95><U0BCB>               /xa7/xb8/xa1 TAMIL GLYPH KOO
-<U0B95><U0BCC>               /xa7/xb8/xaa TAMIL GLYPH KAU
-<U0B99><U0BC7>               /xa7/xb9     TAMIL GLYPH NGEE
-<U0B99><U0BCB>               /xa7/xb9/xa1 TAMIL GLYPH NGOO
-<U0B99><U0BCC>               /xa7/xb9/xaa TAMIL GLYPH NGAU
-<U0B9A><U0BC7>               /xa7/xba     TAMIL GLYPH CEE
-<U0B9A><U0BCB>               /xa7/xba/xa1 TAMIL GLYPH COO
-<U0B9A><U0BCC>               /xa7/xba/xaa TAMIL GLYPH CAU
-<U0B9E><U0BC7>               /xa7/xbb     TAMIL GLYPH NYEE
-<U0B9E><U0BCB>               /xa7/xbb/xa1 TAMIL GLYPH NYOO
-<U0B9E><U0BCC>               /xa7/xbb/xaa TAMIL GLYPH NYAU
-<U0B9F><U0BC7>               /xa7/xbc     TAMIL GLYPH TTEE
-<U0B9F><U0BCB>               /xa7/xbc/xa1 TAMIL GLYPH TTOO
-<U0B9F><U0BCC>               /xa7/xbc/xaa TAMIL GLYPH TTAU
-<U0BA3><U0BC7>               /xa7/xbd     TAMIL GLYPH NNEE
-<U0BA3><U0BCB>               /xa7/xbd/xa1 TAMIL GLYPH NNOO
-<U0BA3><U0BCC>               /xa7/xbd/xaa TAMIL GLYPH NNAU
-<U0BA4><U0BC7>               /xa7/xbe     TAMIL GLYPH TEE
-<U0BA4><U0BCB>               /xa7/xbe/xa1 TAMIL GLYPH TOO
-<U0BA4><U0BCC>               /xa7/xbe/xaa TAMIL GLYPH TAU
-<U0BA8><U0BC7>               /xa7/xbf     TAMIL GLYPH NEE
-<U0BA8><U0BCB>               /xa7/xbf/xa1 TAMIL GLYPH NOO
-<U0BA8><U0BCC>               /xa7/xbf/xaa TAMIL GLYPH NAU
-<U0BAA><U0BC7>               /xa7/xc0     TAMIL GLYPH PEE
-<U0BAA><U0BCB>               /xa7/xc0/xa1 TAMIL GLYPH POO
-<U0BAA><U0BCC>               /xa7/xc0/xaa TAMIL GLYPH PAU
-<U0BAE><U0BC7>               /xa7/xc1     TAMIL GLYPH MEE
-<U0BAE><U0BCB>               /xa7/xc1/xa1 TAMIL GLYPH MOO
-<U0BAE><U0BCC>               /xa7/xc1/xaa TAMIL GLYPH MAU
-<U0BAF><U0BC7>               /xa7/xc2     TAMIL GLYPH YEE
-<U0BAF><U0BCB>               /xa7/xc2/xa1 TAMIL GLYPH YOO
-<U0BAF><U0BCC>               /xa7/xc2/xaa TAMIL GLYPH YAU
-<U0BB0><U0BC7>               /xa7/xc3     TAMIL GLYPH REE
-<U0BB0><U0BCB>               /xa7/xc3/xa1 TAMIL GLYPH ROO
-<U0BB0><U0BCC>               /xa7/xc3/xaa TAMIL GLYPH RAU
-<U0BB2><U0BC7>               /xa7/xc4     TAMIL GLYPH LEE
-<U0BB2><U0BCB>               /xa7/xc4/xa1 TAMIL GLYPH LOO
-<U0BB2><U0BCC>               /xa7/xc4/xaa TAMIL GLYPH LAU
-<U0BB5><U0BC7>               /xa7/xc5     TAMIL GLYPH VEE
-<U0BB5><U0BCB>               /xa7/xc5/xa1 TAMIL GLYPH VOO
-<U0BB5><U0BCC>               /xa7/xc5/xaa TAMIL GLYPH VAU
-<U0BB4><U0BC7>               /xa7/xc6     TAMIL GLYPH LLLEE
-<U0BB4><U0BCB>               /xa7/xc6/xa1 TAMIL GLYPH LLLOO
-<U0BB4><U0BCC>               /xa7/xc6/xaa TAMIL GLYPH LLLAU
-<U0BB3><U0BC7>               /xa7/xc7     TAMIL GLYPH LLEE
-<U0BB3><U0BCB>               /xa7/xc7/xa1 TAMIL GLYPH LLOO
-<U0BB3><U0BCC>               /xa7/xc7/xaa TAMIL GLYPH LLAU
-<U0BB1><U0BC7>               /xa7/xc8     TAMIL GLYPH RREE
-<U0BB1><U0BCB>               /xa7/xc8/xa1 TAMIL GLYPH RROO
-<U0BB1><U0BCC>               /xa7/xc8/xaa TAMIL GLYPH RRAU
-<U0BA9><U0BC7>               /xa7/xc9     TAMIL GLYPH NNNEE
-<U0BA9><U0BCB>               /xa7/xc9/xa1 TAMIL GLYPH NNNOO
-<U0BA9><U0BCC>               /xa7/xc9/xaa TAMIL GLYPH NNNAU
+% <U0B95><U0BC7>               /xa7/xb8     TAMIL GLYPH KEE
+% <U0B95><U0BCB>               /xa7/xb8/xa1 TAMIL GLYPH KOO
+% <U0B95><U0BCC>               /xa7/xb8/xaa TAMIL GLYPH KAU
+% <U0B99><U0BC7>               /xa7/xb9     TAMIL GLYPH NGEE
+% <U0B99><U0BCB>               /xa7/xb9/xa1 TAMIL GLYPH NGOO
+% <U0B99><U0BCC>               /xa7/xb9/xaa TAMIL GLYPH NGAU
+% <U0B9A><U0BC7>               /xa7/xba     TAMIL GLYPH CEE
+% <U0B9A><U0BCB>               /xa7/xba/xa1 TAMIL GLYPH COO
+% <U0B9A><U0BCC>               /xa7/xba/xaa TAMIL GLYPH CAU
+% <U0B9E><U0BC7>               /xa7/xbb     TAMIL GLYPH NYEE
+% <U0B9E><U0BCB>               /xa7/xbb/xa1 TAMIL GLYPH NYOO
+% <U0B9E><U0BCC>               /xa7/xbb/xaa TAMIL GLYPH NYAU
+% <U0B9F><U0BC7>               /xa7/xbc     TAMIL GLYPH TTEE
+% <U0B9F><U0BCB>               /xa7/xbc/xa1 TAMIL GLYPH TTOO
+% <U0B9F><U0BCC>               /xa7/xbc/xaa TAMIL GLYPH TTAU
+% <U0BA3><U0BC7>               /xa7/xbd     TAMIL GLYPH NNEE
+% <U0BA3><U0BCB>               /xa7/xbd/xa1 TAMIL GLYPH NNOO
+% <U0BA3><U0BCC>               /xa7/xbd/xaa TAMIL GLYPH NNAU
+% <U0BA4><U0BC7>               /xa7/xbe     TAMIL GLYPH TEE
+% <U0BA4><U0BCB>               /xa7/xbe/xa1 TAMIL GLYPH TOO
+% <U0BA4><U0BCC>               /xa7/xbe/xaa TAMIL GLYPH TAU
+% <U0BA8><U0BC7>               /xa7/xbf     TAMIL GLYPH NEE
+% <U0BA8><U0BCB>               /xa7/xbf/xa1 TAMIL GLYPH NOO
+% <U0BA8><U0BCC>               /xa7/xbf/xaa TAMIL GLYPH NAU
+% <U0BAA><U0BC7>               /xa7/xc0     TAMIL GLYPH PEE
+% <U0BAA><U0BCB>               /xa7/xc0/xa1 TAMIL GLYPH POO
+% <U0BAA><U0BCC>               /xa7/xc0/xaa TAMIL GLYPH PAU
+% <U0BAE><U0BC7>               /xa7/xc1     TAMIL GLYPH MEE
+% <U0BAE><U0BCB>               /xa7/xc1/xa1 TAMIL GLYPH MOO
+% <U0BAE><U0BCC>               /xa7/xc1/xaa TAMIL GLYPH MAU
+% <U0BAF><U0BC7>               /xa7/xc2     TAMIL GLYPH YEE
+% <U0BAF><U0BCB>               /xa7/xc2/xa1 TAMIL GLYPH YOO
+% <U0BAF><U0BCC>               /xa7/xc2/xaa TAMIL GLYPH YAU
+% <U0BB0><U0BC7>               /xa7/xc3     TAMIL GLYPH REE
+% <U0BB0><U0BCB>               /xa7/xc3/xa1 TAMIL GLYPH ROO
+% <U0BB0><U0BCC>               /xa7/xc3/xaa TAMIL GLYPH RAU
+% <U0BB2><U0BC7>               /xa7/xc4     TAMIL GLYPH LEE
+% <U0BB2><U0BCB>               /xa7/xc4/xa1 TAMIL GLYPH LOO
+% <U0BB2><U0BCC>               /xa7/xc4/xaa TAMIL GLYPH LAU
+% <U0BB5><U0BC7>               /xa7/xc5     TAMIL GLYPH VEE
+% <U0BB5><U0BCB>               /xa7/xc5/xa1 TAMIL GLYPH VOO
+% <U0BB5><U0BCC>               /xa7/xc5/xaa TAMIL GLYPH VAU
+% <U0BB4><U0BC7>               /xa7/xc6     TAMIL GLYPH LLLEE
+% <U0BB4><U0BCB>               /xa7/xc6/xa1 TAMIL GLYPH LLLOO
+% <U0BB4><U0BCC>               /xa7/xc6/xaa TAMIL GLYPH LLLAU
+% <U0BB3><U0BC7>               /xa7/xc7     TAMIL GLYPH LLEE
+% <U0BB3><U0BCB>               /xa7/xc7/xa1 TAMIL GLYPH LLOO
+% <U0BB3><U0BCC>               /xa7/xc7/xaa TAMIL GLYPH LLAU
+% <U0BB1><U0BC7>               /xa7/xc8     TAMIL GLYPH RREE
+% <U0BB1><U0BCB>               /xa7/xc8/xa1 TAMIL GLYPH RROO
+% <U0BB1><U0BCC>               /xa7/xc8/xaa TAMIL GLYPH RRAU
+% <U0BA9><U0BC7>               /xa7/xc9     TAMIL GLYPH NNNEE
+% <U0BA9><U0BCB>               /xa7/xc9/xa1 TAMIL GLYPH NNNOO
+% <U0BA9><U0BCC>               /xa7/xc9/xaa TAMIL GLYPH NNNAU
 <U0BC8>                      /xa8         TAMIL VOWEL SIGN AI
-<U0B95><U0BC8>               /xa8/xb8     TAMIL GLYPH KA
-<U0B99><U0BC8>               /xa8/xb9     TAMIL GLYPH NGA
-<U0B9A><U0BC8>               /xa8/xba     TAMIL GLYPH CA
-<U0B9E><U0BC8>               /xa8/xbb     TAMIL GLYPH NYA
-<U0B9F><U0BC8>               /xa8/xbc     TAMIL GLYPH TTA
-<U0BA3><U0BC8>               /xa8/xbd     TAMIL GLYPH NNA
-<U0BA4><U0BC8>               /xa8/xbe     TAMIL GLYPH TA
-<U0BA8><U0BC8>               /xa8/xbf     TAMIL GLYPH NA
-<U0BAA><U0BC8>               /xa8/xc0     TAMIL GLYPH PA
-<U0BAE><U0BC8>               /xa8/xc1     TAMIL GLYPH MA
-<U0BAF><U0BC8>               /xa8/xc2     TAMIL GLYPH YA
-<U0BB0><U0BC8>               /xa8/xc3     TAMIL GLYPH RA
-<U0BB2><U0BC8>               /xa8/xc4     TAMIL GLYPH LA
-<U0BB5><U0BC8>               /xa8/xc5     TAMIL GLYPH VA
-<U0BB4><U0BC8>               /xa8/xc6     TAMIL GLYPH LLLA
-<U0BB3><U0BC8>               /xa8/xc7     TAMIL GLYPH LLA
-<U0BB1><U0BC8>               /xa8/xc8     TAMIL GLYPH RRA
-<U0BA9><U0BC8>               /xa8/xc9     TAMIL GLYPH NNNA
+% <U0B95><U0BC8>               /xa8/xb8     TAMIL GLYPH KA
+% <U0B99><U0BC8>               /xa8/xb9     TAMIL GLYPH NGA
+% <U0B9A><U0BC8>               /xa8/xba     TAMIL GLYPH CA
+% <U0B9E><U0BC8>               /xa8/xbb     TAMIL GLYPH NYA
+% <U0B9F><U0BC8>               /xa8/xbc     TAMIL GLYPH TTA
+% <U0BA3><U0BC8>               /xa8/xbd     TAMIL GLYPH NNA
+% <U0BA4><U0BC8>               /xa8/xbe     TAMIL GLYPH TA
+% <U0BA8><U0BC8>               /xa8/xbf     TAMIL GLYPH NA
+% <U0BAA><U0BC8>               /xa8/xc0     TAMIL GLYPH PA
+% <U0BAE><U0BC8>               /xa8/xc1     TAMIL GLYPH MA
+% <U0BAF><U0BC8>               /xa8/xc2     TAMIL GLYPH YA
+% <U0BB0><U0BC8>               /xa8/xc3     TAMIL GLYPH RA
+% <U0BB2><U0BC8>               /xa8/xc4     TAMIL GLYPH LA
+% <U0BB5><U0BC8>               /xa8/xc5     TAMIL GLYPH VA
+% <U0BB4><U0BC8>               /xa8/xc6     TAMIL GLYPH LLLA
+% <U0BB3><U0BC8>               /xa8/xc7     TAMIL GLYPH LLA
+% <U0BB1><U0BC8>               /xa8/xc8     TAMIL GLYPH RRA
+% <U0BA9><U0BC8>               /xa8/xc9     TAMIL GLYPH NNNA
 <U00A9>                      /xa9         COPYRIGHT SIGN
 <U0BD7>                      /xaa         TAMIL AU LENGTH MARK
 <U0B85>                      /xab         TAMIL LETTER A
 <U0B86>                      /xac         TAMIL LETTER AA
-%IRREVERSIBLE%<U0B87>        /xad         TAMIL LETTER I
+% In TSCII 1.7 the hex value for TAMIL LETTER I was moved from /xad
+% to /xfe, thus we leave the next line commented out.
+%
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
+%IRREVERSIBLE% <U0B87>                      /xad         TAMIL LETTER I
 <U0B88>                      /xae         TAMIL LETTER II
 <U0B89>                      /xaf         TAMIL LETTER U
 <U0B8A>                      /xb0         TAMIL LETTER UU
@@ -326,63 +359,57 @@  CHARMAP
 <U0BB3>                      /xc7         TAMIL LETTER LLA
 <U0BB1>                      /xc8         TAMIL LETTER RRA
 <U0BA9>                      /xc9         TAMIL LETTER NNNA
-<U0B9F><U0BBF>               /xca         TAMIL GLYPH TI
-<U0B9F><U0BC0>               /xcb         TAMIL GLYPH TII
-<U0B95><U0BC1>               /xcc         TAMIL GLYPH KU
-<U0B9A><U0BC1>               /xcd         TAMIL GLYPH CU
-<U0B9F><U0BC1>               /xce         TAMIL GLYPH TTU
-<U0BA3><U0BC1>               /xcf         TAMIL GLYPH NNU
-<U0BA4><U0BC1>               /xd0         TAMIL GLYPH TU
-<U0BA8><U0BC1>               /xd1         TAMIL GLYPH NU
-<U0BAA><U0BC1>               /xd2         TAMIL GLYPH PU
-<U0BAE><U0BC1>               /xd3         TAMIL GLYPH MU
-<U0BAF><U0BC1>               /xd4         TAMIL GLYPH YU
-<U0BB0><U0BC1>               /xd5         TAMIL GLYPH RU
-<U0BB2><U0BC1>               /xd6         TAMIL GLYPH LU
-<U0BB5><U0BC1>               /xd7         TAMIL GLYPH VU
-<U0BB4><U0BC1>               /xd8         TAMIL GLYPH LLLU
-<U0BB3><U0BC1>               /xd9         TAMIL GLYPH LLU
-<U0BB1><U0BC1>               /xda         TAMIL GLYPH RRU
-<U0BA9><U0BC1>               /xdb         TAMIL GLYPH NNNU
-<U0B95><U0BC2>               /xdc         TAMIL GLYPH KUU
-<U0B9A><U0BC2>               /xdd         TAMIL GLYPH CUU
-<U0B9F><U0BC2>               /xde         TAMIL GLYPH TTUU
-<U0BA3><U0BC2>               /xdf         TAMIL GLYPH NNUU
-<U0BA4><U0BC2>               /xe0         TAMIL GLYPH TUU
-<U0BA8><U0BC2>               /xe1         TAMIL GLYPH NUU
-<U0BAA><U0BC2>               /xe2         TAMIL GLYPH PUU
-<U0BAE><U0BC2>               /xe3         TAMIL GLYPH MUU
-<U0BAF><U0BC2>               /xe4         TAMIL GLYPH YUU
-<U0BB0><U0BC2>               /xe5         TAMIL GLYPH RUU
-<U0BB2><U0BC2>               /xe6         TAMIL GLYPH LUU
-<U0BB5><U0BC2>               /xe7         TAMIL GLYPH VUU
-<U0BB4><U0BC2>               /xe8         TAMIL GLYPH LLLUU
-<U0BB3><U0BC2>               /xe9         TAMIL GLYPH LLUU
-<U0BB1><U0BC2>               /xea         TAMIL GLYPH RRUU
-<U0BA9><U0BC2>               /xeb         TAMIL GLYPH NNNUU
-<U0B95><U0BCD>               /xec         TAMIL GLYPH K
-<U0B99><U0BCD>               /xed         TAMIL GLYPH NG
-<U0B9A><U0BCD>               /xee         TAMIL GLYPH C
-<U0B9E><U0BCD>               /xef         TAMIL GLYPH NY
-<U0B9F><U0BCD>               /xf0         TAMIL GLYPH TT
-<U0BA3><U0BCD>               /xf1         TAMIL GLYPH NN
-<U0BA4><U0BCD>               /xf2         TAMIL GLYPH T
-<U0BA8><U0BCD>               /xf3         TAMIL GLYPH N
-<U0BAA><U0BCD>               /xf4         TAMIL GLYPH P
-<U0BAE><U0BCD>               /xf5         TAMIL GLYPH M
-<U0BAF><U0BCD>               /xf6         TAMIL GLYPH Y
-<U0BB0><U0BCD>               /xf7         TAMIL GLYPH R
-<U0BB2><U0BCD>               /xf8         TAMIL GLYPH L
-<U0BB5><U0BCD>               /xf9         TAMIL GLYPH V
-<U0BB4><U0BCD>               /xfa         TAMIL GLYPH LLL
-<U0BB3><U0BCD>               /xfb         TAMIL GLYPH LL
-<U0BB1><U0BCD>               /xfc         TAMIL GLYPH RR
-<U0BA9><U0BCD>               /xfd         TAMIL GLYPH NNN
+% <U0B9F><U0BBF>               /xca         TAMIL GLYPH TI
+% <U0B9F><U0BC0>               /xcb         TAMIL GLYPH TII
+% <U0B95><U0BC1>               /xcc         TAMIL GLYPH KU
+% <U0B9A><U0BC1>               /xcd         TAMIL GLYPH CU
+% <U0B9F><U0BC1>               /xce         TAMIL GLYPH TTU
+% <U0BA3><U0BC1>               /xcf         TAMIL GLYPH NNU
+% <U0BA4><U0BC1>               /xd0         TAMIL GLYPH TU
+% <U0BA8><U0BC1>               /xd1         TAMIL GLYPH NU
+% <U0BAA><U0BC1>               /xd2         TAMIL GLYPH PU
+% <U0BAE><U0BC1>               /xd3         TAMIL GLYPH MU
+% <U0BAF><U0BC1>               /xd4         TAMIL GLYPH YU
+% <U0BB0><U0BC1>               /xd5         TAMIL GLYPH RU
+% <U0BB2><U0BC1>               /xd6         TAMIL GLYPH LU
+% <U0BB5><U0BC1>               /xd7         TAMIL GLYPH VU
+% <U0BB4><U0BC1>               /xd8         TAMIL GLYPH LLLU
+% <U0BB3><U0BC1>               /xd9         TAMIL GLYPH LLU
+% <U0BB1><U0BC1>               /xda         TAMIL GLYPH RRU
+% <U0BA9><U0BC1>               /xdb         TAMIL GLYPH NNNU
+% <U0B95><U0BC2>               /xdc         TAMIL GLYPH KUU
+% <U0B9A><U0BC2>               /xdd         TAMIL GLYPH CUU
+% <U0B9F><U0BC2>               /xde         TAMIL GLYPH TTUU
+% <U0BA3><U0BC2>               /xdf         TAMIL GLYPH NNUU
+% <U0BA4><U0BC2>               /xe0         TAMIL GLYPH TUU
+% <U0BA8><U0BC2>               /xe1         TAMIL GLYPH NUU
+% <U0BAA><U0BC2>               /xe2         TAMIL GLYPH PUU
+% <U0BAE><U0BC2>               /xe3         TAMIL GLYPH MUU
+% <U0BAF><U0BC2>               /xe4         TAMIL GLYPH YUU
+% <U0BB0><U0BC2>               /xe5         TAMIL GLYPH RUU
+% <U0BB2><U0BC2>               /xe6         TAMIL GLYPH LUU
+% <U0BB5><U0BC2>               /xe7         TAMIL GLYPH VUU
+% <U0BB4><U0BC2>               /xe8         TAMIL GLYPH LLLUU
+% <U0BB3><U0BC2>               /xe9         TAMIL GLYPH LLUU
+% <U0BB1><U0BC2>               /xea         TAMIL GLYPH RRUU
+% <U0BA9><U0BC2>               /xeb         TAMIL GLYPH NNNUU
+% <U0B95><U0BCD>               /xec         TAMIL GLYPH K
+% <U0B99><U0BCD>               /xed         TAMIL GLYPH NG
+% <U0B9A><U0BCD>               /xee         TAMIL GLYPH C
+% <U0B9E><U0BCD>               /xef         TAMIL GLYPH NY
+% <U0B9F><U0BCD>               /xf0         TAMIL GLYPH TT
+% <U0BA3><U0BCD>               /xf1         TAMIL GLYPH NN
+% <U0BA4><U0BCD>               /xf2         TAMIL GLYPH T
+% <U0BA8><U0BCD>               /xf3         TAMIL GLYPH N
+% <U0BAA><U0BCD>               /xf4         TAMIL GLYPH P
+% <U0BAE><U0BCD>               /xf5         TAMIL GLYPH M
+% <U0BAF><U0BCD>               /xf6         TAMIL GLYPH Y
+% <U0BB0><U0BCD>               /xf7         TAMIL GLYPH R
+% <U0BB2><U0BCD>               /xf8         TAMIL GLYPH L
+% <U0BB5><U0BCD>               /xf9         TAMIL GLYPH V
+% <U0BB4><U0BCD>               /xfa         TAMIL GLYPH LLL
+% <U0BB3><U0BCD>               /xfb         TAMIL GLYPH LL
+% <U0BB1><U0BCD>               /xfc         TAMIL GLYPH RR
+% <U0BA9><U0BCD>               /xfd         TAMIL GLYPH NNN
 <U0B87>                      /xfe         TAMIL LETTER I
 END CHARMAP
-
-WIDTH
-<U0B82> 0
-<U0BC0> 0
-<U0BCD> 0
-END WIDTH
diff --git a/localedata/charmaps/WINDOWS-31J b/localedata/charmaps/WINDOWS-31J
index d5ab12abef..63e987f4ac 100644
--- a/localedata/charmaps/WINDOWS-31J
+++ b/localedata/charmaps/WINDOWS-31J
@@ -734,16 +734,19 @@  CHARMAP
 <U337E>     /x87/x8d     SQUARE ERA NAME MEIZI
 <U337D>     /x87/x8e     SQUARE ERA NAME TAISYOU
 <U337C>     /x87/x8f     SQUARE ERA NAME SYOUWA
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2252>     /x87/x90     APPROXIMATELY EQUAL TO OR THE IMAGE OF
 %IRREVERSIBLE%<U2261>     /x87/x91     IDENTICAL TO
 %IRREVERSIBLE%<U222B>     /x87/x92     INTEGRAL
 <U222E>     /x87/x93     CONTOUR INTEGRAL
 <U2211>     /x87/x94     N-ARY SUMMATION
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U221A>     /x87/x95     SQUARE ROOT
 %IRREVERSIBLE%<U22A5>     /x87/x96     UP TACK
 %IRREVERSIBLE%<U2220>     /x87/x97     ANGLE
 <U221F>     /x87/x98     RIGHT ANGLE
 <U22BF>     /x87/x99     RIGHT TRIANGLE
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2235>     /x87/x9a     BECAUSE
 %IRREVERSIBLE%<U2229>     /x87/x9b     INTERSECTION
 %IRREVERSIBLE%<U222A>     /x87/x9c     UNION
@@ -7167,7 +7170,7 @@  CHARMAP
 <U7464>     /xea/xa2     <CJK>
 <U51DC>     /xea/xa3     <CJK>
 <U7199>     /xea/xa4     <CJK>
-
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U7E8A>     /xed/x40     <CJK>
 %IRREVERSIBLE%<U891C>     /xed/x41     <CJK>
 %IRREVERSIBLE%<U9348>     /xed/x42     <CJK>
@@ -9434,6 +9437,7 @@  CHARMAP
 <U2177>     /xfa/x47     SMALL ROMAN NUMERAL EIGHT
 <U2178>     /xfa/x48     SMALL ROMAN NUMERAL NINE
 <U2179>     /xfa/x49     SMALL ROMAN NUMERAL TEN
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U2160>     /xfa/x4a     ROMAN NUMERAL ONE
 %IRREVERSIBLE%<U2161>     /xfa/x4b     ROMAN NUMERAL TWO
 %IRREVERSIBLE%<U2162>     /xfa/x4c     ROMAN NUMERAL THREE
@@ -9448,6 +9452,7 @@  CHARMAP
 <UFFE4>     /xfa/x55     FULLWIDTH BROKEN BAR
 <UFF07>     /xfa/x56     FULLWIDTH APOSTROPHE
 <UFF02>     /xfa/x57     FULLWIDTH QUOTATION MARK
+% The "IRREVERSIBLE" markup is used by iconv testing. Please do not remove.
 %IRREVERSIBLE%<U3231>     /xfa/x58     PARENTHESIZED IDEOGRAPH STOCK
 %IRREVERSIBLE%<U2116>     /xfa/x59     NUMERO SIGN
 %IRREVERSIBLE%<U2121>     /xfa/x5a     TELEPHONE SIGN