[SOLVED]Is glibc responsible for /usr/share/i18n/charmaps/

Message

lfs0a · Post by **lfs0a** » Tue Jun 25, 2019 1:52 pm

Is glibc responsible for /usr/share/i18n/charmaps/CP936.gz missing?

Today I upgraded grub,so I reboot my box.
Before I did that,I checked that my box is up for 4days.
After reboot,I got this when I start xterm

Code: Select all

Warning: couldn't find charset data for zh_CN.UTF-8; using ISO 8859-1.

There are barely nothing about it when I google.
But from the message,It's the charset data missing,
It's in /usr/share/i18n/charmaps?
And I find out /usr/share/i18n/charmaps belongs to glibc.
And looks like I just update it before:

Code: Select all

-rw-r--r-- 1 portage portage 42179 Jun 23 23:09 glibc-2.29-r2.ebuild

Can someone confirm this?

mike155 · Post by **mike155** » Tue Jun 25, 2019 2:31 pm

Is glibc responsible for /usr/share/i18n/charmaps/CP936.gz missing?

It's true that glibc installs files to "/usr/share/i18n/charmaps".

On my machine, there's a file

Code: Select all

/usr/share/i18n/charmaps/CP949.gz

and it belongs to glibc:

Code: Select all

# qfile /usr/share/i18n/charmaps/CP949.gz
sys-libs/glibc (/usr/share/i18n/charmaps/CP949.gz)

But I don't have a file "/usr/share/i18n/charmaps/CP936.gz"

Such a file is NOT part of glibc:

Code: Select all

# tar tvf /usr/portage/distfiles/glibc-2.29.tar.xz | grep charmaps | grep 949
-rw-rw-r-- root/root    733536 2019-01-31 17:45 glibc-2.29/localedata/charmaps/CP949

# tar tvf /usr/portage/distfiles/glibc-2.29.tar.xz | grep charmaps | grep 936
<nothing>

# tar tvf /usr/portage/distfiles/glibc-2.28.tar.xz | grep charmaps | grep 936
<nothing>

# tar tvf /usr/portage/distfiles/glibc-2.25.tar.xz | grep charmaps | grep 936
<nothing>

lfs0a · Post by **lfs0a** » Tue Jun 25, 2019 3:24 pm

mike155 wrote:
Is glibc responsible for /usr/share/i18n/charmaps/CP936.gz missing?
It's true that glibc installs files to "/usr/share/i18n/charmaps".
On my machine, there's a file
Code: Select all
/usr/share/i18n/charmaps/CP949.gz
and it belongs to glibc:
Code: Select all
# qfile /usr/share/i18n/charmaps/CP949.gz
sys-libs/glibc (/usr/share/i18n/charmaps/CP949.gz)
But I don't have a file "/usr/share/i18n/charmaps/CP936.gz"
Such a file is NOT part of glibc:
Code: Select all
# tar tvf /usr/portage/distfiles/glibc-2.29.tar.xz | grep charmaps | grep 949
-rw-rw-r-- root/root    733536 2019-01-31 17:45 glibc-2.29/localedata/charmaps/CP949

# tar tvf /usr/portage/distfiles/glibc-2.29.tar.xz | grep charmaps | grep 936
<nothing>

# tar tvf /usr/portage/distfiles/glibc-2.28.tar.xz | grep charmaps | grep 936
<nothing>

# tar tvf /usr/portage/distfiles/glibc-2.25.tar.xz | grep charmaps | grep 936
<nothing>

I googled /usr/share/i18n/charmaps/CP936.gz,
Very few showup.
I'm not sure what I need is CP936.gz.

What exactly charset data is?
Files in /usr/share/i18n/charmaps?
If it is,it's looking for UTF-8,somehow it can't find it so ISO 8859-1 is using,
But I got /usr/share/i18n/charmaps/UTF-8.gz!
I try en_US.UTF-8,same error.
Is that a glibc-2.29-r2 bug?

mike155 · Post by **mike155** » Tue Jun 25, 2019 3:38 pm

Why do you think that CP936.gz is missing?

You showed us the error message of xterm
Code: Select all
```
couldn't find charset data for zh_CN.UTF-8
```
So it seems that 'zh_CN.UTF-8' is missing???
Please show us the contents of the file '/etc/locale.gen'
Please show use the contents of the file '/etc/env.d/02locale'
Please show us the output of 'eselect locale list'
Please show us the output of 'emerge --info"

lfs0a · Post by **lfs0a** » Tue Jun 25, 2019 4:16 pm

mike155 wrote:
Why do you think that CP936.gz is missing?

You showed us the error message of xterm
Code: Select all
couldn't find charset data for zh_CN.UTF-8
So it seems that 'zh_CN.UTF-8' is missing???
Please show us the contents of the file '/etc/locale.gen'

Please show use the contents of the file '/etc/env.d/02locale'

Please show us the output of 'eselect locale list'

Please show us the output of 'emerge --info"

1，Well，there is a cp936 in NLS of kernel,
zh_CN.UTF-8 is generated by locale-gen.

2,

Code: Select all

W500 /usr/share/i18n # cat /etc/locale.gen
...
en_US.UTF-8 UTF-8
zh_CN.UTF-8 UTF-8

3.

Code: Select all

W500 /usr/share/i18n # cat /etc/env.d/02locale 
# Configuration file for eselect
# This file has been automatically generated.
LANG="en_US.utf8"

4.

Code: Select all

W500 /usr/share/i18n # eselect locale list
Available targets for the LANG variable:
  [1]   C
  [2]   C.utf8
  [3]   en_US.utf8 *
  [4]   POSIX
  [5]   zh_CN.utf8
  [ ]   (free form)

5.

Code: Select all

  emerge --info
Portage 2.3.62 (python 3.6.5-final-0, default/linux/amd64/17.1/no-multilib, gcc-8.3.0, glibc-2.29-r2, 4.19.52-gentoo x86_64)
=================================================================
System uname: Linux-4.19.52-gentoo-x86_64-Intel-R-_Core-TM-2_Extreme_CPU_X9100_@_3.06GHz-with-gentoo-2.6
KiB Mem:     5987700 total,   1789028 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Tue, 25 Jun 2019 00:45:01 +0000
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.31.1 p7) 2.31.1
app-shells/bash:          4.4_p23-r1::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.28.2-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.5.5::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.14.3::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.41.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.31.1-r6::gentoo
sys-devel/gcc:            8.3.0-r1::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.29-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: webrsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
 ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X acl acpi amd64 apm berkdb bluetooth bzip2 cli crypt cxx dri elogind fontconfig gdbm gif iconv infinality jpeg libtirpc ncurses nls nptl openmp pam pcre plasma png policykit pulseaudio readline seccomp ssl svg tcpd tiff truetype unicode xattr zlib" ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmxext mmx sse sse2 sse3 sse4.1 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

mike155 · Post by **mike155** » Tue Jun 25, 2019 5:53 pm

Thanks for the data!

Your output in 2,3,4,5 looks good. I don't see anything wrong.
Code: Select all
```
couldn't find charset data for zh_CN.UTF-8
```
I added 'zh_CN.UTF-8' to my '/etc/locale.gen' and ran 'locale-gen'. I logged in as an ordinary user, and started
Code: Select all
```
LANG=zh_CN.UTF-8 xterm
```
I didn't get an error message and the output of 'ls -la' looked Chinese. I have no idea why you get an error message.
Today, we use Unicode. 20 years ago we used 8-bit character sets like ISO-8859-1 and code pages - but today we try to avoid them. We sometimes need them to convert data from 20 years ago.

I'm happy that you configured your machine to use Unicode (zh_CN.UTF-8 ).
Well，there is a cp936 in NLS of kernel,
Well, it's there. But do you need it?

This option does not enable Chinese languages in the kernel. It allows you to mount old filesystems, where filenames were not encoded in Unicode, but in cp936. You usually mount those filesystems with option "codepage=cp936" or "iocharset=cp936". Do you have such filesystems and do you mount them with option "codepage=cp936" or "iocharset=cp936"?

If not, you probably don't need this option.

In case you have such filesystems: you still don't need a file '/usr/share/i18n/charmaps/CP936.gz' - because the kernel translates filenames encoded in cp936 automatically to Unicode. It doesn't need any files in your filesystem to do that.

So back to the question: do you get any warning or error messages about a missing file '/usr/share/i18n/charmaps/CP936.gz'? Or did you just think: there should be a file '/usr/share/i18n/charmaps/CP936.gz' because there's a cp936 option in the kernel?

lfs0a · Post by **lfs0a** » Wed Jun 26, 2019 4:29 am

mike155 wrote:Thanks for the data!
Your output in 2,3,4,5 looks good. I don't see anything wrong.
Code: Select all
couldn't find charset data for zh_CN.UTF-8
I added 'zh_CN.UTF-8' to my '/etc/locale.gen' and ran 'locale-gen'. I logged in as an ordinary user, and started
Code: Select all
LANG=zh_CN.UTF-8 xterm
I didn't get an error message and the output of 'ls -la' looked Chinese. I have no idea why you get an error message.
Today, we use Unicode. 20 years ago we used 8-bit character sets like ISO-8859-1 and code pages - but today we try to avoid them. We sometimes need them to convert data from 20 years ago.

I'm happy that you configured your machine to use Unicode (zh_CN.UTF-8 ).

Well，there is a cp936 in NLS of kernel,
Well, it's there. But do you need it?

This option does not enable Chinese languages in the kernel. It allows you to mount old filesystems, where filenames were not encoded in Unicode, but in cp936. You usually mount those filesystems with option "codepage=cp936" or "iocharset=cp936". Do you have such filesystems and do you mount them with option "codepage=cp936" or "iocharset=cp936"?

If not, you probably don't need this option.

In case you have such filesystems: you still don't need a file '/usr/share/i18n/charmaps/CP936.gz' - because the kernel translates filenames encoded in cp936 automatically to Unicode. It doesn't need any files in your filesystem to do that.

So back to the question: do you get any warning or error messages about a missing file '/usr/share/i18n/charmaps/CP936.gz'? Or did you just think: there should be a file '/usr/share/i18n/charmaps/CP936.gz' because there's a cp936 option in the kernel?

Hi,mike155,
I didn't get any warning or error messages about a missing file '/usr/share/i18n/charmaps/CP936.gz',that's my guess,sorry I didn't make that clear.

Code: Select all

LANG=zh_CN.UTF-8 xterm

This remind me that I have a line in .Xresources

Code: Select all

xterm*locale:zh_CN.UTF-8

After I deleted this line,the error message is gone,and I still can type and see chinese in xterm.
Curious why the error message didn't show up before.

Code: Select all

Thanks a lot, mike155!

mike155 · Post by **mike155** » Wed Jun 26, 2019 2:06 pm

The more I think about it, the more I believe that "CP936.gz" simply doesn't exist.

Wikipedia writes that there was a lot of confusion about code page 936 and that it was superseded by code pages 54936 (GB 18030) and GBK: https://en.wikipedia.org/wiki/Code_page ... Windows%29

And GB18030.gz and GBK.gz both exist:

Code: Select all

# cd  /usr/share/i18n/charmaps/
# ls -la GB18030.gz GBK.gz 
-rw-r--r-- 1 root root 646644 Jun  6 01:51 GB18030.gz
-rw-r--r-- 1 root root 119347 Jun  6 01:51 GBK.gz

I really hope that those old code pages will go away soon and that the world will agree on Unicode.