Setting up VM with virt-install fail UTF-8 error

Message

bacardischmal · Post by **bacardischmal** » Sat Mar 16, 2019 11:08 am

Hello,

i have some VM's on my System and want to add a new one but it Fails with an utf-8 error.

I have tested a minimal Setup:

Code: Select all

Linux-G ~ # qemu-img create -f qcow2 -o preallocation=metadata /srv/virtual/test.qcow2 1G
Formatting '/srv/virtual/test.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
Linux-G ~ # virt-install \
>               --connect qemu:///system \
>               --virt-type kvm \
>               --name demo \
>               --memory 500 \
>               --disk path=/srv/virtual/test.qcow2,format=qcow2,bus=virtio,cache=none \
>               --cdrom /dev/cdrom
ERROR    Fehler: --disk path=/srv/virtual/test.qcow2,format=qcow2,bus=virtio,cache=none: 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte
Linux-G ~ #

Code: Select all

Linux-G ~ # eselect locale list
Available targets for the LANG variable:
  [1]   C
  [2]   de_DE
  [3]   de_DE.iso88591
  [4]   de_DE.utf8
  [5]   en_US
  [6]   en_US.iso88591
  [7]   en_US.utf8
  [8]   POSIX
  [9]   de_DE.UTF-8 *
  [ ]   (free form)
Linux-G ~ # cat /etc/env.d/02locale
# Configuration file for eselect
# This file has been automatically generated.
LANG="de_DE.UTF-8"
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="C"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL="de_DE.UTF-8"

Linux-G ~ # emerge --info
Portage 2.3.62 (python 2.7.15-final-0, default/linux/amd64/17.0/desktop, gcc-7.3.0, glibc-2.27-r6, 4.19.17-gentoo x86_64)
=================================================================
System uname: Linux-4.19.17-gentoo-x86_64-Intel-R-_Core-TM-_i7-6700_CPU_@_3.40GHz-with-gentoo-2.6
KiB Mem: 65816412 total, 415476 free
KiB Swap: 0 total, 0 free
Timestamp of repository gentoo: Sat, 16 Mar 2019 02:00:01 +0000
Head commit of repository gentoo: 4d44c027b9b9074f0b196eb0a69f17e2e05e4582
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash: 4.4_p23-r1::gentoo
dev-lang/perl: 5.26.2::gentoo
dev-lang/python: 2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake: 3.9.6::gentoo
dev-util/pkgconfig: 0.29.2::gentoo
sys-apps/baselayout: 2.6-r1::gentoo
sys-apps/openrc: 0.38.3-r1::gentoo
sys-apps/sandbox: 2.17::gentoo
sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake: 1.15.1-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils: 2.30-r4::gentoo
sys-devel/gcc: 6.4.0-r1::gentoo, 7.3.0-r3::gentoo, 8.2.0-r6::gentoo
sys-devel/gcc-config: 2.0::gentoo
sys-devel/libtool: 2.4.6-r5::gentoo
sys-devel/make: 4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc: 2.27-r6::gentoo
Repositories:

gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
priority: -1000
sync-rsync-verify-jobs: 1
sync-rsync-verify-metamanifest: yes
sync-rsync-extra-opts:
sync-rsync-verify-max-age: 24

gentoo-ddbridge-sources-overlay
location: /var/lib/layman/gentoo-ddbridge-sources-overlay
masters: gentoo
priority: 50

vdr-devel
location: /var/lib/layman/vdr-devel
masters: gentoo
priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="de_DE.UTF-8"
LC_ALL="de_DE.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="de"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="3dnow 3dnowext a52 aac acl acpi ads alsa amd64 audio berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam flac fortran gdbm gif glamor gpm gtk hddtemp ht iconv jpeg kerberos lcms ldap libnotify libtirpc lm_sensors mad mmx mmxext mng mp3 mp4 mpeg multilib ncurses nptl ogg opengl openmp pam pango pcre pdf png policykit ppds qt5 readline samba� sdl seccomp spell sse sse2 sse3 sse4a ssl ssse3 startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis winbind wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="32 64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="intel i915" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Post by **fedeliallalinea** » Sat Mar 16, 2019 11:18 am

You can post also content of /etc/locale.gen file?

bacardischmal · Post by **bacardischmal** » Sat Mar 16, 2019 11:55 am

fedeliallalinea wrote:You can post also content of /etc/locale.gen file?

en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE.UTF-8 UTF-8
de_DE ISO-8859-1

mike155 · Post by **mike155** » Sat Mar 16, 2019 12:11 pm

Please look at the XML file which was generated in /etc/libvirt/qemu. Does it contain an invalid UTF-8 character?

bacardischmal · Post by **bacardischmal** » Sat Mar 16, 2019 12:16 pm

mike155 wrote:Please look at the XML file which was generated in /etc/libvirt/qemu. Does it contain an invalid UTF-8 character?

There is no new file, only the file from the existing VM, you mean in this file?

mike155 · Post by **mike155** » Sat Mar 16, 2019 12:30 pm

0xa0 is a non-blocking space in Latin-1.

How did you enter the virt-install statement? Did you type it manually? Or did you copy it from a web site or from an article/book? Or did you edit the statement in an editor which uses Latin-1 encoding?

The non-blocking space looks like a normal space, but if you copy it to a console which operates in UTF-8 mode, it will generate an error message similar to the one you posted...

bacardischmal · Post by **bacardischmal** » Sat Mar 16, 2019 12:45 pm

mike155 wrote:0xa0 is a non-blocking space in Latin-1.

How did you enter the virt-install statement? Did you type it manually? Or did you copy it from a web site or from an article/book? Or did you edit the statement in an editor which uses Latin-1 encoding?

The non-blocking space looks like a normal space, but if you copy it to a console which operates in UTF-8 mode, it will generate an error message similar to the one you posted...

I copied it from the manpage, but i tried it also to type in the terminal (ssh-session with putty) and local on the System.

Linux-G ~ # virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk none --cdrom /dev/cdrom
ERROR Fehler beim Validieren der Installationsmedien '/dev/cdrom': 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte

mike155 · Post by **mike155** » Sat Mar 16, 2019 2:18 pm

I tried to reproduce the error message, but I can't. Something is wrong on your machine...

It's not a missing locale. If it were a missing locale, virt-install would tell you '/bin/sh: warning: setlocale: LC_ALL: cannot change locale (xxx).'

Just a few random thoughts - which may or may not be related to the problem:

You said you use PuTTY? From Windows? Did you configure 'Remote character set' = 'UTF-8' in the PuTTY client?
Your file /etc/env.d/02locale looks too complicated.
Code: Select all
```
LANG="de_DE.utf8"
LC_COLLATE="C"
```
should be sufficient.
Why does 'eselect locale list' show 'de_DE.utf8' as well as 'de_DE.UTF-8'? On my machine, it shows only 'de_DE.utf8'. UTF-8 is an alias of utf8. Please re-read https://wiki.gentoo.org/wiki/Localization/Guide/en and check your configuration.
emerge --info doesn't show USE flag 'nls'. Why?
Did you compile your own kernel? Did you enable CONFIG_NLS_{UTF8,850,ISO85591_1,ISO85591_15}?

krinn · Post by **krinn** » Sun Mar 17, 2019 11:51 am

I think your problem is how you name your locale.
Look the subtility:

Code: Select all

       The file /etc/locale.gen lists the locales that are to be generated by the locale-gen command.

       Each line is of the form:

       <locale name> <charset>

       Where  <locale  name>  starts  with  a name as found in /usr/share/i18n/locales/.  It must be unique in the file as it is used as the key to locale variables
       (e.g. when you do `export LANG="<locale name>"`).  For default encodings, the <charset> is typically omitted, else it is appended with a "." separator.

It gave you some rules:
* locale should start always with a "known name", taken from /usr/share/i18n/locales/ ; but may end how you wish, with an exception ; when you add . what is next is the default charmap to use.
de_DE is valid, as set in /usr/share/i18n/locales/
de_DELOL is valid, for the same reason (the de_DE exists, then LOL is happend)
de_DE.UTF-8 is valid, you set the local de_DE and force charmap to use to be UTF-8 ; but de_DE.UTF-8 doesn't mean your locale is name "de_DE.UTF-8", the .UTF-8 will not be part of the name, but an indication of the charmap to use.

and you have create "clash" with names:
de_DE.UTF-8 UTF-8 mean: use the locale name "de_DE" (because the locale name could be de_DEWHATEVER, but if you use a dot, you tell it the charmap to use ; so the locale name is not "de_DE.UTF-8" but "de_DE" and use UTF-8 as charmap)
de_DE ISO-8859-1 mean: use the locale name "de_DE" with charmap ISO-8859-1
got it? in both case the locale name is "de_DE"
And because the de_DE first definition is de_DE.UTF-8, the next definition (the de_DE ISO-8859-1) overwrite it
note that you have the same problem with your en_US definition too, both endup as en_US name, but this time the UTF-8 one appears after the ISO-8859-1 one, and so the definition that overwrite the other is the UTF-8 one, a problem you would seen if you were trying to use "en_US" and except ISO-8859-1 to be use while UTF-8 would always.

bacardischmal · Post by **bacardischmal** » Sun Mar 17, 2019 12:38 pm

Hello,

i tried to fix some stuff with the wiki, addet nls and on my kernel but no effect.
i switched UTF-8 to the and now it should override the first right?

Code: Select all

/etc/locale.gen:
en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE ISO-8859-1
de_DE.UTF-8 UTF-8

Linux-G ~ # locale-gen
 * Generating 4 locales (this might take a while) with 8 jobs
 *  (3/4) Generating en_US.ISO-8859-1 ...                                                                                                                                                      [ ok ]
 *  (1/4) Generating de_DE.ISO-8859-1 ...                                                                                                                                                      [ ok ]
 *  (4/4) Generating en_US.UTF-8 ...                                                                                                                                                           [ ok ]
 *  (2/4) Generating de_DE.UTF-8 ...                                                                                                                                                           [ ok ]
 * Generation complete
 * Adding locales to archive ...   
Linux-G ~ # locale -a
C
POSIX
de_DE
de_DE.iso88591
de_DE.utf8
en_US
en_US.iso88591
en_US.utf8
Linux-G ~ # eselect locale list
Available targets for the LANG variable:
  [1]   C
  [2]   POSIX
  [3]   de_DE
  [4]   de_DE.iso88591
  [5]   de_DE.utf8 *
  [6]   en_US
  [7]   en_US.iso88591
  [8]   en_US.utf8
  [ ]   (free form)
Linux-G ~ # eselect locale set 5
Setting LANG to de_DE.utf8 ...
Run ". /etc/profile" to update the variable in your shell.
Linux-G ~ # . /etc/profile
Linux-G ~ # cat /etc/env.d/02locale
# Configuration file for eselect
# This file has been automatically generated.
LANG="de_DE.utf8"
LC_COLLATE="C"

Kernel Parameter:
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_UTF8=y

Code: Select all

Linux-G ~ # emerge --info
Portage 2.3.62 (python 2.7.15-final-0, default/linux/amd64/17.0/desktop, gcc-7.3.0, glibc-2.27-r6, 4.19.29-gentoo x86_64)
=================================================================
System uname: Linux-4.19.29-gentoo-x86_64-Intel-R-_Core-TM-_i7-6700_CPU_@_3.40GHz-with-gentoo-2.6
KiB Mem:    65816412 total,  55324564 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sun, 17 Mar 2019 02:00:01 +0000
Head commit of repository gentoo: f0c1b72aa0182e639d9888b4eabcb12d229de264
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-lang/perl:            5.26.2::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.38.3-r1::gentoo
sys-apps/sandbox:         2.17::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.15.1-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.30-r4::gentoo
sys-devel/gcc:            6.4.0-r1::gentoo, 7.3.0-r3::gentoo, 8.2.0-r6::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r5::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts:
    sync-rsync-verify-max-age: 24

gentoo-ddbridge-sources-overlay
    location: /var/lib/layman/gentoo-ddbridge-sources-overlay
    masters: gentoo
    priority: 50

vdr-devel
    location: /var/lib/layman/vdr-devel
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="de_DE.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="de"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="3dnow 3dnowext a52 aac acl acpi ads alsa amd64 audio berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam flac fortran gdbm gif glamor gpm gtk hddtemp ht iconv jpeg kerberos lcms ldap libnotify libtirpc lm_sensors mad mmx mmxext mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds qt5 readline samba sdl seccomp spell sse sse2 sse3 sse4a ssl ssse3 startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis winbind wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="32 64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="intel i915" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

krinn · Post by **krinn** » Sun Mar 17, 2019 12:55 pm

well, looks good for me now
you can test if the system do handle it right with printf '\u00a3\u00a0\u00ab' ; i suppose a screenshot would had been better (as they might also not appears in your browser too), but here's the result, pound sign, non breaking space and << sign

Code: Select all

£ «

from visual point of view, non breakage space is just a space

if it work, then the problem should be more tied to the virt-install program

mike155 · Post by **mike155** » Sun Mar 17, 2019 1:15 pm

I agree with krinn: your locale settings look good now.

Did you run

Code: Select all

emerge --update --deep --newuse -av @world

after you added USE flag 'nls'?

bacardischmal · Post by **bacardischmal** » Sun Mar 17, 2019 3:58 pm

Yes I did an emerge update after adding nls use-flag.
I think UTF-8 work correct.
Linux-G ~ # printf '\u00a3\u00a0\u00ab'
£ «Linux-G ~ #

but virt-install still doesn't work.

guitou · Post by **guitou** » Mon Mar 18, 2019 12:52 pm

Hello.

Not sure this would apply to your issue as well, but I often happen to mistakenly insert unbreakable spaces in files when writing (typing AltGr+space instead of space), and these may lead to errors (for example in shell scripts, when feeding to shellcheck).
Replacing them with standard space is enough to get things back to normal behaviour.

++
Gi)

bacardischmal · Post by **bacardischmal** » Tue Mar 19, 2019 8:35 pm

guitou wrote:Hello.
Not sure this would apply to your issue as well, but I often happen to mistakenly insert unbreakable spaces in files when writing (typing AltGr+space instead of space), and these may lead to errors (for example in shell scripts, when feeding to shellcheck).
Replacing them with standard space is enough to get things back to normal behaviour.
++
Gi)

I typed it in manually in the terminal to avoid copying Errors.
Here is a debug Output from virt-install maybe it helps

Code: Select all

Linux-G ~ # virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk none --cdrom /dev/cdrom --debug
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (cli:200) Launched with command line: /usr/share/virt-manager/virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk none --cdrom /dev/cdrom --debug
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (virt-install:243) Distilled --network options: ['default']
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (virt-install:159) Distilled --disk options: ['none']
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (cli:214) Requesting libvirt URI qemu:///system
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (cli:217) Received libvirt URI qemu:///system
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (installertreemedia:52) Error validating install location
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtinst/installertreemedia.py", line 48, in validate_path
    dev.path = path
  File "/usr/share/virt-manager/virtinst/devices/disk.py", line 393, in _set_path
    (vol_object, parent_pool) = diskbackend.manage_path(self.conn, newpath)
  File "/usr/share/virt-manager/virtinst/diskbackend.py", line 148, in manage_path
    if not path_is_url(path) and not path_is_network_vol(conn, path):
  File "/usr/share/virt-manager/virtinst/diskbackend.py", line 187, in path_is_network_vol
    for volxml in conn.fetch_all_vols():
  File "/usr/share/virt-manager/virtinst/connection.py", line 233, in fetch_all_vols
    self._fetch_cache[key] = self._fetch_all_vols_raw()
  File "/usr/share/virt-manager/virtinst/connection.py", line 221, in _fetch_all_vols_raw
    ret.extend(self._fetch_vols_raw(poolxmlobj))
  File "/usr/share/virt-manager/virtinst/connection.py", line 208, in _fetch_vols_raw
    self, pool, {}, lambda obj, ignore: obj)
  File "/usr/share/virt-manager/virtinst/pollhelpers.py", line 142, in fetch_volumes
    pool.listAllVolumes, build_func)
  File "/usr/share/virt-manager/virtinst/pollhelpers.py", line 30, in _new_poll_helper
    connkey = obj.name()
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3478, in name
    ret = libvirtmod.virStorageVolGetName(self._o)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (cli:253)   File "/usr/share/virt-manager/virt-install", line 962, in <module>
    fail(main_e)
  File "/usr/share/virt-manager/virtinst/cli.py", line 253, in fail
    logging.debug("".join(traceback.format_stack()))

[Di, 19 Mär 2019 21:30:36 virt-install 23816] ERROR (cli:254) Fehler beim Validieren der Installationsmedien '/dev/cdrom': 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte
[Di, 19 Mär 2019 21:30:36 virt-install 23816] DEBUG (cli:256)
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtinst/installertreemedia.py", line 48, in validate_path
    dev.path = path
  File "/usr/share/virt-manager/virtinst/devices/disk.py", line 393, in _set_path
    (vol_object, parent_pool) = diskbackend.manage_path(self.conn, newpath)
  File "/usr/share/virt-manager/virtinst/diskbackend.py", line 148, in manage_path
    if not path_is_url(path) and not path_is_network_vol(conn, path):
  File "/usr/share/virt-manager/virtinst/diskbackend.py", line 187, in path_is_network_vol
    for volxml in conn.fetch_all_vols():
  File "/usr/share/virt-manager/virtinst/connection.py", line 233, in fetch_all_vols
    self._fetch_cache[key] = self._fetch_all_vols_raw()
  File "/usr/share/virt-manager/virtinst/connection.py", line 221, in _fetch_all_vols_raw
    ret.extend(self._fetch_vols_raw(poolxmlobj))
  File "/usr/share/virt-manager/virtinst/connection.py", line 208, in _fetch_vols_raw
    self, pool, {}, lambda obj, ignore: obj)
  File "/usr/share/virt-manager/virtinst/pollhelpers.py", line 142, in fetch_volumes
    pool.listAllVolumes, build_func)
  File "/usr/share/virt-manager/virtinst/pollhelpers.py", line 30, in _new_poll_helper
    connkey = obj.name()
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3478, in name
    ret = libvirtmod.virStorageVolGetName(self._o)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virt-install", line 955, in <module>
    sys.exit(main())
  File "/usr/share/virt-manager/virt-install", line 943, in main
    guest, installer = build_guest_instance(conn, options)
  File "/usr/share/virt-manager/virt-install", line 537, in build_guest_instance
    installer = build_installer(options, guest)
  File "/usr/share/virt-manager/virt-install", line 496, in build_installer
    install_bootdev=install_bootdev)
  File "/usr/share/virt-manager/virtinst/installer.py", line 64, in __init__
    cdrom = InstallerTreeMedia.validate_path(self.conn, cdrom)
  File "/usr/share/virt-manager/virtinst/installertreemedia.py", line 60, in validate_path
    (str(path), e))
ValueError: Fehler beim Validieren der Installationsmedien '/dev/cdrom': 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte
Linux-G ~ #

guitou · Post by **guitou** » Thu Mar 21, 2019 1:55 pm

Hello.

Taking the risk to tell bullshit again, I would bet that issue lies in your iso/cdrom content, not in your command.

++
Gi)

Post by Hu » Fri Mar 22, 2019 1:46 am

In my opinion, it is always a bug for a Python script to die with a decoding error like this. If the input is ill-formed, the script should state that, describe specifically where it encountered ill-formed input, then either exit or correct the error, depending on whether it is better to limp along on modified data or abort and let the user fix the input properly.

mike155 · Post by **mike155** » Fri Mar 22, 2019 11:11 am

I guess there's a configuration file or a script on your machine that contains an invalid UTF-8 character. Most probably, it is a file that you edited in the past. You were either logged in with wrong locale settings or you copied something from a website or book, which had Latin-1 encoding. Remember that before you read the rest of this post - and ignore any files you haven't changed in the past.

I would use strace:
Code: Select all
```
strace -f -o /tmp/strace-log virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk none --cdrom /dev/cdrom 
```
If you are familiar with strace: open /tmp/strace-log in an editor. Find the location where the error message was emitted (probably near the end of the file). Note down the pid. Then go up and look which files were processed prior to the error (calls to read(), only lines with the same pid). Find the corresponding open() statements. The open() statements will tell you the filenames. Check those files for invalid UTF-8 characters.

If your not familiar with strace, look at the output of
Code: Select all
```
grep "open" /tmp/strace-log
```
The output will tell you all files that were processed. The file you are looking for is most probably near the end of the list.
https://stackoverflow.com/questions/115 ... alid-utf-8 shows you how to check a file for invalid UTF-8 characters. Please note that not all files are supposed to be UTF-8 encoded. Executables, databases, images, videos are NOT UTF-8 encoded. Look only for configuration files or shell scripts that can be edited in an editor.
You could also try to scan your machine for files with invalid UTF-8 characters. Something like
Code: Select all
```
cd /etc
find . -print -type f -exec iconv -f UTF-8 {} -o /dev/null \;
```
will do. Please note that you will get some false alerts (binary or non-UTF-8 files like hwdb.bin, terminfo, etc.)

krinn · Post by **krinn** » Fri Mar 22, 2019 2:31 pm

You know the "when you have remove the impossible..."

Code: Select all

Installation Method options

-c CDROM , --cdrom=CDROM

(that's virt-install manual)

-c space cdrom_device || --cdrom=cdrom_device != --cdrom space cdrom_device

Could be a weak command line validation?

bacardischmal · Post by **bacardischmal** » Sun Mar 24, 2019 4:30 pm

Hello,

that's not only the cdrom Option it ist the same with a disk.

Linux-G /etc # strace -f -o /tmp/strace-log virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk path=/srv/virtual/test.qcow2,format=qcow2,bus=virtio,cache=none
ERROR Fehler: --disk path=/srv/virtual/test.qcow2,format=qcow2,bus=virtio,cache=none: 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte

I'm not very familiar with strace, here is the Output, i can see some python utf paths are not existing, but they are necessary?

filtered:
https://paste.ee/p/zAKOh

unfiltered:
https://paste.ee/p/mMnyU

mike155 · Post by **mike155** » Sun Mar 24, 2019 8:13 pm

Thanks for the strace file.

Please run the commands below. Do you get any errors?

Code: Select all

iconv -f UTF-8 /etc/ld.so.conf -o /dev/null
iconv -f UTF-8 /etc/ssl/openssl.cnf -o /dev/null
iconv -f UTF-8 /proc/mounts -o /dev/null
iconv -f UTF-8 /root/.cache/virt-manager/virt-install.log -o /dev/null
/sbin/ldconfig -p | iconv -f UTF-8 -o /dev/null

I need the lines in the log file ' /root/.cache/virt-manager/virt-install.log' that were generated when you created the strace file. If you did not call virt-install after you created the strace file, you could use the statement below:
Code: Select all
```
tail -n 50  /root/.cache/virt-manager/virt-install.log
```

bacardischmal · Post by **bacardischmal** » Sun Mar 24, 2019 8:51 pm

Hello,

yes I had error's in the virt-install.log, but only from the old entries after removing the old entries its clean

Code: Select all

Linux-G ~ # iconv -f UTF-8 /etc/ld.so.conf -o /dev/null
Linux-G ~ # iconv -f UTF-8 /etc/ssl/openssl.cnf -o /dev/null
Linux-G ~ # iconv -f UTF-8 /proc/mounts -o /dev/null
Linux-G ~ # iconv -f UTF-8 /root/.cache/virt-manager/virt-install.log -o /dev/null
iconv: ungültige Eingabe-Sequenz an der Stelle 10
Linux-G ~ # /sbin/ldconfig -p | iconv -f UTF-8 -o /dev/null
Linux-G ~ #
Linux-G ~ # iconv -f UTF-8 /root/.cache/virt-manager/virt-install.log -o /dev/null
Linux-G ~ #

here are the lines from the logfile:
https://paste.ee/p/SUzVv

mike155 · Post by **mike155** » Sun Mar 24, 2019 9:21 pm

The log file shows

Code: Select all

So, 24 MÃ¤r 2019

Why does it show "MÃ¤r" instead of "Mär"?

Is it just paste.ee?
Do you see it like that in your terminal? Did you set your terminal to UTF-8?
Is already wrong in /root/.cache/virt-manager/virt-install.log? Could you check the bytes in that file, please? "Mär" should be:
Code: Select all
```
4D C3 A4 72  Mär
```

bacardischmal · Post by **bacardischmal** » Sun Mar 24, 2019 9:27 pm

mike155 wrote:The log file shows
Code: Select all
So, 24 MÃ¤r 2019
Why does it show "MÃ¤r" instead of "Mär"? Is it just paste.ee? Or do you see it like that in your terminal? Did you set your terminal to UTF-8?

oh, that's from past.ee i changed the link in the post.

bacardischmal · Post by **bacardischmal** » Thu May 23, 2019 10:21 pm

I still have this problem, does anyone have another idea?