Setting up VM with virt-install fail UTF-8 error

[bacardischmal]

Hello,

i have some VM's on my System and want to add a new one but it Fails with an utf-8 error.

I have tested a minimal Setup:

[fedeliallalinea]

You can post also content of /etc/locale.gen file?
_________________
Questions are guaranteed in life; Answers aren't.

[bacardischmal]

[mike155]

Please look at the XML file which was generated in /etc/libvirt/qemu. Does it contain an invalid UTF-8 character?

[bacardischmal]

[mike155]

0xa0 is a non-blocking space in Latin-1.

How did you enter the virt-install statement? Did you type it manually? Or did you copy it from a web site or from an article/book? Or did you edit the statement in an editor which uses Latin-1 encoding?

The non-blocking space looks like a normal space, but if you copy it to a console which operates in UTF-8 mode, it will generate an error message similar to the one you posted...

[bacardischmal]

[mike155]

I tried to reproduce the error message, but I can't. Something is wrong on your machine...

It's not a missing locale. If it were a missing locale, virt-install would tell you '/bin/sh: warning: setlocale: LC_ALL: cannot change locale (xxx).'

Just a few random thoughts - which may or may not be related to the problem:

1. You said you use PuTTY? From Windows? Did you configure 'Remote character set' = 'UTF-8' in the PuTTY client?
   
   
2. Your file /etc/env.d/02locale looks too complicated.
   

   Code:

   LANG="de_DE.utf8" LC_COLLATE="C"

   
   should be sufficient.
   
   
3. Why does 'eselect locale list' show 'de_DE.utf8' as well as 'de_DE.UTF-8'? On my machine, it shows only 'de_DE.utf8'. UTF-8 is an alias of utf8. Please re-read https://wiki.gentoo.org/wiki/Localization/Guide/en and check your configuration.
   
   
4. emerge --info doesn't show USE flag 'nls'. Why?
   
   
5. Did you compile your own kernel? Did you enable CONFIG_NLS_{UTF8,850,ISO85591_1,ISO85591_15}?
   

[krinn]

I think your problem is how you name your locale.
Look the subtility:

[bacardischmal]

Hello,

i tried to fix some stuff with the wiki, addet nls and on my kernel but no effect.
i switched UTF-8 to the and now it should override the first right?

[krinn]

well, looks good for me now
you can test if the system do handle it right with printf '\u00a3\u00a0\u00ab' ; i suppose a screenshot would had been better (as they might also not appears in your browser too), but here's the result, pound sign, non breaking space and << sign

[mike155]

I agree with krinn: your locale settings look good now.

Did you run

[bacardischmal]

Yes I did an emerge update after adding nls use-flag.
I think UTF-8 work correct.
Linux-G ~ # printf '\u00a3\u00a0\u00ab'
£ «Linux-G ~ #

but virt-install still doesn't work.

[guitou]

Hello.

Not sure this would apply to your issue as well, but I often happen to mistakenly insert unbreakable spaces in files when writing (typing AltGr+space instead of space), and these may lead to errors (for example in shell scripts, when feeding to shellcheck).
Replacing them with standard space is enough to get things back to normal behaviour.

++
Gi)

[bacardischmal]

[guitou]

Hello.

Taking the risk to tell bullshit again, I would bet that issue lies in your iso/cdrom content, not in your command.

++
Gi)

[Hu]

In my opinion, it is always a bug for a Python script to die with a decoding error like this. If the input is ill-formed, the script should state that, describe specifically where it encountered ill-formed input, then either exit or correct the error, depending on whether it is better to limp along on modified data or abort and let the user fix the input properly.

[mike155]

I guess there's a configuration file or a script on your machine that contains an invalid UTF-8 character. Most probably, it is a file that you edited in the past. You were either logged in with wrong locale settings or you copied something from a website or book, which had Latin-1 encoding. Remember that before you read the rest of this post - and ignore any files you haven't changed in the past.

1. I would use strace:
   

   Code:

   strace -f -o /tmp/strace-log virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk none --cdrom /dev/cdrom

   
   If you are familiar with strace: open /tmp/strace-log in an editor. Find the location where the error message was emitted (probably near the end of the file). Note down the pid. Then go up and look which files were processed prior to the error (calls to read(), only lines with the same pid). Find the corresponding open() statements. The open() statements will tell you the filenames. Check those files for invalid UTF-8 characters.
   
   If your not familiar with strace, look at the output of
   

   Code:

   grep "open" /tmp/strace-log

   
   The output will tell you all files that were processed. The file you are looking for is most probably near the end of the list.
   
   
2. https://stackoverflow.com/questions/115210/how-to-check-whether-a-file-is-valid-utf-8 shows you how to check a file for invalid UTF-8 characters. Please note that not all files are supposed to be UTF-8 encoded. Executables, databases, images, videos are NOT UTF-8 encoded. Look only for configuration files or shell scripts that can be edited in an editor.
   
   
3. You could also try to scan your machine for files with invalid UTF-8 characters. Something like
   

   Code:

   cd /etc find . -print -type f -exec iconv -f UTF-8 {} -o /dev/null \;

   
   will do. Please note that you will get some false alerts (binary or non-UTF-8 files like hwdb.bin, terminfo, etc.)
   

[krinn]

You know the "when you have remove the impossible..."

[bacardischmal]

Hello,

that's not only the cdrom Option it ist the same with a disk.

Linux-G /etc # strace -f -o /tmp/strace-log virt-install --connect qemu:///system --virt-type kvm --name demo --memory 500 --disk path=/srv/virtual/test.qcow2,format=qcow2,bus=virtio,cache=none
ERROR Fehler: --disk path=/srv/virtual/test.qcow2,format=qcow2,bus=virtio,cache=none: 'utf-8' codec can't decode byte 0xa0 in position 28: invalid start byte

I'm not very familiar with strace, here is the Output, i can see some python utf paths are not existing, but they are necessary?

filtered:
https://paste.ee/p/zAKOh

unfiltered:
https://paste.ee/p/mMnyU

[mike155]

Thanks for the strace file.

1. Please run the commands below. Do you get any errors?
   

   Code:

   iconv -f UTF-8 /etc/ld.so.conf -o /dev/null iconv -f UTF-8 /etc/ssl/openssl.cnf -o /dev/null iconv -f UTF-8 /proc/mounts -o /dev/null iconv -f UTF-8 /root/.cache/virt-manager/virt-install.log -o /dev/null /sbin/ldconfig -p | iconv -f UTF-8 -o /dev/null

   
   
2. I need the lines in the log file ' /root/.cache/virt-manager/virt-install.log' that were generated when you created the strace file. If you did not call virt-install after you created the strace file, you could use the statement below:
   

   Code:

   tail -n 50  /root/.cache/virt-manager/virt-install.log

   
   

[bacardischmal]

Hello,

yes I had error's in the virt-install.log, but only from the old entries after removing the old entries its clean

[mike155]

The log file shows

[bacardischmal]

[bacardischmal]

I still have this problem, does anyone have another idea?