| View previous topic :: View next topic |
| Author |
Message |
Vieri
l33t
Joined: 18 Dec 2005
Posts: 894
|
 Posted: Thu May 25, 2023 4:58 pm Post subject: python application not working in non-root environment |
 |
|
Hi,
Sorry for the rookie question.
I installed vosk on Gentoo Linux with:
| Code: | # pip install vosk --user
Collecting vosk
Using cached vosk-0.3.45-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (7.2 MB)
Requirement already satisfied: tqdm in /root/.local/lib/python3.9/site-packages (from vosk) (4.65.0)
Requirement already satisfied: cffi>=1.0 in /usr/lib/python3.9/site-packages (from vosk) (1.15.0)
Requirement already satisfied: srt in /root/.local/lib/python3.9/site-packages (from vosk) (3.5.3)
Requirement already satisfied: requests in /usr/lib/python3.9/site-packages (from vosk) (2.26.0)
Requirement already satisfied: websockets in /root/.local/lib/python3.9/site-packages (from vosk) (11.0.3)
Requirement already satisfied: pycparser in /usr/lib/python3.9/site-packages (from cffi>=1.0->vosk) (2.21)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests->vosk) (1.26.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.9/site-packages (from requests->vosk) (3021.3.16)
Requirement already satisfied: charset_normalizer~=2.0.0 in /usr/lib/python3.9/site-packages (from requests->vosk) (2.0.9)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests->vosk) (3.3)
Requirement already satisfied: setuptools in /usr/lib/python3.9/site-packages (from certifi>=2017.4.17->requests->vosk) (59.8.0)
Installing collected packages: vosk
WARNING: The script vosk-transcriber is installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed vosk-0.3.45
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv |
I can run from console just fine:
| Code: | # /root/.local/bin/vosk-transcriber -l XX -i my.wav -o ./my.txt
|
I get the transcribed text.
I'd like to run the vosk transcriber tool from other instances such as Apache/PHP or Asterisk/AGI.
How do you install a python program and it's dependencies system-wide when there is no ebuild for it?
I don't want it in /root because I won't be able to exec() it from Apache/PHP, etc.
Even if I did give Apache or Asterisk permission to access the binary I'd still get module loading errors such as:
| Code: | from vosk.transcriber.cli import main
ModuleNotFoundError: No module named 'vosk'
|
I have a feeling I'd need to create a custom ebuild for this package.
I'm not sure how hard it is to actually create a python module installer ebuild.
If that's my only route could I have a reference to a guide on how to do this?
This is certainly very detailed but a bit too confusing for a novice: https://projects.gentoo.org/python/guide/distutils.html
Regards,
Vieri |
|
| Back to top |
|
 |
Phoenix591
Guru
Joined: 17 Sep 2007
Posts: 492
|
| Posted: Fri May 26, 2023 1:23 am Post subject: |
|
|
it looks like a bit of a pain to package since its not just a pure python package and needs another library that isn't packaged.
much simpler however, should be just installing it into a virtual environment somewhere, and then pointing apache or whatever to it. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 894
|
| Posted: Fri May 26, 2023 8:09 am Post subject: |
|
|
So I did the following, and I was able to properly run the program from the virtual env.
| Code: | # mkdir /opt/custom_pyvenvs
# cd /opt/custom_pyvenvs
# python -m venv vosk
# source vosk/bin/activate
(vosk) devvm15 /opt/custom_pyvenvs # python -m pip install vosk
Collecting vosk
Using cached vosk-0.3.45-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (7.2 MB)
Collecting cffi>=1.0
Downloading cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
|████████████████████████████████| 441 kB 3.4 MB/s
Collecting requests
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
|████████████████████████████████| 62 kB 1.9 MB/s
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting srt
Using cached srt-3.5.3.tar.gz (28 kB)
Collecting websockets
Using cached websockets-11.0.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)
Collecting pycparser
Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
|████████████████████████████████| 118 kB 62.0 MB/s
Collecting urllib3<3,>=1.21.1
Downloading urllib3-2.0.2-py3-none-any.whl (123 kB)
|████████████████████████████████| 123 kB 64.9 MB/s
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
|████████████████████████████████| 199 kB 50.2 MB/s
Collecting certifi>=2017.4.17
Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
|████████████████████████████████| 156 kB 59.1 MB/s
Collecting idna<4,>=2.5
Downloading idna-3.4-py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 171 kB/s
Using legacy 'setup.py install' for srt, since package 'wheel' is not installed.
Installing collected packages: urllib3, pycparser, idna, charset-normalizer, certifi, websockets, tqdm, srt, requests, cffi, vosk
Running setup.py install for srt ... done
Successfully installed certifi-2023.5.7 cffi-1.15.1 charset-normalizer-3.1.0 idna-3.4 pycparser-2.21 requests-2.31.0 srt-3.5.3 tqdm-4.65.0 urllib3-2.0.2 vosk-0.3.45 websockets-11.0.3
WARNING: You are using pip version 21.2.4; however, version 23.1.2 is available.
You should consider upgrading via the '/opt/custom_pyvenvs/vosk/bin/python -m pip install --upgrade pip' command.
(vosk) devvm15 /opt/custom_pyvenvs # ./vosk/bin/vosk-transcriber -l XX -i /tmp/test.wav
[...]
INFO:root:Execution time: 0.494 sec; xRT 0.154
(vosk) devvm15 /opt/custom_pyvenvs # deactivate |
I needed a way to call the program from the host -- not from the interactive shell within the env.
I didn't know how to do that "cleanly", so I modified the bin/activate BASH script to pass $1 and $2 args to the vosk-transcriber program. So, as root and from my ssh shell I can run:
| Code: | | # source /opt/custom_pyvenvs/vosk/bin/activate LANG /tmp/my_file.wav |
I don't know if there's a better way, but at least it works.
Now for the Apache et al. part...
Here's a simple PHP script that I can run from console as root and works fine:
| Code: | # cat test.php
#!/usr/bin/php -q
<?php
$fileFormat = "wav";
$fileName = "/tmp/myfile";
$language = "XX";
if (file_exists($fileName.".".$fileFormat)) {
unset($exec_out);
exec("source /opt/custom_pyvenvs/vosk/bin/activate ".$language." ".$fileName.".".$fileFormat, $exec_out, $exec_ret);
$textFound = strtolower($exec_out[0]);
echo $textFound." - ".$exec_ret;
print_r($exec_out);
}
?> |
Now, if I remove the shebang and call it as a web page from a client browser the output is empty and the exit code is 0.
I do not see any errors in apache's logs.
It doesn't seem to be a permissions issue (at least not the initial sourcing of the activate script).
What can I try? |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 894
|
| Posted: Fri May 26, 2023 9:16 am Post subject: |
|
|
OK, I solved the issue as the Vosk program was trying to read the language / grammar models in a root-only readable dir.
So I copied the models to a globally readable dir, and this is how I modified the bin/activate BASH script (added this code at the end of it):
| Code: | vosk_language=$1
vosk_file=$2
VOSK_MODEL_PATH="/opt/vosk_models"
export VOSK_MODEL_PATH
${VIRTUAL_ENV}/bin/vosk-transcriber -l ${vosk_language} -i ${vosk_file} 2>/dev/null 3>/dev/null
deactivate |
This "works for me", but is there a better way to do this than to hack the bin/activate script? |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9594
Location: beyond the rim
|
| Posted: Fri May 26, 2023 11:13 am Post subject: |
|
|
Why don't you just use "pip install vosk" ? Without specifying the --user option or any of the virtualenv setup? While the latter are considered better practice as it helps with isolating packages used only for specific tasks, system-wide installation with pip is probably the easier approach in this case.
Alternatively just write a wrapper script that handles the virtualenv stuff instead of modifying the activate script, and call the wrapper from your PHP page. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 894
|
| Posted: Fri May 26, 2023 11:27 am Post subject: |
|
|
I'm not sure I can by-pass this:
| Code: | # pip install vosk
ERROR: (Gentoo) Please run pip with the --user option to avoid breaking python-exec |
As far as the wrapper script is concerned, I'm not sure how to "activate" the virtual environment by "feeding" it the commands I want to run in it.
My wrapper script needs to call "bin/activate" anyway, right?
If it does so, it ends up in a shell.
I don't know how to run:
source ...bin/activate && "run programs within that env and capture output" (without user interaction of course)
[EDIT] Maybe you meant that I duplicate the activate script and customize it to my liking? |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9594
Location: beyond the rim
|
| Posted: Thu Jun 15, 2023 9:54 am Post subject: |
|
|
Ah right, forgot about the python-exec shenanigangs on Gentoo.
Just create a script that contains
| Code: |
#!/bin/bash
source bin/activate
|
And then add any other commands you want to run within the virtual environment after the call to activate. Including anything you need for debugging problems.
For example in one django project at work we have a cronjob for exporting data from the database on a daily basis, which just calls the following script:
| Code: | #!/bin/bash
DIR=$(dirname $0)
source "${DIR}"/../../bin/activate
python ${DIR}/../manage.py shell < ${DIR}/daily-export.py
|
The activate script doesn't do anything special btw, basically it just adjusts some variables (PATH, VIRTUAL_ENV and PS1) and defines the 'deactivate' function to leave the environment (most of which you don't actually need when calling it from PHP). So you might get away with ignoring the activate script and just define PATH and VIRTUAL_ENV properly in your apache/php config if that is easier for you. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
