[SOLVED] Perl script: run Python env activate script

[Vieri]

Hi,

I'd like a Perl script to properly run a program within a Python virtual environment and retrieve its stdout text.

Something like:

Code: Select all

`source /opt/custom_pyvenvs/program/bin/activate param1 param2'

If I try to run this in Perl I get a "source: No such file or directory" error.

I know source is a BASH built-in command, so can I run the "activate" script directly from the Perl script within a bash shell even though the activate script is supposed to be "sourced"?

Regards,

Vieri

[bstaletic]

Sourcing a bash script in a perl interpreter makes no sense.
On the other hand, replicating what the activate script does is almost trivial:

• Define $VIRTUAL_ENV to be the root directory of the virtual environment
• Prepend $VIRTUAL_ENV/bin to $PATH
• Mess with your $PS1
• Define deactivate function to undo everything.

I'm sure perl has its way of altering the environment.

[Vieri]

Thanks for the feedback.

I do not wish to reimplement what the activate script does.
In any case, the "real" issue at hand is how to properly call a bash script from Perl. I don't see why I can't source a script with the built-in source command in BASH.
In PHP-cli I can do this:

Code: Select all

exec("source /myenv/bin/activate params", $out, $ret);

So in Perl I thought the backticks would do the same. I read somewhere that Perl's backticks would run /bin/sh which in my case points to bash.
So then I tried the following:

Code: Select all

$out = `bash -c "source /myenv/bin/activate params"`;

It worked.
Not sure why, but it worked.

Thanks

[EDIT]
In other words, this works:

Code: Select all

$out = `bash -c "source /myenv/bin/activate params"`;

whereas this does not:

Code: Select all

$out = `source /myenv/bin/activate params`;

[Hu]

Code: Select all

exec("source /myenv/bin/activate params", $out, $ret);

That actually works for your use case, or are you just showing the PHP code equivalent to what your Perl code is trying (and failing) to do? Perhaps PHP always runs a shell, even when it doesn't need one.

Code: Select all

$out = `bash -c "source /myenv/bin/activate params"`;

It worked.
Not sure why, but it worked.

This runs bash, and gives it a shell directive on the command line to run. Contrast that to:

[EDIT]

Code: Select all

$out = `source /myenv/bin/activate params`;

This searches $PATH and runs a program named source, if such a program exists. However, source is not a program; it is a shell builtin. Per perldoc perlop, backticks use a shell if required. For simple cases, no shell is required, so none is used, and shell builtins are therefore not recognized.

[Vieri]

Thanks for the explanation!

For completeness, the PHP code I posted "works for me".
So I guess it's as you say: PHP always runs a shell.

I missed the part in Perl about "backticks use a shell if required". How does it decide if it's required or not, I still don't know.
I'll look into it asap.

Thanks again.

[RumpletonBongworth]

Both the backticks and qx are operators which act as the readpipe function does. The documentation could do a better job of explaining it. The perlop documentation briefly mentions that, as with the system function, "if the string contains no shell metacharacters then it will be executed directly". As such, a better explanation can be found by reading the documentation for system. Consider the following code.

Code: Select all

readpipe('source foo')

What happens is that Perl scans the string and finds no shell metacharacters. Consequently, it acts somewhat similarly to a real shell in so far as it considers the string as whitespace separated words, before trying to execve(2) the first of those words, with the remaining words constituting the argument vector. Since there is no executable named "source" in PATH, the execve(2) syscall fails with ENOENT (which can be observed by proceeding to inspect the special $! variable).

While readpipe can be coerced into always running /bin/sh, going about it in this way is fundamentally incorrect. The reason is that the sh executable should only ever be assumed to provide the features of the Shell Command Language, as defined by POSIX - and not necessarily all of the additional bells and whistles that the bash shell offers. While Gentoo defaults to having /bin/sh be a symlink to bash, it can be changed, and there are many other distributions and platforms that have sh as something else. What's more, if bash finds itself being launched as sh, it changes its behaviour so as to more strictly adhere to the specification. This can sometimes affect the behaviour of code that is specificially written for bash.

To whit, if you need to source shell code that is written for bash, you should make a point of explicitly executing bash itself. Here is one way of going about it in Perl that does not capture any output:

Code: Select all

# Because more than one arg is given, Perl will never execute sh
system('bash', '-c', 'source foo; # maybe do more stuff here ...');

If, on the other hand, you need to capture the standard output, here is how to do it using core Perl (that is, without any third party libraries/modules required):

Code: Select all

my @cmd = ('bash', '-c', 'source foo; # maybe do more stuff here ...');
open my $pipe, '-|', @cmd or die "Couldn't exec $cmd[0]: $!";
my @stdout = readline $pipe; # captures output lines to an array variable
close $pipe; # after closing, $? will equal 0 if $cmd[0] exited without error

The exec() function of PHP always executes sh with the -c operand. While this is sometimes useful, there are many occasions for which it is a nuisance, in which case directly executing a given program - not to mention capture its output - requires more labour in PHP than it does in Perl (for that, see proc_open, whose interface was improved by the release of PHP 7.4).

All that being said, the last time I looked at the code of the activate script generated for Python venvs, I found that the only thing it was doing that was of any great importance was to modify PATH. Well, one can just as easily do that in the execution environment of Perl itself! Below is an example.

Code: Select all

local $ENV{PATH} = "/myenv/bin:$ENV{PATH}";
my @stdout = readpipe('some-cool-bin-in-my-pyenv');

Indeed, the use of the local keyword there has it so that the change to PATH is only effective for the enclosing lexical scope. That can be useful if, say, calling into a subroutine and where it may be considered desirable for the changes to PATH to be effectively undone once the subroutine returns.

EDIT: Clarified a minor detail as to what happens when execve fails for Perl.