unexpected EOF while looking for matching `}'

[coalms]

I have quite the extended list of COMMON_FLAGS, naturally not following the norm tends to break things
I have found the solution to the following problem

Code: Select all

ebuild.sh, line  136:  Called src_compile
 *   environment, line 2773:  Called meson-multilib_src_compile
 *   environment, line 1651:  Called multilib-minimal_src_compile
 *   environment, line 1870:  Called multilib_foreach_abi 'multilib-minimal_abi_src_compile'
 *   environment, line 2137:  Called multibuild_foreach_variant '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
 *   environment, line 1830:  Called _multibuild_run '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
 *   environment, line 1828:  Called _multilib_multibuild_wrapper 'multilib-minimal_abi_src_compile'
 *   environment, line  532:  Called multilib-minimal_abi_src_compile
 *   environment, line 1864:  Called multilib_src_compile
 *   environment, line 2357:  Called meson_src_compile
 *   environment, line 1719:  Called die

This wouldn't be match of a "help me post" if my solution didn't have any problems
I use the following env for broken packages through /etc/portage/env/fnopie and then passing them to the "broken" packages to package.env

Code: Select all

COMMON_FLAGS="${COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now}"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}
"

The problem starts on emerge where I get spammed

Code: Select all

"/etc/portage/env/fnopie,line 2 unexpected EOF while looking for matching `}"

for each and every one package I disable these flags for, a quick qgrep multilib-minimal_src gives me enough of an understanding of how many packages are going to "break" and by extend pop up unwanted emerge errors

Is there a better way to disable these global make.conf flags per package or /dev/null these errors specifically?

[Hu]

I have quite the extended list of COMMON_FLAGS, naturally not following the norm tends to break things

What exactly did you put in your COMMON_FLAGS that is breaking so many things? The flags you appear to be trying to remove are best done in other ways.

I have found the solution to the following problem

Code: Select all

ebuild.sh, line  136:  Called src_compile
 *   environment, line 2773:  Called meson-multilib_src_compile
 *   environment, line 1651:  Called multilib-minimal_src_compile
 *   environment, line 1870:  Called multilib_foreach_abi 'multilib-minimal_abi_src_compile'
 *   environment, line 2137:  Called multibuild_foreach_variant '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
 *   environment, line 1830:  Called _multibuild_run '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
 *   environment, line 1828:  Called _multilib_multibuild_wrapper 'multilib-minimal_abi_src_compile'
 *   environment, line  532:  Called multilib-minimal_abi_src_compile
 *   environment, line 1864:  Called multilib_src_compile
 *   environment, line 2357:  Called meson_src_compile
 *   environment, line 1719:  Called die

This is the generic "You have failed" error output. It can be caused by any error in compilation of a meson-based project that uses multilib. Your particular failure may be caused by bad CFLAGS, but there are many other possible causes.

This wouldn't be match of a "help me post" if my solution didn't have any problems
I use the following env for broken packages through /etc/portage/env/fnopie and then passing them to the "broken" packages to package.env

Perhaps you should fix the packages instead of disabling these flags for them.

Code: Select all

COMMON_FLAGS="${COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now}"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}
"

Is the interpreter for this file able to handle GNU bash string substitutions? As I read man 5 portage, the shown file will only be directly read by GNU bash for a package named fnopie. From what you wrote, I think you used, but did not show, a package.env directive to read this file. If so, then according to the documentation, the file is interpreted according to the rules for make.conf, not for GNU bash. Putting all that together, I suspect you are passing the shown text as a literal on the command line to every affected package, when you intended to have GNU bash rewrite the COMMON_FLAGS to drop certain fields. If you follow the instructions at failure time and post the full build log, we could confirm or refute that theory.

The problem starts on emerge where I get spammed

Code: Select all

"/etc/portage/env/fnopie,line 2 unexpected EOF while looking for matching `}"

Please show the exact error text and how we can reproduce this. That fnopie,line 2 looks to me like you paraphrased the error.

for each and every one package I disable these flags for, a quick qgrep multilib-minimal_src gives me enough of an understanding of how many packages are going to "break" and by extend pop up unwanted emerge errors

Why is multilib-minimal_src relevant here? Are you speculatively disabling these flags on every package that uses it?

Is there a better way to disable these global make.conf flags per package or /dev/null these errors specifically?

If you just want not to see errors, you can redirect stderr to /dev/null. That will not solve anything though, and the packages will likely still fail.

As an aside, all of these are standard hardening flags. They should be set by using a hardened compiler, not passed on the command line.

[coalms]

Hey, you misunderstand a few things, I will reiterate

As you would expect, I do have a package.env file being

Code: Select all

dev-java/openjdk stat64_musl
sys-libs/libselinux stat64_musl
x11-libs/pango fnopie
media-libs/dav1d fnopie

The COMMON_FLAGS I am trying to remove for selective packages are "-fPIE -pie -Wl,-z,relro,-z,now"
Regarding "multilib-minimal_src", it was a common link between all the failing packages so I tried to emerge any other packages in the database I could find using qmerge and the ones I tested at random all failed

that being said the warnings I am receiving are as soon I start emerging

Code: Select all

These are the packages that would be merged, in order:

Calculating dependencies \"/etc/portage/env/fnopie", line 2: unexpected EOF while looking for matching `}'
 /"/etc/portage/env/fnopie", line 2: unexpected EOF while looking for matching `}'
"/etc/portage/env/

additionally it is not a problem of "why can I NOT remove these flags PER package" since the removal process DOES work, packages I list inside package.env stop complaining and start emerging normaly, the problem is the errors appearing leads me to believe I am doing the removal process the wrong way

I was not planning to "emerge -vauNDU @world > /dev/null" anyhow, maybe I should have phrased it better, the stated warnings appear every time I am trying to emerge related to the packages I have marked with " fnopie in my package.env

Is the interpreter for this file able to handle GNU bash string substitutions? As I read man 5 portage, the shown file will only be directly read by GNU bash for a package named fnopie.

I did not fully understand this one line I would appreciate further explanation

Putting all that together, I suspect you are passing the shown text as a literal on the command line to every affected package, when you intended to have GNU bash rewrite the COMMON_FLAGS to drop certain fields

If you suspect I am typing COMMON_FLAGS="abc....xyz" emerge -vauNDU atom-to-emerge every time then honestly it sounds like you find me kinda boring,no

I do not want to be misunderstood here, the /etc/portage/env/fnopie and /etc/portage/package.env files are parsed "correctly", the actions I am trying to take being cut off a few flags ARE happening successfully, the unreasonable thing to me is the warnings that keep appearing

but for reference my common flags are
COMMON_FLAGS="-march=haswell -O2 -pipe -D_GLIBCXX_ASSERTIONS -fstack-protector-all --param ssp-buffer-size=4 -fstack-clash-protection -D_FORTIFY_SOURCE=3 -fcf-protection=full -ftrivial-auto-var-init=pattern -fPIE -pie -Wl,-z,relro,-z,now -Wl,-z,noexecstack -flto", I do not expect speed I expect hardness

[grknight]

package.env is a portage extension to the PMS that is not very well documented.

It appears Pattern Match substitution is not available for unknown reasons or any Parameter expansion for that matter.
One can do simple full replace or prepend/append but no expansion.

Perhaps someone with intimate knowledge of Portage's bash parsing could give more details.

[flexibeast]

Code: Select all

COMMON_FLAGS="${COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now}"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}
"

A few issues with this:

* Braces, '{' and '}', delimit the start and end of variable names. (In shell, they're usually not required, but it's good practice to use them in order to avoid certain interpolation issus.) So the closing brace, '}', on the first line, should actually be immediately after the 'S' in "COMMON_FLAGS". The above will try to insert the value of a variable named "COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now", and of course there's no such variable.

* i'm not sure what you intended with the '/' just before "-fPIE" in the first line; i presume it's a typo?

* There is a newline before the closing quote of the value for CXXFLAGS, which i assume is not what you intended, and which might cause problems when interpolated in certain contexts.

[coalms]

Code: Select all

COMMON_FLAGS="${COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now}"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}
"

A few issues with this:

* Braces, '{' and '}', delimit the start and end of variable names. (In shell, they're usually not required, but it's good practice to use them in order to avoid certain interpolation issus.) So the closing brace, '}', on the first line, should actually be immediately after the 'S' in "COMMON_FLAGS". The above will try to insert the value of a variable named "COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now", and of course there's no such variable.

* i'm not sure what you intended with the '/' just before "-fPIE" in the first line; i presume it's a typo?

* There is a newline before the closing quote of the value for CXXFLAGS, which i assume is not what you intended, and which might cause problems when interpolated in certain contexts.

you misunderstand, in bash --and since it works here as well in portage make.conf rules-- while inside {} inserting / after a variable name is the same thing as doing echo $variable | sed 's/words to remove//' , otherwise if you set a double // like ${var//exp} its like doing a global sed ie remove every instance of "exp",

what I used works, but I recieve an unecessary error

[grknight]

what I used works, but I recieve an unecessary error

My limited tests show that it may not work as you think it does. For some reason, it removed all CFLAGS set not just the requested ones. Look very close at the logs to verify this.

[pingtoo]

coalms,

Can we try to debug this?

I suggest you modify /etc/portage/env/fnopie as follow

Code: Select all

COMMON_FLAGS="${COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now}"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}
"
#
# Debugging what happen to COMMON_FLAGS
#
if [ "${EBUILD_PHASE}" == "setup" ]; then
  echo "COMMON_FLAGS=${COMMON_FLAGS}"
  echo "CFLAGS=${CFLAGS}"
fi

I guess the COMMON_FLAGS line should be

Code: Select all

COMMON_FLAGS="${COMMON_FLAGS/-fPIE -pie -Wl,-z,relro,-z,now/}"

Just append a '/' after 'now'

[steve_v]

Can we try to debug this?

Watching to see if there's actually a logical solution to this, but FWIW I've just been there with the "debug" bit.

Short version: AFAICT @grknight is correct, and package.env is clearly not being parsed directly by bash, as none of FOO="${BAR/substring}", FOO="${BAR/substring/}" or FOO="${BAR/${BAZ}/}" work, nor do any of the other string manipulation operations one would expect.

This:

Code: Select all

if [ "${EBUILD_PHASE}" == "setup" ]; then
  echo "COMMON_FLAGS=${COMMON_FLAGS}"
  echo "CFLAGS=${CFLAGS}"
fi

also blows up with a nice cryptic:

Code: Select all

Invalid token '[' (not '=')

despite working just fine in bash.

I assume you lifted that snippet from wiki:/etc/portage/bashrc, where it does work, despite wiki:/etc/portage/packge.env saying:

See /etc/portage/bashrc for more information on what can be done in these files.

which implies they share syntax and functionality.

I take from this that /etc/portage/bashrc is parsed by bash as the name implies (when exactly, and what can we do besides "be careful" because "portage doesn't provide a complete list of contradictions"? ), and /etc/portage/package.env (along with make.conf apparently) is parsed by sasquatch... err, some poorly documented python with a syntax that looks just close enough to bash to provide a cunning trap for the unwary.

Presumably this all is why portage-bashrc-mv exists... Though one does wonder why those features are not in portage proper.

[Zucca]

To my knowledge the env files portage uses are only just env files.
I think it follows quite the same rules as pam_env files. Take a look inside of /etc/env.d for example.
You can reference other variables, but not anything fancy like ${var%*.ext}.

[pingtoo]

Can we try to debug this?

Watching to see if there's actually a logical solution to this, but FWIW I've just been there with the "debug" bit.

Short version: AFAICT @grknight is correct, and package.env is clearly not being parsed directly by bash, as none of FOO="${BAR/substring}", FOO="${BAR/substring/}" or FOO="${BAR/${BAZ}/}" work, nor do any of the other string manipulation operations one would expect.

This:

Code: Select all

if [ "${EBUILD_PHASE}" == "setup" ]; then
  echo "COMMON_FLAGS=${COMMON_FLAGS}"
  echo "CFLAGS=${CFLAGS}"
fi

also blows up with a nice cryptic:

Code: Select all

Invalid token '[' (not '=')

despite working just fine in bash.

I assume you lifted that snippet from wiki:/etc/portage/bashrc, where it does work, despite wiki:/etc/portage/packge.env saying:

See /etc/portage/bashrc for more information on what can be done in these files.

which implies they share syntax and functionality.

I take from this that /etc/portage/bashrc is parsed by bash as the name implies (when exactly, and what can we do besides "be careful" because "portage doesn't provide a complete list of contradictions"? ), and /etc/portage/package.env (along with make.conf apparently) is parsed by sasquatch... err, some poorly documented python with a syntax that looks just close enough to bash to provide a cunning trap for the unwary.

Presumably this all is why portage-bashrc-mv exists... Though one does wonder why those features are not in portage proper.

Thank you for testing it out, at least we know what is not working.

Something new learned today.

[Hu]

I assume you lifted that snippet from wiki:/etc/portage/bashrc, where it does work, despite wiki:/etc/portage/packge.env saying:

See /etc/portage/bashrc for more information on what can be done in these files.

which implies they share syntax and functionality.

They do, but you are quoting out of context.

Amend an ebuild function

A package-specific file in the /etc/portage/env directory could help modifying an ebuild function

Code: Select all

...

...

See /etc/portage/bashrc for more information on what can be done in these files.

Files named after packages, as in the example file on the Wiki, are parsed as bash. Files read because package.env mentions them are parsed by Portage, and have the much more limited syntax.

Yes, it is a bit confusing that /etc/portage/env is both the directory where Portage searches for files named after packages and the implicit prefix of any files mentioned in /etc/portage/package.env.

[steve_v]

To my knowledge the env files portage uses are only just env files.
I think it follows quite the same rules as pam_env files.

On closer reading, apparently it depends... In a confusing, inconvenient (for purposes of myself and the OP) and extremely easy to overlook given the way the wiki page is structured manner:

To have a file called when emerging a specific package, it should be named following the pattern "/etc/portage/env/<category>/<package_name>" (versions can be included, see portage man page), the contents being as in /etc/portage/bashrc. These files can hook into specific phases of the emerge process.

If all that is needed is to set environment variables, use a free-form file name directly in /etc/portage/env, then add a line in /etc/portage/package.env with a package atom followed by the chosen file name

So if we want something sourced during emerge (in bash syntax), we must create a directory structure in /etc/portage/env/ for each individual package, and if we want to have multiple packages source the same file without it being global (i.e. /etc/portage/bashrc), we have to duplicate it in multiple directory structures or screw around with symlinks?
Seriously? There's no mechanism to have

• source [file containing shell fragment]? And the interpreter used depends on the directory structure?
  
  Was this designed to be as irritating as possible or what? What happens if you name a file /etc/portage/env/<category>/<package_name>, then reference it from /etc/portage/package.env as well? Is it a bashrc or an env file, or does it only settle into a fixed state when observed?
  
  ed.

  Yes, it is a bit confusing that /etc/portage/env is both the directory where Portage searches for files named after packages and the implicit prefix of any files mentioned in /etc/portage/package.env.

  As I independently realiased on more careful scrutiny of that example, and was unforgivably slow in typing. Comments on ease of overlooking the distinction and confusing design stand. Why are we mixing files with completely different syntax and purpose in the same directory and wiki entry to begin with?
  
  More to the point, how do I do s/foo/bar/g on an environment variable like CFLAGS, for an arbitrary list of packages, as is the obvious and logical course when dealing with e.g. the current lto-by-default transition or needing to remove a single flag from a long default for a few select packages?

[Zucca]

Yes, it is a bit confusing that /etc/portage/env is both the directory where Portage searches for files named after packages and the implicit prefix of any files mentioned in /etc/portage/package.env.

Just a few days ago I read that same. I thought it could be a mistake on the wiki page, although I haven't tested this.

In the past some users have asked if it was possible to patch ebuilds (instead of the package sources) without crating a complete overlay/repository. The answer given back then was simple 'no'. So according to the wiki it is, in fact, possible.
I wonder if this is a new feature...

[Hu]

I have a bashrc-style override in /etc/portage/env dated from 2011, so I think this is not new.

If I recall the Portage manual (not the Wiki), these paths are correct.

The issue with users who want to patch ebuilds is that you can override an entire phase, but you cannot add or remove a line in the middle of a phase. For a complex or messy ebuild, that means copying a large volume of text into the override area, at which point an overlay would be cleaner.

[mark4]

I bet this is a direct result of using a zig zag bracing style. C is not python, braces are required on all blocks and they should be vertically aligned with one another and therefore visually associated with one another. putting an opening brace on the end of one line and the closing brace at the start of the line has always IMHO been a recipe for absolute disaster, socially with the way people keep nesting indentations to the umpteenth level.

void foo(void
{
....code here
....{
........code here
....}
}

the above gives a direct visual association between the opening and closing brace. yes, it increases the vertical size of a function but your eyes scan vertically MUCH easier than they scan horizontally, this is why newspapers publish their articles in tall thin columns. with this style you are not relying on spaces to show where a block starts and ends you can instantly see where a block starts and ends and can instantly see that the blocks and the braces are in perfect agreement with one another.

when you zig zag your braces you lose that visual verification that a block IS where it seems to be in the logic of the code.

[Genone]

Files referred to by package.env are processed by the python part of portage, treated like make.conf and are parsed by the internal getconfig() function, which uses pythons shlex parser module. The downside is that this only supports a very limited subset of sh syntax, the upside is that modifications can influence the dependency generation and performance is better
Files matching the cat/pkg pattern in the /env directory on the other hand are processed by the bash part of portage (they are literally sourced by bash) and therefore can use the full bash syntax. The downside is that the bash side is generally not invoked for dependency generation, only when the ebuild itself is actually processed. So it is fine for changing things like CFLAGS, but not for anything affecting ebuild metadata (e.g. USE and related).

This is also documented in portage(5):

package.env
Per-package environment variable settings. Entries refer to environment files that are placed in the /etc/portage/env/ directory and have the same format as make.conf(5). Note that these files are interpreted much earlier than the package-specific bashrc files which are described in a later section about /etc/portage/env/. Beginners should be careful to recognize the difference between these two types of files. When environment variable settings are all that's needed, package.env is the recommended approach to use.

I don't know why/how both of these file types ended up in the same directory structure.

What happens if you name a file /etc/portage/env/<category>/<package_name>, then reference it from /etc/portage/package.env as well? Is it a bashrc or an env file, or does it only settle into a fixed state when observed?

Then both mechanics will attempt to parse it. They are completely unrelated logic paths intended for different use cases. As said, I don't know why they ended up using the same location. I think the /env file parsing by bash was originally a bashrc hack that was incorporated into portage itself at some point to overcome the limitations of the package.env implementation, but that's just a guess.

Long story short: If you want to use any syntax that is not supported by shlex you cannot use package.env. Either you'll have to rely on the cat/pkg naming mechanism, or craft a bashrc lookup mechanism.

In the past some users have asked if it was possible to patch ebuilds (instead of the package sources) without crating a complete overlay/repository. The answer given back then was simple 'no'. So according to the wiki it is, in fact, possible.
I wonder if this is a new feature...

Not really new, for better or worse it basically was possible since bashrc support was added (e.g. before elog was implemented, people used a bashrc hack to rewrite ebuild messaging functions for logging). However these days there are certain restrictions in place what you can do in bashrc to avoid inconsistent states. So the answer as in most cases is "it depends".

[steve_v]

I don't know why/how both of these file types ended up in the same directory structure.
...
They are completely unrelated logic paths intended for different use cases.

That's the root cause of all this confusion, and IMO it makes no sense at all - logically package env should be in package.env, and package bashrc should be in package.bashrc.
Manual notwithstanding, storing those files in the same location is asking for people to mix them up.

At any rate, thanks for the clarification.

[Genone]

I don't know why/how both of these file types ended up in the same directory structure.
...
They are completely unrelated logic paths intended for different use cases.

That's the root cause of all this confusion, and IMO it makes no sense at all - logically package env should be in package.env, and package bashrc should be in package.bashrc.
Manual notwithstanding, storing those files in the same location is asking for people to mix them up.

Strangely enough while looking into this I found that there actually is package.bashrc support that will load files from /bashrc, but that only works for profile directories (it should work in /etc/portage/profile/, but no guarantees). Profiles don't have package.env or an /env directory though.

Going though the original bug report for per-package env settings it is as I suspected: Someone implemented the bash variant outside portage, then someone else implemented package.env in portage. And when the bash version was later merged into portage people stuck to the original paths for compatibility reasons leading to the current mess. And trust me, changing such things when they're already in use doesn't go well at all with users (still surprised that there was almost no uproar when package.keywords was renamed to package.accept_keywords).

Fortunately it seems very few people try to actually combine package.env and full bash syntax.