Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[HOWTO] Obtain some safe CPU flags
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
frankenputer
n00b
n00b


Joined: 09 Mar 2016
Posts: 26

PostPosted: Wed Jul 13, 2016 8:00 pm    Post subject: [HOWTO] Obtain some safe CPU flags Reply with quote

Updated the wiki to include the code below which allows the user to obtain some safe CPU flags.

Copy 'n paste ready.

Code:
gcc '-###' -march=native -x c - 2>&1 | gawk -lfilefuncs '{ gsub(/\"/,""); for (x=1; x < NF; x++) { if (match($x,/march|cache/)) { cflagz[$x]++; } else if (match($x,/mpopcnt|m3dnow|msse|mssse|maes/)) { x86[substr($x,3)]++; } else if (match($x,/mcx16|mabm|mlzcnt|msahf/)) { x86[substr($x,2)]++; } } } function concatMe(arr) { ret=""; for (x in arr) { str=(match(x,/cache/) ? "--param"" "x : x); ret=ret" "str; } return ret; } END { PROCINFO["sorted_in"]="@ind_str_asc"; cpuf="/proc/cpuinfo"; if (0 == stat(cpuf, buf)) { while (0 != (getline cur_line < cpuf)) { if (match(cur_line, "flags")) { split(cur_line, arr, " "); if (0 != arr_len=length(arr)) { for (x=1; x < arr_len; x++) { if (match(arr[x],/mmxext|3dnowext/)) { x86[arr[x]]++; } } } break; } } close(cpuf); } if (0 != length(x86)) { printf("CPU_FLAGS_X86=\"...%s\"\n",concatMe(x86)); } if (0 != length(cflagz)) { printf("CFLAGS=\"...%s\"\n",concatMe(cflagz)); } }'


The script used to be ~ 5 lines long at the beginning.

Filename cpuorakle.awk

Code:
#!/usr/bin/awk -f
@load "filefuncs";
{
  gsub(/\"/,"");
  for (x=1; x < NF; x++) {
    if (match($x,/march|cache/)) {
      cflagz[$x]++;
    } else if (match($x,/mpopcnt|m3dnow|msse|mssse|maes/)) {
      x86[substr($x,3)]++;
    } else if (match($x,/mcx16|mabm|mlzcnt|msahf/)) {
      x86[substr($x,2)]++;
    }
  }
}
function concatMe(arr) {
  ret="";
  for (x in arr) {
    str=(match(x,/cache/) ? "--param"" "x : x);
    ret=ret" "str;
  }
  return ret;
}
END {
  PROCINFO["sorted_in"]="@ind_str_asc";
  cpuf="/proc/cpuinfo";
  if (0 == stat(cpuf, buf)) {
    while (0 != (getline cur_line < cpuf)) {
      if (match(cur_line, "flags")) {
        split(cur_line, arr, " ");
        if (0 != arr_len=length(arr)) {
          for (x=1; x < arr_len; x++) {
            if (match(arr[x],/mmxext|3dnowext/)) {
              x86[arr[x]]++;
            }
          }
        }
        break;
      }
    }
    close(cpuf);
  }

  if (0 != length(x86)) {
    printf("CPU_FLAGS_X86=\"...%s\"\n",concatMe(x86));
  }

  if (0 != length(cflagz)) {
    printf("CFLAGS=\"...%s\"\n",concatMe(cflagz));
  }
}


Execute it with:

Code:
chmod +x cpuorakle.awk
gcc '-###' -march=native -x c - 2>&1 | ./cpuorakle.awk


Sample output:

Code:
CPU_FLAGS_X86="... 3dnow 3dnowext mabm mcx16 mlzcnt mmxext msahf popcnt sse sse2 sse3 sse4a"
CFLAGS="... -march=amdfam10 --param l1-cache-line-size=64 --param l1-cache-size=64 --param l2-cache-size=512"


---

Synced the code with the upstream repo.


Last edited by frankenputer on Sun Jul 17, 2016 3:02 am; edited 8 times in total
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 3615
Location: UK

PostPosted: Thu Jul 14, 2016 2:59 am    Post subject: Reply with quote

I have an ugly script that generates human-readable output for CPU_FLAGS_X86, may be of use to someone:

Code:
#!/bin/sh
flags_path="$(portageq get_repo_path / gentoo)/profiles/desc/cpu_flags_x86.desc"
flags_regex='('$(sed -n '/^flags/ { s/^.*:\s//; y/ /|/; p; q }' /proc/cpuinfo)')'

awk -F' - ' '!/^#/ { printf "%-10s %s\n", $1, $2 }' "$flags_path" \
    | egrep --color "(^|\[)$flags_regex\W"

_________________
Quantity is not quality.
overlay | runit-scripts
Back to top
View user's profile Send private message
frankenputer
n00b
n00b


Joined: 09 Mar 2016
Posts: 26

PostPosted: Thu Jul 14, 2016 2:44 pm    Post subject: Reply with quote

Ant P. wrote:
I have an ugly script that generates human-readable output for CPU_FLAGS_X86, may be of use to someone:

Code:
#!/bin/sh
flags_path="$(portageq get_repo_path / gentoo)/profiles/desc/cpu_flags_x86.desc"
flags_regex='('$(sed -n '/^flags/ { s/^.*:\s//; y/ /|/; p; q }' /proc/cpuinfo)')'

awk -F' - ' '!/^#/ { printf "%-10s %s\n", $1, $2 }' "$flags_path" \
    | egrep --color "(^|\[)$flags_regex\W"


Thank you, added /proc/cpuinfo parsing.
Back to top
View user's profile Send private message
Randy Andy
Veteran
Veteran


Joined: 19 Jun 2007
Posts: 1103
Location: /dev/koelsch

PostPosted: Thu Jul 14, 2016 5:11 pm    Post subject: Safe CFLAGS and CPU Flags without reinventing the wheel Reply with quote

Hi frankenputer,

your sample output contains lots of faults regarding the CPU_FLAGS_X86="" line.
It should only contain CPU Flags which also exist as USE-Flags.

But you don't have to fiddle this out by your own, if you would like to read the news message from:

Quote:
2015-01-28-cpu_flags_x86-introduction
and follow the instruction, except on little detail. :wink:
Since yesterday the package has been renamed to:
Quote:
app-portage/cpuid2cpuflags


Would be great if someone could release a new message regarding the renaming:
https://packages.gentoo.org/packages/app-portage/cpuid2cpuflags
and also correct the wiki asap:
https://wiki.gentoo.org/wiki/CPU_FLAGS_X86

Regarding dertermine safe CFLAGS:
Inspired by a lecture, I gave at the workshop here (sorry, only german language):
https://forums.gentoo.org/viewtopic-t-1017200-highlight-trolug.html
Sebastian Pipping wrote a nice python-script which does the job quite well.
The package is in the tree since almost a year and is called:
app-misc/resolve-march-native
After installation you can call it e.g. with:
Code:
resolve-march-native -a

and take over the line to you CFLAGS="" line.

A hint to this from the
https://wiki.gentoo.org/wiki/Safe_CFLAGS would be also a great idea, cause it make things a lot easier for the user.

Best regards, Andy.
_________________
If you want to see a Distro done right, compile it yourself!
Back to top
View user's profile Send private message
frankenputer
n00b
n00b


Joined: 09 Mar 2016
Posts: 26

PostPosted: Thu Jul 14, 2016 7:16 pm    Post subject: Re: Safe CFLAGS and CPU Flags without reinventing the wheel Reply with quote

Randy Andy wrote:
Hi frankenputer,
your sample output contains lots of faults regarding the CPU_FLAGS_X86="" line.
It should only contain CPU Flags which also exist as USE-Flags.


Hi Andy,

The Mihai's program detects avx512* flags which doesn't exists as USE flags at this point.
Including those existing cpu instructions that does not exists as CPU_FLAGS_X86 USE flags won't break the system.
In case they are added upstream, all users that have avx512* flags in their configs will be able to leverage them.

---

Quote:
But you don't have to fiddle this out by your own, if you would like to read the news message from:


Running the Mihai's program:

Code:
cpuinfo2cpuflags-x86
CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a"


My gawk kung fu:

Code:
CPU_FLAGS_X86="... 3dnow 3dnowext mabm mcx16 mlzcnt mmxext msahf popcnt sse sse2 sse3 sse4a "


Compare both CPU_FLAGS_X86 lines.

mmx is included in vanilla make.conf

---

Quote:
Sebastian Pipping wrote a nice python-script which does the job quite well.
The package is in the tree since almost a year and is called:
app-misc/resolve-march-native
After installation you can call it e.g. with:
Code:
resolve-march-native -a

and take over the line to you CFLAGS="" line.


Running the program:

Code:
resolve-march-native -a
-march=amdfam10 --param l1-cache-line-size=64 -O2 -pipe


My gawk kung fu:

Code:
CFLAGS="... -march=amdfam10 --param l1-cache-line-size=64 --param l1-cache-size=64 --param l2-cache-size=512 "


Compare both CFLAGS lines.

-O2 -pipe is included in vanilla make.conf

---

I'm not sure how stable avx* flags are, since I don't have intel cpu. This along with the following quote is the reason not to include them in first place.

Quote:
GCC depresses SSEx instructions when -mavx is used. Instead, it generates new AVX instructions or AVX equivalence for all SSEx instructions when needed.


At the end of the day the user has choice, either mine gawk kung fu, Mihai's program or the Sebastian's one or the Ant P. one.

I appreciate negative and positive feedback, but can't stand next to "jump to conclusions" ones.
Back to top
View user's profile Send private message
Randy Andy
Veteran
Veteran


Joined: 19 Jun 2007
Posts: 1103
Location: /dev/koelsch

PostPosted: Sun Jul 17, 2016 3:28 pm    Post subject: Re: Safe CFLAGS and CPU Flags without reinventing the wheel Reply with quote

Hi frankenputer,

frankenputer wrote:

The Mihai's program detects avx512* flags which doesn't exists as USE flags at this point.
Including those existing cpu instructions that does not exists as CPU_FLAGS_X86 USE flags won't break the system.
In case they are added upstream, all users that have avx512* flags in their configs will be able to leverage them.

Right, when your prediction once occurs.

---

frankenputer wrote:

Running the Mihai's program:

Code:
cpuinfo2cpuflags-x86
CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a"


My gawk kung fu:

Code:
CPU_FLAGS_X86="... 3dnow 3dnowext mabm mcx16 mlzcnt mmxext msahf popcnt sse sse2 sse3 sse4a "


Compare both CPU_FLAGS_X86 lines.

mmx is included in vanilla make.conf


Only cause mmx is named into the vanilla make.conf, does not guaranty that it would be thrown away by the average user.
You leave it out by your kung fu, which mean that packages which makes uses of it, doesn't compile this functionality into, although you set the superior grade mmext flag.
Much more packages makes usage of mmx compared to mmext, just compare the output of:
Code:
equery h cpu_flags_x86_mmx
vs.
Code:
equery h cpu_flags_x86_mmxext


Similar for your mabm mcx16 mlzcnt msahf, these doen't exist as cpu_flags_x86 and probably never would.
Theses flags belongs to the CFLAGS, if there's a need activate (-m) or deactivate (-mno) it there.

---

frankenputer wrote:


Running the program:

Code:
resolve-march-native -a
-march=amdfam10 --param l1-cache-line-size=64 -O2 -pipe


My gawk kung fu:

Code:
CFLAGS="... -march=amdfam10 --param l1-cache-line-size=64 --param l1-cache-size=64 --param l2-cache-size=512 "


Compare both CFLAGS lines.

-O2 -pipe is included in vanilla make.conf

Regarding resolve-march-native -a

The option -a is only for:
Quote:


--add-recommended, -a
add recommended flags

to give the user an idea what makes sense to add to his line.

Your claim here the 2 missing cache-size parameters, but since I found a package (openmpi), which doesn't compile successfully here on my westmere Xeon cpu if I use
the l1-cache-size= parameter, I'm not sure if leaving out the cache parameters is good or bad. How big would be the difference regarding performance? I never tested it.
In the sense of giving safe CFLAGS, it seems to be better to leave it out to me. To present safe CFLAGS is lastly the intention of this little helper script.

The power user (or worser the ricer) hast to decide by himself, how to optimize the settings stronger, but that's not what the title of such wikis reflect.

---

frankenputer wrote:


At the end of the day the user has choice, either mine gawk kung fu, Mihai's program or the Sebastian's one or the Ant P. one.

I appreciate negative and positive feedback, but can't stand next to "jump to conclusions" ones.

Only to ask GCCs CPU recognition fails sometimes, there are lots of bug reports which shows that (references at the bottom).
That's why the Safe_CFLAGS Wiki and Sebastians program tries a different approach, compiling two files with different settings and compare it afterwards, to filter out wrongly recognized flags.

So far my feedback, but I agree, lastly the users decide what to use.

Regards, Andy.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39851
https://gcc.gnu.org/ml/gcc-help/2009-04/msg00301.html
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=43718
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39851
https://gcc.gnu.org/ml/gcc-help/2009-04/msg00305.html
_________________
If you want to see a Distro done right, compile it yourself!
Back to top
View user's profile Send private message
frankenputer
n00b
n00b


Joined: 09 Mar 2016
Posts: 26

PostPosted: Sun Jul 17, 2016 5:15 pm    Post subject: Reply with quote

Quote:
Only cause mmx is named into the vanilla make.conf, does not guaranty that it would be thrown away by the average user.
You leave it out by your kung fu, which mean that packages which makes uses of it, doesn't compile this functionality into, although you set the superior grade mmext flag.


Learn what mmxext is.

Use the appropriate page to lookup such cpu instructions: https://packages.gentoo.org/useflags/cpu_flags_x86_mmxext

https://forums.gentoo.org/viewtopic-t-554289-start-0.html for the padlock cpu_flags_x86.

Quote:
Similar for your mabm mcx16 mlzcnt msahf, these doen't exist as cpu_flags_x86 and probably never would.
Theses flags belongs to the CFLAGS, if there's a need activate (-m) or deactivate (-mno) it there.

Your claim here the 2 missing cache-size parameters, but since I found a package (openmpi), which doesn't compile successfully here on my westmere Xeon cpu if I use
the l1-cache-size= parameter, I'm not sure if leaving out the cache parameters is good or bad. How big would be the difference regarding performance? I never tested it.
In the sense of giving safe CFLAGS, it seems to be better to leave it out to me. To present safe CFLAGS is lastly the intention of this little helper script.


Never say never, also google which cpu's are supporting avx512* and when they first appeared. Also it's bad idea to enable such CFLAGS globally, as it's bad idea to use USE flags globally instead per-package USE/CFLAG.

The "big" difference is that you've not used Google, also it tells me that you have not programmed in low level language.

Your two comments contradict with each other.

So you've found a bug regarding the cache size, can you be a nice person and report it to openmpi, so they can fix it ?

Quote:
mmx ...


Same could be said for pipe, just because it ships in vanilla make.conf and is hard-coded in Sebastian's program doesn't make it "safe":

It tells the compiler to use pipes instead of temporary files during the different stages of compilation, which uses more memory. On systems with low memory, GCC might get killed. In those cases do not use this flag.

Also, no one needs hard-coded -O2 -pipe just to tell them that those flags have been "detected".

---

I'll stop right here and ask any moderator/administrator to lock this thread, before some **already angry** people start insulting me and embarrass themselves even further than they are currently.
Back to top
View user's profile Send private message
Randy Andy
Veteran
Veteran


Joined: 19 Jun 2007
Posts: 1103
Location: /dev/koelsch

PostPosted: Mon Jul 18, 2016 8:57 am    Post subject: Reply with quote

frankenputer wrote:

Learn what mmxext is.

The "big" difference is that you've not used Google, also it tells me that you have not programmed in low level language.

I'll stop right here and ask any moderator/administrator to lock this thread, before some **already angry** people start insulting me and embarrass themselves even further than they are currently.


Wow,

it seems that you know more about me, than myself.

Respect, never experienced that before here, maybe because you're relative new to this outstanding forum.

It looks like you're not really interested in technical based criticism and react very emotional when you get one.
Then you blame on me on a personal basis and would try to lock the thread - not bad guy, but to late.

No need to, cause I'm not longer willing to discuss/escalate it with you in that way!
Bye.
_________________
If you want to see a Distro done right, compile it yourself!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum