[IMPROVED] Shell script for pretty kernel config comparison

[Goverp]

The (somewhat long, sorry) shell script at the end will produce a succinct and pretty (and colourful on terminals) comparison between two kernel configuration files as produced by "make menuconfig" etc.
Its Unique Selling Point is that it retains the comments that group the settings in the menus, and uses them to produce an indented list.
For example,

Code: Select all

bin/cfcfg /boot/config-6.2.12-git /boot/config-6.3.1-git | less

produces the following output on my system (sadly lacking colour):

Code: Select all

# Linux/x86 6.2.12 Kernel Configuration ---> # Linux/x86 6.3.1 Kernel Configuration
   # General setup
-      GCC12_NO_ARRAY_BOUNDS=y 
+      SCHED_MM_CID=y 
   # Processor type and features
       # Performance monitoring
-          PERF_EVENTS_AMD_POWER=y 
-          PERF_EVENTS_AMD_UNCORE=y 
+  AS_GFNI=y 
-  BLOCK_COMPAT=y 
   # Networking options
       # Core Netfilter Configuration
-          NETFILTER_FAMILY_ARP=y 
       # IP: Netfilter Configuration
-          IP_NF_TARGET_CLUSTERIP=m 
   # Device Drivers
       # Intel thermal drivers
-          X86_PKG_TEMP_THERMAL=m 
       # Media drivers
           # Media drivers
               # FireWire (IEEE 1394) Adapters
+                  UVC_COMMON=m 
       # Graphics support
+          DRM_DISPLAY_HDCP_HELPER=y 
       # HID support
+          HID_SUPPORT=y 
   # File systems
+      LEGACY_DIRECT_IO=y 
+      RPCSEC_GSS_KRB5=y

And here's the script. The help text at the top is also the documentation:

Code: Select all

#!/bin/sh
# cfcfg - Produce a succinct kernel configuration comparison

# Copyright (C) 2023 Paul Gover
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

ME="${0##*/}"
set -e	# Abort on errors - will leave some temp files

### Help and documentation:
doHelp() {
cat << endhelp
SYNTAX: $ME [-c] [-m] [-w nnn] oldcondfigfile newconfigfile

Produce a succinct comparison of two kernel config files,
which MUST be the unmodified .config file from the kernel "make" configurator.
Changes such as sorting or manual editing will probably cause erroneous and voluminous output.
Retains contextual information comments such as Processor options, Device driver,
including nested contexts.
"Succinct" means that all unset items and irrelevant comments get stripped from the output,
as are the "CONFIG_" prefixes to item names.

Can also be used to pretty-print a succinct config file extract
by making one of the comparison configs /dev/null.

OPTIONS:
-c      Force output colouring when output to file or pipe.
-m      Conver module settings (=m) to builtin (=y) before comparison
        This will remove non-functional differences.
-w nnn  Width of "diff" columns used internally for the comparison.
        Output lines can be up to twice this length, plus gutters and indentation,
        but usually a lot less.

Note that the output is truncated to fit the screen, but the comparison is based on all the text.
This means that the display of long lines might not reach the differences.

The indentation is based on the location of comments in the configuration files.
This is not a defined API, so the algorithm used may not always get it right.
There are missing comments such as Bluetooth and duplicates such as Media drivers.

Uses "diff", "gawk" and "mktemp".  It might not work with other awk implemenations.
These come from diffutils, gawk and coreutils packages.
Coded for Posix shells such as dash, and works with bash.
endhelp
}

### Pipe filters using gawk
# BEWARE: the gawk program parameters are passed as '-commented strings over several lines
# so the gawk command line almost invariable ends with a ' matched several lines later

### Filter useful stuff from a config file
strip() {
        # We have to build the awk program in bits to allow shell variable substitution
        local stripper
        stripper='
                /^CONFIG_.*/ {'

        [ "$ignoremodules" ] && stripper="$stripper "'gsub( /=m$/ , "=y") ;'  # Convert module items to inline

        stripper="$stripper"'print substr($0, 8); next }        # Strip out the fixed prefix - less work for diff

                /^[#[:space:]]*$/       { next }                # Skip comments without text
                /^#.*is not set/        { next }                # Skip unset items
                /^#.*generated file/    { next }                # Skip boilerplate heading
                /^#/                    { print ; next }        # Print comments with text
                { print "Unexpected", $0 > "/dev/stderr" }      # Catchall for cruft
        '

        gawk "$stripper"
}

### Filter "end of" comments so we can identify nesting comments
ended() {
        gawk -v FIELDWIDTHS="1 1:$width 1:*" '$2~/^# end of/ { gsub( /[[:space:]]*$/ , "", $2) ; print substr($2, 10) ; next }'
}

### Split side-by-side diff output containing comments
# into two separate lines (or one if both halves of the line are equal)
# Remove non-comment lines with no changes
# Reorder the fields to put the indication first
# There's probably a better way than using side-by-side...
split() {
        gawk -v FIELDWIDTHS="$width 1:1 1:*" '
        NR<5 && /^#.*Kernel Configuration.*[\\|\/]/     {
                print "|", $1, $3 ; next
                }                                               # Leave the header line
        /^#/ && $2~/ /          { print " ", $1 ; next }        # Matched comments, print once
        $2~/ /                  { next }                        # Matched settings - ignore
        $2~/[\\|\/]/ && ($1~/^#/ || $3~/^#/)    {
                print "-", $1 ; print "+", $3 ; next
                }                                               # Change involving a comment, do both
        $2~/[\\|\/]/            { print "|", $1, $3 ; next }    # Changed settings
        $2~/[>]/                { print "+", $3 ; next }        # Added line, comment or setting
        $2~/[<]/                { print "-", $1 ; next }        # Removed line, comment or setting
        { print "What?", $2, $1, $3 > "/dev/stderr" }           # Anything else is an error
        '
}

### Parse diff output:
# Handle initial label comment (describing kernel versions).
# Use the list of ending with "# end of <foo>";
# they indicate nesting. so we can count indentation levels.
# Abbreviate inserts, deletions and changes for a concise report.
reduce() {
        gawk -v FIELDWIDTHS="1 1:$width 1:*" -v endedFile="$endsFile" -v colors="$ADD:$REM:$CHG:$NIL" '
        BEGIN {
                while( getline < endedFile ) { matched[$0]=0 }  # Build lookup array of nesting comments

                # Set up an array of ANSI colour codes for additions, removals and changes.
                split(colors, colours, /:/)
                colours["+"]=colours[1] ; colours["-"]=colours[2] ; colours[">"]=colours[3] ; reset=colours[4]
                depth=0+0               # Current nesting level of comments; numeric
                last=depth              # Nesting level of last printed output

                # Initialize stack of nested comments
                comment[depth]="SPURIOUS"
                matched[comment[depth]]=0
        }
        function indent(n) {
                return substr("                                                                     ", 1, 4*n)
        }
        function debug(what, this        , i) {
                print what, this, "depth=" depth, "last=" last, "matched=" (this in matched)
                for(i=0; i<=depth; i++) print i, comment[i]
        }
        function dostart(this) {
                if(comment[depth] in matched) comment[++depth]=this
                else { comment[depth]=this ; if(last>=depth) last=depth-1 }
                # debug("Start", this)
        }
        function doend(this     , nest) {
                nest=1
                while( nest<depth && comment[nest]!=this) nest++
                if(comment[nest]==this) depth=nest-1
                if(depth<last) last=depth
                # debug("End", this)
        }
        function dosetting(type, item) {
                # if (last!=depth) debug("Stack", item)
                while(last<depth) { print " ", indent(last++), "#", comment[last] }
                print type, indent(depth) colours[type], item, reset
        }
        function dochange(o, n          , old, new) {
                split(o, old,  /=/)
                split(n, new,  /=/)
                if(old[1]==new[1])  dosetting(">", old[1] "=" old[2] "--->" new[2])
                else {
                        dosetting("-", o)
                        dosetting("+", n)
                }
        }

        { gsub( /[[:space:]]*$/ , "", $2) }     # Trim trailing space from first field

        1==NR && /Kernel Configuration/ { print $2, "--->", $3 ; next }         # Handle the header line

        # Comments
        $2~/^# end of/  { doend(substr($2, 10)) ; next }
        $2~/^#/         { dostart(substr($2, 3)) ; next }

        # Settings
        $1=="-"         { dosetting("-", $2) ; next }
        $1=="+"         { dosetting("+", $2) ; next }
        $1=="|"         { dochange($2, $3) ; next }

        # Cruft
        { print "Oops", $0 > "/dev/stderr" }'
}

### Shell functions

# Create a tempfile, using /run ramdisk if possible
Tempfile() {
        local id dir
        id="$(id -u)"

        if      [ "$id" = "0" ]
        then
                dir="${XDG_RUNTIME_DIR:-/run}"
                [ -d "$dir" ] || dir=""
                mktemp -p "$dir" "tmp.$ME.XXXXXXXXXX"
        else    mktemp -p "${XDG_RUNTIME_DIR}" "tmp.$ME.XXXXXXXXXX"
        fi
}

# Print message to stderr and exit 1
Die() {
        local template
        template="$1" ; shift
        printf "$CHG$template\n" "$@" >&2
        exit 1
}

### Mainline code

# Handle parameters - flags ask, pretend and verbose will either be null or set to themselves, so "if [ $flag ]" works
while getopts '?hw:mc' f
do
        case "$f" in
                c)      colour="always" ;;
                m)      ignoremodules="true" ;;
                w)      width="$OPTARG" ;;
                *)      doHelp ; exit ;;
        esac
done
shift $(( OPTIND - 1 ))

# Set coloured output if necessary
if [ -t 1 ] || [ "$colour" = "always" ]
then
        ADD="\033[32m"          # Green
        REM="\033[34m"          # Blue
        CHG="\033[33m"          # Yellow/brown
        NIL="\033[0m"           # Back to white
fi

[ "2" = "$#" ] || Die "Must be exactly two arguments."
[ -r "$1" ] || Die "File 1 - $1 - must be readable."
[ -r "$2" ] || Die "File 2 - $2 - must be readable."
case "$width" in
        *[!0-9]*)       Die "Width (-w) must be numeric." ;;
esac

cfgFile1=$(Tempfile)
cfgFile2=$(Tempfile)
splitFile=$(Tempfile)
endsFile=$(Tempfile)

strip < "$1" > "$cfgFile1"
strip < "$2" > "$cfgFile2"

w="${width:=80}"
w=$(( w + w +3 ))

diff --expand-tabs --minimal --side-by-side --width="$w" "$cfgFile1" "$cfgFile2" \
        | split > "$splitFile"

ended < "$splitFile" >"$endsFile"       # Build list of "# end of ..." comments
reduce < "$splitFile"

rm "$cfgFile1" "$cfgFile2" "$splitFile" "$endsFile"

[pietinger]

I would love it ...

Code: Select all

 ~ # LC_MESSAGES=C ./cfcfg -c /home/peter/Check/config-orig /usr/src/linux/.config
./cfcfg: line 207: warning: here-document at line 14 delimited by end-of-file (wanted `endhelp')
./cfcfg: line 208: syntax error: unexpected end of file

[Goverp]

...

Code: Select all

 ~ # LC_MESSAGES=C ./cfcfg -c /home/peter/Check/config-orig /usr/src/linux/.config
./cfcfg: line 207: warning: here-document at line 14 delimited by end-of-file (wanted `endhelp')
./cfcfg: line 208: syntax error: unexpected end of file

I was about to ask which shell you are running, when I think I diagnosed the problem.
I use tabs to indent, and the start of the help text is

Code: Select all

cat <<- endhelp

which strips leading tabs, including on the "endhelp" line, so that "endhelp" effectively appears in column 1. But I suspect the cut-and-paste to put the program in the forum changed the tabs to spaces.
Cure is to remove the leading spaces on the endhelp line. I'll hack it in the original post.

[Hu]

The forum is known to mangle tabs like that, which tends to break GNU patch files too. As a workaround, the reader can choose to quote your post, then copy the program from the quoted post textbox, instead of copying from the thread reading view. The version in the textbox used for composing the reply has its tabs intact. The reply can be discarded without posting it, to avoid cluttering the thread.

[pietinger]

[...] I'll hack it in the original post.

Thank you very much ! Works like a charm.

@Hu

First of all: Thanks a lot for this information and confirmation of @Goverp's supposition. I didnt knew that ... and I think ... many others dont know that either; so, the hack is IMHO the best solution (everyone can copy from the first post).

[Hu]

Yes, if the OP is available to edit it (or if a moderator is willing and able to adjust it), removing tabs can make things easier, when that does not break the blob. For GNU patch files, removing tabs likely breaks the patch file anyway, so the blob must be used as provided.

The quote-based approach is useful if you find a post where someone used tabs, the tabs are important, and you cannot or will not wait for them to fix it (whether because you want to try the thing immediately or because you expect that the poster might be away for weeks).

[grknight]

FWIW, a simple tool is also available, without color or indentation, included with every kernel sources.

/usr/src/linux/scripts/diffconfig is also good for basic comparison. It defaults to comparing .config.old to .config but can be fed any 2 files.

[Goverp]

FWIW, a simple tool is also available, without color or indentation, included with every kernel sources.

/usr/src/linux/scripts/diffconfig is also good for basic comparison. It defaults to comparing .config.old to .config but can be fed any 2 files.

Indeed, though I'd question "simple", as it does a similar amount of processing, but in python instead of awk.
It's main deficit, IMHO, is that it reorders the statements (which is why I wrote mine), but it is even more succinct.

[CaptainBlood]

Goverp,
Nice

Thks 4 ur attention, interest & support.

[CaptainBlood]

Code: Select all

Tempfile() {

for consistency maybe?
Thks 4 ur attention, interest & support.

[mv]

Code: Select all

	printf "$template\n" "$@" >2&

In a POSIX shell like dash, this redirects to the file 2 and executes the line in the background.
What you want is probably

Code: Select all

	printf "$template\n" "$@" >&2

[mv]

I did not investigate deeply, but the script showed me for the change of my default configuration with nconfig from 6.2 to 6.3 less than 10 changes while e.g. kccmp showed me more than 30.

I remember that I had a similar issue with the script from the kernel subdirectory, but there I could add some options and change the order of config files to see the remaining ones.
So unfortunately, it seems that I will stick with kccmp, although I would be happy about a cli replacment.

[Goverp]

...
What you want is probably

Code: Select all

	printf "$template\n" "$@" >&2

Thanks, corrected!

[Goverp]

I did not investigate deeply, but the script showed me for the change of my default configuration with nconfig from 6.2 to 6.3 less than 10 changes while e.g. kccmp showed me more than 30.
...

Interesting. Could you post or PM me the above examples so I can see why there's a difference.
The ebuild in your link points to a domain name that's up for sale

[mv]

The ebuild in your link points to a domain name that's up for sale :-(

Thanks. Fixed.

Here is a minimal example

# Automatically generated file; DO NOT EDIT.
#
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_SND_X86 is not set

#
# HID support
#
CONFIG_HID=m
CONFIG_HID_BATTERY_STRENGTH=y
# CONFIG_HIDRAW is not set
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=m

# Automatically generated file; DO NOT EDIT.
#
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_SND_X86 is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=m
CONFIG_HID_BATTERY_STRENGTH=y
# CONFIG_HIDRAW is not set
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=m

Probably, the problem is that two sections were merged in kernel 6.3. Surprisingly cfcfg not only misses the new CONFIG_HID_SUPPORT=y but also reports a non-change for CONFIG_THREAD_INFO_IN_TASK=y.

[Goverp]

mv, thanks.

I think I know the problem - my code isn't handling the case where a comment exists in only one of the two files and gets paired up with a spare CONFIG item - in that case it ignores whatever gets matched. diff is probably matching a spurious change with the comment to the CONFIG_HID_SUPPORT=y. I'll have to handle the comments a bit more carefully.

[Goverp]

I've improved the code, and updated the first post with the latest version. Enjoy.
It fixes brackets for Cap'n Blood, and I hope mv's problems, at the cost of another temp file - it ought to be possible to avoid that with a smarter call to "diff", but that's too complex for me... It also has all tabs expanded to spaces, so cut and paste should now work on the source.
mv's test case now produces (having removed the spurious blank at the end of the last line of 6.3.config)

Code: Select all

   # HID support
+      HID_SUPPORT=y 

It also fixes the mess I'd made of choosing the comment hierarchy printed. It's not perfect, but now that just reflects the limitations of the config file contents.
With a dummy "# end of General setup" comment at the end of the 6.2 and 6.3 configs, you get

Code: Select all

   # General setup
       # HID support
+          HID_SUPPORT=y

[mv]

Goverp, thanks a lot for your nice script!

[pietinger]

Goverp, thanks a lot for your nice script!

++

Again, thank you very much Goverp ... I am using it excessive !


(and I linked to this thread from my wiki article )

[mv]

Just for the records: kccmp still shows a lot more differences. Unless I missed something, these are all *unset* options which exist only in one of the two config files, that is, comment lines of the form

Code: Select all

# CONFIG_.... is not set

It would be nice if these lines could be treated like variable assignments (possibly optional). Of course, up to you (I can modify the script myself if needed).

BTW, what do you think about uploading your script to github so that one can easily (and publicly) easily suggest patches like this (and has no problem with tabs with downloading); in fact, one could then even have an ebuild for it. (As the script is GPL3+ I guess that I can theoretically upload it into my github account even without asking, but I would do this only if you explicitly agree and do not want to do it by yourself.)

[Goverp]

Just for the records: kccmp still shows a lot more differences. Unless I missed something, these are all *unset* options which exist only in one of the two config files, that is, comment lines of the form

Code: Select all

# CONFIG_.... is not set

It would be nice if these lines could be treated like variable assignments (possibly optional). Of course, up to you (I can modify the script myself if needed).

BTW, what do you think about uploading your script to github so that one can easily (and publicly) easily suggest patches like this (and has no problem with tabs with downloading); in fact, one could then even have an ebuild for it. (As the script is GPL3+ I guess that I can theoretically upload it into my github account even without asking, but I would do this only if you explicitly agree and do not want to do it by yourself.)

mv, pietinger, thanks for the interest.

The behaviour to exclude unsets is deliberate. I guess it would be easy to add them in. Certainly in the world of modules, they're new modules not being built, and printing that out would be about as annoying as "make oldconfig" asking if you want something with default No .

As for putting it on github or gitlab, I've a couple of items on sourceforge; I could put it there, or start a gitlab account. The trouble then is I'd probably want to fix the way it uses diff, and then there's building a dictionary of prompts so you could restrict the output further to visible settings (with prompts) rather than the hidden ones used to make "make" work. There's a big python script somewhere to scan Kconfig files, though I suspect a mix of rg (or grep with bells on) and more awk could build the dictionary. The real issue is that's it's hacky, whatever we do; the comments in the .config file aren't an API, so it might break at any moment, but they're the big feature! Hmmm...

I'll ponder this over the weekend, while seeing if I can do something about the diff. I'll also see if I can find a Kconfig maintainer to ask about the comments' status (and point out the glaring errors at the moment!).

[Goverp]

OK, there's now a project on sourceforge.
It has a new "-u" option to retain unset items.

[CaptainBlood]

Nice,

Any advice where to put cfcfg.1.man? (sorry, noob here)

EDIT:
Because cfcfg is located in /usr/local/share/bin here, I did;

Code: Select all

cp <source directory>/cfcfg.1.man /usr/local/share/man/man1/cfcfg.1
chown root:root /usr/local/share/man/man1/cfcfg.1
mandb #may be unrequired...

Thks 4 ur attention, interest & support

[mv]

OK, there's now a project on sourceforge.
It has a new "-u" option to retain unset items.

That's great, thanks a lot.
There is now an ebuild in the mv overlay.

[CaptainBlood]

@mv, Neat.

Thks 4 ur attention, interest & support.