hashing out every second line after hashed line [SOLVED]

ryszardzonk · Last edited by ryszardzonk on Sat Jun 30, 2018 8:22 am; edited 1 time in total

Adblock2privoxy converter I use for some reason sometimes converts rules twice instead of just once while the second rule is worse than first and should not be there in the first place. Therefore question arises before converted gets fixed. Is it possible to hash out a line that comes two lines after a hash and if the new line with hash is reached start the count over?

File has about 700 000 lines and is recreated every 2 days so manual intervention is not feasible.

Original example

Hu · Moderator Joined: 06 Mar 2007 Posts: 21607

This could be a bit simpler, but is written long to make it clearer.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3131

ryszardzonk · Posted: Sat Jun 30, 2018 7:59 am Post subject:

Wow, I spend quite some time reading to get some sort of magical one liner using sed, but I guess it is much more complicated than that

@szatox
I run your code with cat filter.orig |./hash_fix.sh >filter.new and it did exactly what it was told with the exception that it also removed \ from all the lines which is undesired. Not sure why it did that, but when I removed \ from the code it obviously failed with improper syntax.

@Hu
Yes. It worked and worked even better than I expected as it turned out there were cases which I did not anticipated. Your code hashed not only second line after hash, but all lines following it until next hash which is not what I asked for, but because I have not seen it before converter sometimes created more than two rules while the every next one was more greedy than previous one. Thanks to you that problem is now gone before I even new it existed

part of diff after running cat filter.orig |python hash_fix.py >filter.new

Hu · Moderator Joined: 06 Mar 2007 Posts: 21607

Those are Useless Uses of Cat. You can redirect stdin from the file instead of using cat file |.

That is a useless double use of sed. You can do both expressions in a single run. You could even do them in a single expression with the right syntax. For two expressions in one run, sed -i -e expr1 -e expr2 file instead of your sed -i -e expr1 file; sed -i -e expr2 file.

If there are particular lines that need to be excluded, you can modify the Python script not to mangle them in the first place. Change the else: to elif condition:, where condition matches lines that are commented out by my script, but that you want to exclude from modification. Based on your shown sed, I think you want elif line.startswith('}') or line.startswith('TAG'):.