Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Unreproducible errors during compilation
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
acheron2
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jan 2007
Posts: 100

PostPosted: Sat May 19, 2012 6:16 pm    Post subject: Unreproducible errors during compilation Reply with quote

I've been using media-video/swfmill with these ebuilds for years on my old computer. New versions compiled without problems, but some people reported compilation errors.

After moving to a new multicore computer I started getting errors too. Odd things are:

1. Compiling the sources in my home directory worked.
2. Compiling the sources with the command
Code:
ebuild swfmill-0.3.2.ebuild compile
worked.
3. Compiling the sources with the command
Code:
emerge swfmill
failed several times. Each time with a different error.

What could this be? Some sort of race condition?

When I tried the command
Code:
MAKEOPTS="-j1" emerge swfmill
it worked. After that compilation started to always work, so I can't reproduce the errors anymore.

If this is a race condition, how can I ascertain that? Or what else could this be?
Back to top
View user's profile Send private message
acheron2
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jan 2007
Posts: 100

PostPosted: Sat May 19, 2012 6:24 pm    Post subject: Reply with quote

Another reason that makes me think it's a race condition, is this. All errors were in *.cpp files generated by xsltproc. All errors were either about unmatched bracket or undeclared variable. When I checked the file, where the error occurred, all brackets were matched and there was no variable with such name, but instead a variable starting with it — no "Trac", but "Trace". As if the compiler read the incomplete source file.

If this is true, how should I change the makefile to avoid it?
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 7679

PostPosted: Sat May 19, 2012 7:53 pm    Post subject: Reply with quote

The most likely cause for the behavior you are seeing is that someone wrote a rule of the form:
Code:
a.cpp b.cpp: code.xml.in
    : call-to-create-both-outputs
The problem with this is that it is a shorthand for specifying a.cpp and b.cpp as separate targets, so this tells Make to run that command if any of those files is needed. If more than one is needed, Make will run instance of the command for each required file, and one might step on the other. The fix is to identify the set of all source files generated by a single xsltproc call and change the Makefile so that every member of the set depends on a flag file, and runs no commands. Then, make the flag file run the xsltproc command.
Code:
.sources.generated:
    : call-to-create-both-C++-files && touch .sources.generated

a.cpp b.cpp: .sources.generated
Back to top
View user's profile Send private message
acheron2
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jan 2007
Posts: 100

PostPosted: Sat May 19, 2012 10:29 pm    Post subject: Reply with quote

Thanks.

In Makefile xsltproc is called only once:
Code:

$(CODEGEN_GENERATEDSOURCES) $(CODEGEN_GENERATEDHEADERS): $(CODEGEN_SRC)
        xsltproc $(srcdir)/codegen/mk.xsl $(srcdir)/codegen/source.xml


The variables are defined as:
Code:

CODEGEN_GENERATEDSOURCES = \
        gSWFParseXML.cpp \
        gSWFWriteXML.cpp \
        gSWFParser.cpp \
        gSWFWriter.cpp \
        gSWFDumper.cpp \
        gSWFBasics.cpp \
        gSWFSize.cpp \
        $(NULL)

CODEGEN_GENERATEDHEADERS = \
        SWF.h \
        $(NULL)

CODEGEN_SRC = \
        codegen/basic.xsl \
        codegen/basics.xsl \
        codegen/dumper.xsl \
        codegen/header.xsl \
        codegen/mk.xsl \
        codegen/parser.xsl \
        codegen/parsexml.xsl \
        codegen/size.xsl \
        codegen/source.xml \
        codegen/writer.xsl \
        codegen/writexml.xsl \
        $(NULL)


There are rules that have $(CODEGEN_GENERATEDSOURCES) as a prerequisite (via other intermediate variables) and rules that require only a single generated gSWF*.cpp.

So, what do you suggest? If I replace the top rule with this:
Code:

$(CODEGEN_GENERATEDSOURCES) $(CODEGEN_GENERATEDHEADERS): .sources.generated

.sources.generated: $(CODEGEN_SRC)
        xsltproc $(srcdir)/codegen/mk.xsl $(srcdir)/codegen/source.xml
        touch .sources.generated
will it be enough? (Also, there should be rules to delete .sources.generated during "clean", to not include it in the distributions, etc.)

And another question still remains: how to make sure the errors stopped because of this fix?
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 7679

PostPosted: Sat May 19, 2012 10:46 pm    Post subject: Reply with quote

Yes, the change you describe should work. You can test whether the problem is solved by checking how many times xsltproc is run during compilation. It should run only once.
Back to top
View user's profile Send private message
acheron2
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jan 2007
Posts: 100

PostPosted: Sun May 20, 2012 7:33 am    Post subject: Reply with quote

Hu wrote:
You can test whether the problem is solved by checking how many times xsltproc is run during compilation. It should run only once.


The problem is: it sometimes runs only once with default settings. Even now I don't know, whether -j1 truly solves the problem, or swfmill compilation stopped failing by accident. I did not keep the logs of failed compilations, and now I can't reproduce them :)

Another question. (No, I am not very familiar with how make works.) For example, xsltproc already started and is in the middle of writing gSWFWriteXML.cpp. Then make tries to execute a rule, that has gSWFWriteXML.cpp as the only prerequisite. What will happen? Will make wait for .sources.generated, or will it start compiling gSWFWriteXML.cpp since it's already present?
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 7679

PostPosted: Sun May 20, 2012 3:57 pm    Post subject: Reply with quote

Given the rule you showed above, I believe using -j1 was the reason the failures ceased. For your second question, it depends on other factors. If a single Make is managing the entire process, then I think the order of events would be correct. Make should inventory all of the goals, determine which objects it needs to satisfy those, then determine which sources it needs to satisfy those objects. It should then begin generating those sources, but notice that those sources require the target .sources.generated, so it will build that first. It will then run the null rules defined for the individual sources, then move to compiling. I think the failure mode you describe can occur only if Make delays dependency resolution. You could get into problems with recursive make, since a recursive make may not have all the rules present in a single process.

You could test this by replacing the xsltproc rule with one which writes an ill-formed gSWFWriteXML.cpp, sleeps long enough that other jobs ought to finish, then replaces the ill-formed gSWFWriteXML.cpp with a valid one, and exits successfully. Then run a parallel build and see if it defers the compilations until the sleep terminates.
Back to top
View user's profile Send private message
acheron2
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jan 2007
Posts: 100

PostPosted: Sun May 20, 2012 6:43 pm    Post subject: Reply with quote

Thanks, I'll try that. Probably next weekend :(
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum