Some of the servers on DCC's keep going down quite often. The server going down most often (since a few days basically exclusively) is the CTF server (about once a day). So maybe the cause is some problem with certain maps?

I don't find a hint on what went wrong in the server log files. What can I do to get more information?

The servers are running on Debian 4.0 with AMD64. When using the release 2.3 binary there were pretty often segfaults at server startup. I have tried a locally built binary with CPUOPTIMIZATIONS=-march=athlon64 -m64. Since then the servers aren't crashing right at start anymore. Maybe there are issues with Debian / AMD64?

According to Dmesg several segfaults were caused by nexuiz processes (but also by apache processes so maybe there is something wrong with the installation).

In the logs there is no trace of a possible shutdown via Rcon.

The servers are restarted each morning.

First of all, try using our builds. I have used Gentoo long enough (some years) to not trust its addiction to unnecessary optimizations.

Secondly, enable core dumps (ulimit -c unlimited) before starting Nexuiz, then you get core dumps. These can be used to identify the line where the problem happens. However, for this to work, the binaries need to have debug information, and I have no idea how to make Gentoo do that, but you can always compile the sources yourself. http://141.2.16.3/nex/nexuizdebug20070531d_2.3.zip contains the same binaries as 2.3, just with this debug information (actually, the official binaries were created from these using "strip"). When you got a core dump of an official nexuiz build, you can still use these builds to analyze the core dump (by passing it as argument to gdb).

Without the information you can get from a core dump, we can't help you. "It is crashing" is no accurate error description, I'm afraid.

ladder.tchr.no is a A64 with debian 4. dead stable. used nexuiz download, didnt try repository

The server does not run as a root process

Oops... sorry, I saw -march= and already concluded Gentoo...

okay, so can you try getting a core dump out of our very release binaries?

DivVerent, thanks for your help. I will try using the debug build you mentioned to get a core dump. And tChr, good to know that Debian 4 and AMD64 work well and thus there only can be some local problem.

Nil wrote:DivVerent, thanks for your help. I will try using the debug build you mentioned to get a core dump. And tChr, good to know that Debian 4 and AMD64 work well and thus there only can be some local problem.

Crundandcrap.. my bad, ladder.tchr.no is sarge.. I'll upgrade. I'll get back to you.

If you do that, can you take a look at here? http://forums.alientrap.local/viewtopic.php?t=1860

Server is now etch but I will be away for a few days.. We'll see if it still runs when I'm back

The PlanetNexuiz CTF server happened to crash once, 2 days ago (segfault). If this keeps being such a little problem I won't further bother about it. If it occurs more often I'll use the debug builds. coredumps are active afaik anyway.

Crash has been reported and fixed in svn DarkPlaces. It only happens to long running servers (it is an integer overflow in a collision routine, so it only happens when that routine was called >2147483647 times)

Same here, my server crash often (3-4 times this week)

Code: Select all


Core was generated by `./nexuiz-linux-686-dedicated-debug +exec spacectf.cfg'.
Program terminated with signal 11, Segmentation fault.
#0  0x08113350 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81836bc "customizeentityforclient: NULL function")
    at prvm_exec.c:508
508     prvm_exec.c: No such file or directory.
        in prvm_exec.c
(gdb)


Ronan wrote:Same here, my server crash often (3-4 times this week)

Since the DCC's servers are running the debug binary (4th of july) there were only two crashes in mid of running and three crashes on startup.
The segfault on your machine seem to have a different cause than on DCC's

Kadaverjack had a look at the core dump of the in the mid of running crash and came to the resolution that the problem probably isn't an engine bug but caused by damaged ram or hard disk.
What happens is that an entity (mg bullet) is trying to attach itself to itself causing a loop till segfault:

Code: Select all

#39986 0x00000000004ea84a in SV_BuildEntityState (cs=0xa7e0730, ent=0xac207c0, enumber=446) at sv_main.c:937
#39987 0x00000000004ec907 in SV_SendClientMessages () at sv_main.c:1203
#39988 0x000000000046dc07 in Host_Main () at host.c:691
#39989 0x0000000000402050 in main (argc=5, argv=0x7fff8fa16478) at sys_linux.c:254


I paste the backtrace of one of the startup crashes too (the others look the same), maybe it helps to find a bug or it provides more indication of hardware problems on the DCC's machine:

Code: Select all

Core was generated by `./nexuiz-server +exec server-keyhunt.cfg +sv_public 1'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000043a9da in COM_ParseToken_VM_Tokenize (datapointer=0x7fffec61a890, returnnewline=0) at common.c:1147
1147    common.c: No such file or directory.
        in common.c
(gdb) bt
#0  0x000000000043a9da in COM_ParseToken_VM_Tokenize (datapointer=0x7fffec61a890, returnnewline=0) at common.c:1147
#1  0x00000000004be255 in VM_tokenize () at prvm_cmds.c:1983
#2  0x00000000004cedfe in PRVM_ExecuteProgram (fnum=<value optimized out>, errormessage=0x5210fe "") at prvm_execprogram.h:307
#3  0x00000000004c69f5 in PRVM_ED_LoadFromFile (
    data=0xacb1b0e "\r\n{\r\n"classname" "info_null"\r\n"origin" "-120.000000 -265.000000 370.000000"\r\n"targetname" "t128"\r\n}\r\n{\r\n"classname" "info_player_deathmatch"\r\n"origin" "-770.000000 560.000000 190.000000"\r\n"angle" "0"\r"...) at prvm_edict.c:1279
#4  0x00000000004edb56 in SV_SpawnServer (server=0x7fffec61ae00 "aggressor") at sv_main.c:2431
#5  0x00000000004711dc in Host_Map_f () at host_cmd.c:275
#6  0x0000000000434872 in Cmd_ExecuteString (text=0x7fffec61aed0 " map aggressor ", src=<value optimized out>) at cmd.c:1111
#7  0x0000000000434995 in Cbuf_Execute () at cmd.c:188
#8  0x000000000046d84e in Host_Init () at host.c:1021
#9  0x000000000046d87b in Host_Main () at host.c:539
#10 0x0000000000402050 in main (argc=5, argv=0x7fffec623078) at sys_linux.c:254


Ronan wrote:Same here, my server crash often (3-4 times this week)

Code: Select all


Core was generated by `./nexuiz-linux-686-dedicated-debug +exec spacectf.cfg'.
Program terminated with signal 11, Segmentation fault.
#0  0x08113350 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81836bc "customizeentityforclient: NULL function")
    at prvm_exec.c:508
508     prvm_exec.c: No such file or directory.
        in prvm_exec.c
(gdb)


Need more information, do you still have the core dump? Would be nice if you could make gdb find the sources (extract the enginesource zip file) and do a "bt full" to see the local variables of PRVM_ExecuteProgram.

Nil wrote:What happens is that an entity (mg bullet) is trying to attach itself to itself causing a loop till segfault:

Code: Select all

#39986 0x00000000004ea84a in SV_BuildEntityState (cs=0xa7e0730, ent=0xac207c0, enumber=446) at sv_main.c:937
#39987 0x00000000004ec907 in SV_SendClientMessages () at sv_main.c:1203
#39988 0x000000000046dc07 in Host_Main () at host.c:691
#39989 0x0000000000402050 in main (argc=5, argv=0x7fff8fa16478) at sys_linux.c:254


Still no idea how this can happen, still - it could be a bug in the QC code. I just have no idea what could be causing a self-attached entity, that is, how someent.owner == someent could have been caused.

Code: Select all

Core was generated by `./nexuiz-server +exec server-keyhunt.cfg +sv_public 1'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000043a9da in COM_ParseToken_VM_Tokenize (datapointer=0x7fffec61a890, returnnewline=0) at common.c:1147
1147    common.c: No such file or directory.
        in common.c
(gdb) bt
#0  0x000000000043a9da in COM_ParseToken_VM_Tokenize (datapointer=0x7fffec61a890, returnnewline=0) at common.c:1147
#1  0x00000000004be255 in VM_tokenize () at prvm_cmds.c:1983
#2  0x00000000004cedfe in PRVM_ExecuteProgram (fnum=<value optimized out>, errormessage=0x5210fe "") at prvm_execprogram.h:307
#3  0x00000000004c69f5 in PRVM_ED_LoadFromFile (
    data=0xacb1b0e "\r\n{\r\n"classname" "info_null"\r\n"origin" "-120.000000 -265.000000 370.000000"\r\n"targetname" "t128"\r\n}\r\n{\r\n"classname" "info_player_deathmatch"\r\n"origin" "-770.000000 560.000000 190.000000"\r\n"angle" "0"\r"...) at prvm_edict.c:1279
#4  0x00000000004edb56 in SV_SpawnServer (server=0x7fffec61ae00 "aggressor") at sv_main.c:2431
#5  0x00000000004711dc in Host_Map_f () at host_cmd.c:275
#6  0x0000000000434872 in Cmd_ExecuteString (text=0x7fffec61aed0 " map aggressor ", src=<value optimized out>) at cmd.c:1111
#7  0x0000000000434995 in Cbuf_Execute () at cmd.c:188
#8  0x000000000046d84e in Host_Init () at host.c:1021
#9  0x000000000046d87b in Host_Main () at host.c:539
#10 0x0000000000402050 in main (argc=5, argv=0x7fffec623078) at sys_linux.c:254


Can you please do:
(gdb) print *datapointer

It should show what DP was trying to tokenize. Either that will help reproducing it, or it is a hardware bug.

divVerent wrote:Can you please do:
(gdb) print *datapointer

It should show what DP was trying to tokenize. Either that will help reproducing it, or it is a hardware bug.

Here it is:

Code: Select all

(gdb) print *datapointer
$1 = 0x2b4bc0fa2e7c <Address 0x2b4bc0fa2e7c out of bounds>

That's unfortunately not really useful... try instead:

(gdb) up
(gdb) print num_tokens
(gdb) print tokens[0]
(gdb) print tokens[1]

Code: Select all

Core was generated by `./nexuiz-linux-686-dedicated-debug +exec spacectf.cfg'.
Program terminated with signal 11, Segmentation fault.
#0  0x08113350 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81836bc "customizeentityforclient: NULL function")
    at prvm_exec.c:508
508             f = &prog->functions[fnum];
(gdb)


Code: Select all

(gdb) bt full
#0  0x08113350 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81836bc "customizeentityforclient: NULL function")
    at prvm_exec.c:508
        st = <value optimized out>
        startst = <value optimized out>
        f = <value optimized out>
        newf = <value optimized out>
        ed = <value optimized out>
        ptr = <value optimized out>
        jumpcount = <value optimized out>
        cachedpr_trace = <value optimized out>
        exitdepth = <value optimized out>
        restorevm_tempstringsbuf_cursize = <value optimized out>

#1  0x08135b33 in SV_BuildEntityState (cs=0x0, ent=0x12ec2d1c, enumber=294)
    at sv_main.c:830
        i = <value optimized out>
        tagentity = <value optimized out>
        modelindex = <value optimized out>
        effects = <value optimized out>
        flags = <value optimized out>
        glowsize = <value optimized out>
        lightpflags = <value optimized out>
        specialvisibilityradius = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        customizeentityforclient = 520
        f = 0
        cullmins = {0, 0, 0}
        cullmaxs = {0, 0, 0}
        model = <value optimized out>
        val = <value optimized out>
#2  0x08135f64 in SV_BuildEntityState (cs=0x0, ent=0x12ec2d1c, enumber=294)
    at sv_main.c:937
        i = <value optimized out>
        isbmodel = <value optimized out>
        i = <value optimized out>
        tagentity = 294
        modelindex = 55
        effects = 4194848
        flags = <value optimized out>
        glowsize = 0
        lightpflags = <value optimized out>
        specialvisibilityradius = 0
        customizeentityforclient = <value optimized out>
        f = 0
        cullmins = {0, 0, 0}
        cullmaxs = {0, 0, 0}
        model = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        val = <value optimized out>
#3  0x08135f64 in SV_BuildEntityState (cs=0x0, ent=0x12ec2d1c, enumber=294)
    at sv_main.c:937
        i = <value optimized out>
        isbmodel = <value optimized out>
        i = <value optimized out>
        tagentity = 294
        modelindex = 55
        effects = 4194848
        flags = <value optimized out>
        glowsize = 0
        lightpflags = <value optimized out>
        specialvisibilityradius = 0
        customizeentityforclient = <value optimized out>
        f = 0
        cullmins = {0, 0, 0}
        cullmaxs = {0, 0, 0}
        model = <value optimized out>
        val = <value optimized out>



is it enough ?

Here it is:

Code: Select all

#0  0x000000000043a9da in COM_ParseToken_VM_Tokenize (datapointer=0x7fffec61a890, returnnewline=0) at common.c:1147
1147            for (;*data <= ' ' && ((*data != '\n' && *data != '\r') || !returnnewline);data++)
(gdb) up
#1  0x00000000004be255 in VM_tokenize () at prvm_cmds.c:1983
1983            while(COM_ParseToken_VM_Tokenize(&p, false))
(gdb) print num_tokens
$1 = 1
(gdb) print tokens[0]
$2 = 333591
(gdb) print tokens[1]
$3 = 325613

I'll come to IRC if this isn't enough information.

another crash...

Code: Select all

Core was generated by `./nexuiz-linux-686-dedicated-debug +exec spacectf.cfg'.
Program terminated with signal 11, Segmentation fault.
#0  0x0811337c in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81836bc "customizeentityforclient: NULL function")
    at prvm_exec.c:519
519             st = &prog->statements[PRVM_EnterFunction (f)];


Code: Select all

(gdb) bt full
#0  0x0811337c in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81836bc "customizeentityforclient: NULL function")
    at prvm_exec.c:519
        st = <value optimized out>
        startst = <value optimized out>
        f = (mfunction_t *) 0x13d3fb34
        newf = <value optimized out>
        ed = <value optimized out>
        ptr = <value optimized out>
        jumpcount = <value optimized out>
        cachedpr_trace = <value optimized out>
        exitdepth = 0
        restorevm_tempstringsbuf_cursize = 0



It would help immensely if you could reproduce this crash in a debug build (make debug in the engine sources, then run it using ./darkplaces-glx -nexuiz), the variable values would show up then.

My guess is that prog is NULL.

muy server seems to be running fine btw.. vanilla debian 4 with nexuiz running from downloaded zip, not from reps.

I'm using nexuizdebug20070531d_2.3.zip with enginesource20070531.zip (from nexuizsource20070531d_2.3.zip)
Should I use another build ?

Just for the sake of completeness: the new DCC box is a dual-core CPU (Opteron 1212).

Ronan wrote:I'm using nexuizdebug20070531d_2.3.zip with enginesource20070531.zip (from nexuizsource20070531d_2.3.zip)
Should I use another build ?

nexuizdebug is actually a release build with debug info, so stuff is still optimized out. It mainly is to be used to find crash locations from the release build, which is the debug build after stripping.

For real debugging, you should compile your own build. Inside the enginesource, type "make sv-debug && ln -snf darkplaces-dedicated nexuiz-dedicated" and use that one.

Sxen wrote:Just for the sake of completeness: the new DCC box is a dual-core CPU (Opteron 1212).

mine is an a64x2

With the debug build :

Code: Select all

Core was generated by `./nexuiz-dedicated +exec spacectf.cfg'.
Program terminated with signal 11, Segmentation fault.
#0  0x0814d828 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81dd9c0 "customizeentityforclient: NULL function") at prvm_exec.c:519
519             st = &prog->statements[PRVM_EnterFunction (f)];



Code: Select all

(gdb) bt full
#0  0x0814d828 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81dd9c0 "customizeentityforclient: NULL function") at prvm_exec.c:519
        st = (dstatement_t *) 0x0
        startst = (dstatement_t *) 0x0
        f = (mfunction_t *) 0x12df856c
        newf = (mfunction_t *) 0x0
        ed = (prvm_edict_t *) 0x0
        ptr = (prvm_eval_t *) 0x0
        jumpcount = 0
        cachedpr_trace = 0
        exitdepth = 0
        restorevm_tempstringsbuf_cursize = 0


new crash :

Code: Select all

#0  0x08183842 in SV_BuildEntityState (cs=0x0, ent=0x12cdad84, enumber=164)
    at sv_main.c:825
825             customizeentityforclient = PRVM_EDICTFIELDVALUE(ent, prog->fieldoffsets.customizeentityforclient)->function;



Code: Select all

(gdb) bt full
#0  0x08183842 in SV_BuildEntityState (cs=0x0, ent=0x12cdad84, enumber=164)
    at sv_main.c:825
        i = 0
        tagentity = 0
        modelindex = 0
        effects = 0
        flags = 0
        glowsize = 0
        lightstyle = 0
        lightpflags = 0
        light = {0, 0, 0, 0}
        specialvisibilityradius = 0
        customizeentityforclient = 0
        f = 0
        cullmins = {0, 0, 0}
        cullmaxs = {0, 0, 0}
        netcenter = {0, 0, 0}
        model = (model_t *) 0x0
        val = (prvm_eval_t *) 0x0


With so many crashs I suppose it's a hardware problem...

Ronan wrote:With the debug build :

Code: Select all

Core was generated by `./nexuiz-dedicated +exec spacectf.cfg'.
Program terminated with signal 11, Segmentation fault.
#0  0x0814d828 in PRVM_ExecuteProgram (fnum=520,
    errormessage=0x81dd9c0 "customizeentityforclient: NULL function") at prvm_exec.c:519
519             st = &prog->statements[PRVM_EnterFunction (f)];



I'm unsure how it is possible for this line to crash, prog is valid (considering it got past the fnum check above), &prog->statements[] can't crash in any way I can imagine, PRVM_EnterFunction is not inlined in a debug build so it can't be crashing on f in any way.

This takes more inspection of the variables at the time of the crash.

Please paste the results of these commands:
print prog
print prog->statements
print prog->statements[0]
print f
print *f

The DCC's servers are running a debug binary build with optimization turned off since five days now and there has not been a single crash.

Cheers.

Nil wrote:The DCC's servers are running a debug binary build with optimization turned off since five days now and there has not been a single crash.

Cheers.

So it is optimization related? That can't be good.

What architecture is the binary, and do you have any ideas on when the crash occurs?

LordHavoc wrote:So it is optimization related? That can't be good.

What architecture is the binary, and do you have any ideas on when the crash occurs?

Since my last message there have been 3 crashes at server start (the servers are getting killed and restarted each morning by a cron script). They are started within "screen" (screen manager with VT100/ANSI terminal emulation). Maybe this could be a problem?

The pattern is the same as before. But still no crash once the servers are up and running.

The binary has been built with "make sv-debug" from the Nexuiz 2.3 source with "CPUOPTIMIZATIONS=" in makefile.inc (rest not changed). So I guess the architecture its generic 386.

I have pasted the backtrace to http://paste.debian.net/33770.

Common.c is modified as suggested bei Divverent by adding the following lines at top of COM_ParseToken_VM_Tokenize in order to find out more about the data whose processing causes the segmentation fault:

Code: Select all

static const char *string_to_tokenize;
string_to_tokenize = *datapointer;

Printing "string_to_tokenize" in gdb just shows that the address is out of bound, so this did not help much.