Server Crashes

[tChr]

all my servers run in screen. not running in a screen would require a BG job wich would have to be niced, or it would be short on recources.

[LordHavoc]

The binary has been built with "make sv-debug" from the Nexuiz 2.3 source with "CPUOPTIMIZATIONS=" in makefile.inc (rest not changed). So I guess the architecture its generic 386.

Actually it is x86_64 - the bt full output contains many 48bit addresses. I asked only because it narrows down the conditions to test for optimizer bugs, but you are also reporting this in debug builds which are immune to such bugs.

The most helpful information you could give is the stdout output (from the game itself) after issuing this command in gdb:
print Host_Error("")

That command will call the Host_Error function, aborting the code execution immediately, and printing out state information about the QuakeC VM, such as which function is currently executing (the most important tidbit of information we're missing at the moment).

[Sxen]

Code: Select all

[2007-08-01 09:38:08] "maxplayers" set to "6"
[2007-08-01 09:38:08] Server using port 26006
[2007-08-01 09:38:08] Server listening on address 0.0.0.0:26006
[2007-08-01 09:38:08] Loaded maps/aggressor.ent

Program received signal SIGSEGV, Segmentation fault.
0x0000000000454550 in COM_ParseToken_VM_Tokenize (datapointer=0x7fffd1035400, returnnewline=0) at common.c:1149
1149            for (;*data <= ' ' && ((*data != '\n' && *data != '\r') || !returnnewline);data++)
(gdb) print Host_Error("")
[2007-08-01 09:38:19] Host_Error:
[2007-08-01 09:38:19] Quake Error: Host_Error:

Program exited with code 01.
The program being debugged stopped while in a function called from GDB.
When the function (Host_Error) is done executing, GDB will silently
stop (instead of continuing to evaluate the expression containing
the function call).

Hm.

[Sxen]

I can reproduce the crash quite reliably by starting the code repeatly, it happens about once per 5 starts.

Olli

[LordHavoc]

Hm.

OK, that's kind of odd... Try print PRVM_Crash() instead, that shouldn't abort the program but should print the VM state.

[Nil]

Next try:

Code: Select all

[2007-08-02 19:31:37] Loaded maps/aggressor.ent

Program received signal SIGSEGV, Segmentation fault.
0x0000000000454550 in COM_ParseToken_VM_Tokenize (datapointer=0x7fff31f53300, returnnewline=0) at common.c:1149
1149            for (;*data <= ' ' && ((*data != '\n' && *data != '\r') || !returnnewline);data++)
(gdb) print PRVM_Crash()
[2007-08-02 19:31:51] QuakeC crash report for server:
[2007-08-02 19:31:51] s16036: STORE_F    GLOBAL1, GLOBAL10
[2007-08-02 19:31:51] s16037: STORE_F     (='), GLOBAL13
[2007-08-02 19:31:51] s16038: CALL4      strcat (=())
[2007-08-02 19:31:51] s16039: STORE_S    GLOBAL1,
[2007-08-02 19:31:51] s16040: ADD_F      GLOBAL3783, GLOBAL250, GLOBAL3783
[2007-08-02 19:31:51] s16041: GOTO       s16028

Program received signal SIGSEGV, Segmentation fault.
0x000000000051b7c8 in PRVM_ED_GlobalAtOfs (ofs=4) at prvm_edict.c:297
297             for (i=0 ; i<prog->progs->numglobaldefs ; i++)
The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use "set unwindonsignal on"
Evaluation of the expression containing the function (PRVM_Crash) will be abandoned.
(gdb) print Host_Error("")
[2007-08-02 19:31:51] s16042: STORE_F     (='dm_klzegypt''dm_fragcity''dm_revdm1''dm_geo-core''dm_starship''dm_aggressor''dm_skyway''dm_dieselpower''dm_zpdm01''dm_storm3dm3''dm_bloodprisonctf-1.0''dm_aneurysm''dm_kaznexdm1''dm_oilrig''dm_pukka3dm2''dm_79drdm5_beta2''dm_stormkeep''dm_slimepit''dm_bleach''dm_revdm3''dm_evilspace''dm_ame7q3dm3''dm_silvercity''dm_subhumandm1''dm_batcula''dm_darkzone''dm_bloodprison''dm_monolith''dm_runningman_1on1remix''dm_handsofgod''dm_farewell''dm_pushmoddm1''dm_warfare''dm_downer''dm_reslimed''dm_nexdmlc2''dm_final_rage''dm_toxic''dm_basement''dm_shdwdm1''dm_CMP1-dm6''dm_dmtower''dm_soylent''dm_79drgc2''dm_ruiner''dm_runningman'), Host_Error:
[2007-08-02 19:32:48] Quake Error: Host_Error:

Program received signal SIGSEGV, Segmentation fault.
0x00000000005656dd in SV_VM_Begin () at sv_main.c:2711
2711            prog->globals.server->time = (float) sv.time;
The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use "set unwindonsignal on"
Evaluation of the expression containing the function (Host_Error) will be abandoned.

[John Galt]

Hope you guys are on track to finding the bug that crashes the server. It's becoming quite a nuisance. I whipped up a couple cron scripts to check and see if server is running and start if not. I'll post them below if they're of any use.

There are two scripts. One is nexcheck.sh, the other startnex.sh... The names betray their purpose.

nexcheck.sh :

Code: Select all

#!/bin/bash
# Check for nexuiz servers via cronjob.  Start if not running.
# Aug 15 2007
# Mike Shade (John Galt) mshade@mshade.org / johngalt@mshade.org

CONFIGS="server-ctf server-min" # Name of config to check for.  server-ctf.cfg for example. Leave off .cfg extension
PS=`ps ax`
NEXDATA="/home/mshade/.nexuiz/data" # Where to store runfile
STARTNEX="/home/mshade/bin/startnex.sh" # Path to sister script startnex.sh

for server in $CONFIGS; do
        if echo "$PS" |grep -q "$server.cfg";
                then touch $NEXDATA/nexcheck.run
        else
                echo "$server not running. Starting $server ..."
                screen -R $server -X exec $STARTNEX $server
        fi
done


startnex.sh:

Code: Select all

[mshade@opteron bin]$ cat startnex.sh
#!/bin/bash
$NEXDIR=/home/mshade/Nexuiz  # path to Nexuiz root
cd $NEXDIR
exec ./nexuiz-linux-686-dedicated +exec $1.cfg


nexcheck.sh assumes that you have a screen session running with the name of your config file. My CTF server uses server-ctf.cfg, so I start an initial screen session like this:

Code: Select all

screen -S server-ctf


nexcheck.sh will restart the server using the existing screen sessions with the given names. I'll probably add a function to check for and/or start a relevant screen session, but for now that's manual.

[LordHavoc]

No current progress that I'm aware of, but the crash in question was on the maplist cycling, more specifically a piece of code that breaks it apart into the individual map names, but I'm not aware of any bugs in that code, I simplified it a bit today in the svn engine but I don't know if that will help or not.