My specs are Athlon 62 x2 4400+ @ 2.7Ghz
2GB DDR2 RAM
8600GT with 256 vram.

I can start it Nexuiz normally but when i change the resolution to 1280x800, the screen turns into my desktop even though the application is running on top and only the sound can be heard. I tried reversing it but it gives me the same results.

What OS are you running?

You mean that now you can't 'really' start/play Nexuiz anymore?
Find the file config.cfg and rename it.. The file holds all the settings and if you rename it Nexuiz will use the defaults which should work again.
However i have no idea why that should happen. Does this resolution work when you choose it for windows? Maybe its a resolution your monitor does not supoort?

I'm using Vista Home Premium. I'm running it at 1280x800, my monitor's native resolution is 1440x900.

It managed to start up on default settings but when I change the resolution it hangs again

Edit: Managed to make it work. Can someone tell me the graphic settings that affect gameplay the most? I find it weird that my 8600GT which is way above the recommended graphic card having performance issues at ultimate,high settings. My 5 year old ATI 9600 Pro only did slightly worst. =/

The 8600 series was not particularly strong. Under some circumstances the 7600GT will match it. You won't be able to run max settings on that, you can't on much right now. I have a 9600GT which can't quite take ultimate without getting a little slow at some points.

I would recommend the native resolution over any other, the others will look blurry so any added detail level is partially wasted. The detail levels can then be adjusted to suit. I wouldn't bother with the ultimate detail level on any system. The highest reflection detail level isn't that noticeable above the lower ones so you can take that down a lot. Realtime dynamic lights and shadows can be offputting at times and slow things down a lot. Try without, you might prefer it.

Performance also varies a lot by map. Some maps are not well optimised in some areas. Any map with reflective surfaces that aren't in plane with each other will bring things to a crawl for instance.

Ultimate settings are really demanding on the hardware and activate really advanced features. At ultimate even my 8800gtx mobile has problems keeping up when there is much action on the screen. But i usually play at low settings.. where the card manages over 300fps for "timedemo demos/demo1"