[Note: added external link to log output]

In any rc 151 and Lord Havoc's snapshots on Icculus, GLSL is completely broken in Nexuiz on Mac OS.

Previously in both 1.5 and 1.2.x (and nexuizengine snapshots February 2006 and earlier) only the shadows were broken when GLSL was enabled in Nexuiz (and by the shadows being broken I mean half of the realtime shadows coming from a dlight such as weapon effect or rocket flare etc. were not rendered so the shadows look like "broken glass").

Now, GLSL enabled produces nothing but the skybox background and weapons fire [shotgun pellets for example]. No textures, no models (players or weapons) etc. - just black screen with skybox background and the projectiles from weapons :(

To be specific these options:

Code: Select all

r_shadow_glsl "1"
r_shadow_realtime_dlight "1"
r_shadow_realtime_world_dlightshadows "1"
r_shadow_realtime_world "0"
r_shadow_realtime_world_shadows "0"

The world shadows are too slow for my setup, the dlight are too slow without GLSL - really, really it is needed for the dlights to be usable, or things like the "Quad" drag the FPS to %50 of what it should be.

So to reiterate: previously I disabled dlight and dlightshadows and enabled GLSL, because of the dlightshadows looking broken [like shards of glass]. This is no longer enough in the 151 rc's and the latest engine snapshots, because even with dlights and slightshadows disabled, enabling GLSL makes the entire screen black with no textures and no models being rendered (only 'skybox' and weapon-fire).

To me this is just unacceptable, the latest engine is fast, very very fast compared to the previous - I can see a LOT of work being done to it, but it's just sad that this problem exists for Mac OS and I want to give as much feedback as possible to see it hopefully corrected.

System specs:
Mac OS X 10.4.5 all updates
Dual-Core G5 @ 2Ghz
NVidia 7800 GT 256MB PCI-Express
2GB DDR2 (667) Ram

I have noticed there is no GL_EXT_texture3D implementation for the NVidia drivers on OS X, I thought this was pretty standard and was surprised it was missing (well standard for Linux/Windows/Mesa/etc.).

Console output gives errors finding shaders and one about DrawQ_ResetClipArea: not in 2d, and here's a link to the console log with developer option enabled:
qconsole.log.gz

Anyone on OS X with an ATI video card having same issues or is this NVidia specific?

TYIA

Could you run with the commandline -developer and post the log somewhere (a nopaste site would be fine) and give me the link (it would be a bit big for a forum post)? by default it does not print the reason the warnings/errors from the shader compilation process, so I need you to enable -developer mode to enable them.

Thanks.

P.S. in the mean time you can play with r_glsl 0, or -nofragmentshader on the commandline, but as you know this is slower.

[edit: requested log added to parent post]

Not sure if you have a Mac accessible to you in any fashion, but the "OpenGL Shader Builder" that comes with the Developer's Tools would probably be of help if more insight is needed.

If there's something else needed let me know.

Thanks.

Yikes.

That driver is completely broken, it's not honoring the #ifdef directives which determine which part of the file to compile as vertex shader (geometry processing) and which part of the file to compile as fragment shader (pixel processing), so it's trying to compile both parts of the file into the vertex shader, which causes internal conflicts.

The shaders use a LOT of #ifdef directives to reduce maintenance (otherwise it would take, literally, 3000 different shader scripts, or 30 or so which would run much slower than necessary), so they simply won't work with a broken driver like that!

I have only two options on this, one is modifying the engine to do the work the driver should be doing (#ifdef stuff), the other is leaving it broken.

I bought the OpenGL Shading Language book, and it is quite clear on the full-featured nature of the OpenGL Shading Language, this driver is broken.

Please notify Apple on this issue, it's not my fault

P.S. it works fine on other NVIDIA drivers (windows x86, windows x64, Linux x86, Linux x86_64, freebsd x86 are known to work), the MacOS NVIDIA drivers are maintained primarily by Apple, so the ball is in their court, not NVIDIA's.

Do you know if it is in the ARB spec that the driver must honor these conditionals? I would assume it is, but honestly if I am going to file a bug report that has any chance I need to know some things.

I don't blame you for this problem - you may be interested to know that since owning this machine (only a few months btw) this is most certainly not the first bug I have come across, including a rather dumb regression causing an endless driver restart (until the pre-defined reload limit) upon which being greeted with the "Black Menu of Death". Anyone who has seen this knows what I am talking about, but that program was the SolarWinds screensaver port to OS X (not Nexuiz!).

So to get back on track do you happen to know if it's in the spec to support that? What I need to submit is pretty much covered in this mail from an Apple engineer: http://lists.apple.com/archives/Mac-opengl/2004/Nov/msg00088.html

Would you be so kind as to post a very basic code snippet I may use as an example of what fails, please?

Also how would you describe this error in technical terms from your point-of-view (being an OpenGL developer)? For example I would describe this issue as the NVidia OpenGL driver on OS X not respecting the conditional directives in a C project using GLSL code: the fragment shaders get clobbered (interpreted as vertex routines) regardless of the conditionals in the C code. Then I'd insert a small code example. Is this an accurate description in your opinion?

FYI I actually wanted an ATI card, because I knew about the lack of direct involvement by NVidia with the Apple GL drivers for their cards (including some other reasons). Unfortunately, there were no and still are no ATI video cards for PCI-Express based Apple G5 PowerMacs.

Thanks for your time, I really appreciate it!

ds01 wrote:Do you know if it is in the ARB spec that the driver must honor these conditionals? I would assume it is, but honestly if I am going to file a bug report that has any chance I need to know some things.

http://oss.sgi.com/projects/ogl-sample/registry/ARB/GLSLangSpec.Full.1.10.59.pdf - Section 3.3 describes the preprocessor directives supported in the GLSLang and their exact behavior, DarkPlaces uses only a few of them.

ds01 wrote:I don't blame you for this problem - you may be interested to know that since owning this machine (only a few months btw) this is most certainly not the first bug I have come across, including a rather dumb regression causing an endless driver restart (until the pre-defined reload limit) upon which being greeted with the "Black Menu of Death". Anyone who has seen this knows what I am talking about, but that program was the SolarWinds screensaver port to OS X (not Nexuiz!).

Strange.

ds01 wrote:So to get back on track do you happen to know if it's in the spec to support that? What I need to submit is pretty much covered in this mail from an Apple engineer: http://lists.apple.com/archives/Mac-opengl/2004/Nov/msg00088.html

Yes it's in the spec.

ds01 wrote:Would you be so kind as to post a very basic code snippet I may use as an example of what fails, please?

Code: Select all

const char *shaderstring =
"#define TESTING1\n"
"#ifdef TESTING1\n"
"void main(void) {}\n"
"#endif\n"
"#ifdef TESTING2\n"
"void main(void) {}\n"
"#endif\n";
int vertexshaderobject = glCreateShaderObjectARB(GL_VERTEX_SHADER_ARB);
glShaderSourceARB(vertexshaderobject, 1, &shaderstring, NULL);
glCompileShaderARB(vertexshaderobject);
int vertexshadercompiled;
glGetObjectParameterivARB(vertexshaderobject, GL_OBJECT_COMPILE_STATUS_ARB, &vertexshadercompiled);
char compilelog[65536];
qglGetInfoLogARB(vertexshaderobject, sizeof(compilelog), NULL, compilelog);
printf("vertex shader compile log:\n%s\n", compilelog);


If working correctly the first main() should be compiled into the vertex shader, and the second main() should be ignored, however the observed behavior is that BOTH are compiled into the vertex shader, causing errors in the compilelog.

ds01 wrote:Also how would you describe this error in technical terms from your point-of-view (being an OpenGL developer)? For example I would describe this issue as the NVidia OpenGL driver on OS X not respecting the conditional directives in a C project using GLSL code: the fragment shaders get clobbered (interpreted as vertex routines) regardless of the conditionals in the C code. Then I'd insert a small code example. Is this an accurate description in your opinion?

Not quite an accurate interpretation, the conditional directives are precompiler directives in the GLSL code, not the C code. GLSL is a very very C-like language.

ds01 wrote:FYI I actually wanted an ATI card, because I knew about the lack of direct involvement by NVidia with the Apple GL drivers for their cards (including some other reasons). Unfortunately, there were no and still are no ATI video cards for PCI-Express based Apple G5 PowerMacs.

Funny how that works, on PC NVIDIA is respected for their superior drivers (ATI's are buggy putting increased strain on developer relations), on Mac ATI is respected because Apple writes their Mac drivers.

ds01 wrote:Thanks for your time, I really appreciate it!

No problem.

ds01 wrote:Unfortunately, there were no and still are no ATI video cards for PCI-Express based Apple G5 PowerMacs.

Isn't there any way to flash-update a PC ati card, so that it can work with a mac ?
There are ati cards for AGP based G5 (i've got one), so perhaps...

Havoc,

Is there a way to dump the shaders [to the console and thus the log] before they're fed to the backend compiler (or before they're compiled), so I can verify that the defines that are supposed to be "fed" into them are? In this way I can verify if the driver is ignoring the conditionals in the GLSL code [string], or if something else is going on (like for some reason the defines aren't being placed into the shaders before being compiled).

Thanks.

lol Obi Wan, think I'll wait until they EOL the G5 powermacs and see if ATI ever releases a card for them before I go to such drastic measures

ds01 wrote:Havoc,

Is there a way to dump the shaders [to the console and thus the log] before they're fed to the backend compiler (or before they're compiled), so I can verify that the defines that are supposed to be "fed" into them are? In this way I can verify if the driver is ignoring the conditionals in the GLSL code [string], or if something else is going on (like for some reason the defines aren't being placed into the shaders before being compiled).

Thanks.

Sure.

In GL_Backend_CompileProgram which is at gl_backend.c:715, add something like this:

Code: Select all

if (developer.integer >= 100)
{
int i;
Con_Printf("Compiling shader:\n");
if (vertexstrings_count)
{
Con_Printf("------ VERTEX SHADER ------\n");
for (i = 0;i < vertexstrings_count;i++)
Con_Print(vertexstrings_list[i]);
Con_Print("\n");
}
if (fragmentstrings_count)
{
Con_Printf("------ FRAGMENT SHADER ------\n");
for (i = 0;i < fragmentstrings_count;i++)
Con_Print(fragmentstrings_list[i]);
Con_Print("\n");
}
}


(I've just now commited this to the darkplaces engine cvs)

Note that -developer on the commandline sets developer to 100, so the same commandline should now print this to the log.

Also note that GLSL takes a whole list of strings to compile, so darkplaces sends the #define lines as the first strings, then the actual shader source is the last string.

I am having trouble uploading the logs to that site, so I'll have to try again later. If you want me to email them to you, my email is on record or just pm me.

Here is a code snip from the first log - unmodified snapshot with the default GLSL code mashed together (both fp/vp).

At first I had forgotten that I split the vertex and fragment code (duplicated the shared code so it's in both), so there's a second log -these things are huge so not spamming the forum.

I just grabbed the first shader output + compile from the log, the code clip here is unedited from it's start to end. It only shows one fp compile after spitting out one each of vp then fp [then the one fp log], but as you see it does include the defines. Why all the logs, in the entire file even - do the same thing: one vertex shader + one fragement shader = one fragment compile log? It sending one each from the list, sequentially, but both to compile as fp? I saw the same thing in the other log too.

[edit: removed the code block - (link to the log is in a post below)]

I think you mean one vertex shader compile log.

The reason it's not printing a fragment shader compiler log is because the vertex shader failed to compile so it didn't even bother trying to compile the fragment shader (which would not work without the vertex shader).

Without going into a lot of detail about GLSL compilation, I'll just say that DarkPlaces makes two shader objects (one for vertex shader, one for fragment shader) and links them to make a program object (the thing that I bind for rendering), the error encountered here would occur on either the vertex shader or fragment shader, whichever is compiled first, because the compiler is simply ignoring the #ifdef statements, so all code for all cases is included at once (which won't work).

Yes I meant vertex compile log. The reason I interpreted the output in the way that I did is for a couple of reasons.

In the error log the first error that main already has a body, well that does not happen if you split the 'builtinshaderstring' into [i.e.] builtinvpstring and builtinfpstring in gl_rmain.c. So if it does it like I think you are saying it does, then it should still have the main error right?

Also, when I split the strings like that it didn't show compile logs for vertex only for fragment. If the conditionals were not working then a main error would be reported, because as you obviously already know there's that "Easter Egg" conditional which would have caused a double main error (but there was no such main err in the fp log).

There must be something else going on here, if the conditionals were broken there would also be a double main err. Well in that case of spliting vp and fp there is no double main err in the fp compile log.

Managed to get the logs uploaded.
The second one qconsole.2.ds.log.gz is the one with split strings.

New Logs

ds01 wrote:Managed to get the logs uploaded.
The second one qconsole.2.ds.log.gz is the one with split strings.

New Logs

They used to be split in the first place, I merged them to reduce maintenance.

I have greatly simplified the shader yesterday (a lot less #ifdefs and #if defined stuff) and could split it into two files again if you prefer, I merged it to make it more of a unit, since the two parts are made for eachother.

What is certain is that this is a driver bug, we're just not entirely sure what kind of driver bug.

I know they used to be split and realize it must be easier to manage merged I did that to try and narrow the field of possibilites to more easily find the problem over here.

Since there were no GLSL errors printed out to console when using builds before you merged the fragment and shader code and changed the backend compiler, etc. etc. like said before I know a LOT has been done since then, that's why I've been trying different things in an attempt to find the exact issue.

I'm interested in the more simplified GLSL block you mentioned, so I'll keep checking cvs to see if you commit those. In the meantime just got an idea...

Thanks again for your time and patience

Oh I forgot I hadn't committed that yet, I had gotten caught up in a debate over whether USESPECULAR should be optional or always on, ended up deciding to keep it optional, but was too tired to commit it at the time.

Committed now.

Hey it's a lot more simplified hehe.

This funky driver still freaks out on the merged shaderstring (doesn't stop at last VP #endif), so I split it again and then got an error which looks like it gets confused lower down in the fragment shader (it never gets the tempcolor).

Odd you should mention whether or not to make USESPECULAR default, because if I force that to always be enabled (comment out the leading/trailing conditionals around those in the FP) then both the VP and the FP programs compile... but they don't link

Do I need to change something in the code so that the backend compiler knows how to use the compiled objects, or does this look like another driver bug to you?

Code: Select all

^7program link log:                                                                                                                                 
^7WARNING: Vertex shader writes varying 'TexCoordLightmap' which is not read by fragment shader.
^7WARNING: Vertex shader writes varying 'EyeVector' which is not read by fragment shader.
^7ERROR: Fragment shader reads varying 'CubeVector' which is not written by vertex shader.


Thanks.

ds01 wrote:Do I need to change something in the code so that the backend compiler knows how to use the compiled objects, or does this look like another driver bug to you?

You could make it always calculate CubeVector, although this would be quite pointless for lightmapped surfaces.

Good luck, I'm not sure the driver will even render it right if it's having this much trouble compiling it

You must have given me a lucky clover lol, because I got it working. The bug is the conditionals where it doesn't skip the block when it should even though it stores them. I know it stores/recognizes them, because if you add extra checks then it parses and compiles everything ok.

So I dunno if you want to include this - all I did was add extra checks. Of course it should also work on drivers that actually obey the ARB spec (Yes I am going to go file bug report and report it on the list, not sure when/if it will be fixed.)

Here's a patch against the darkplaces cvs:
gl_rmain.c.extrachecks.diff.gz
[edit: patch updated rev. 2]

Just now I realize bloom and fog won't work with that. Bloody maddening :-\

[edit: rev. 2 patch bloom works]

ds01 wrote:Just now I realize bloom and fog won't work with that. Bloody maddening :-\

That might not be entirely your fault. The neon tubes on nexdm08 are broken with current cvs builds & r_glsl enabled.
However bloom (on health packs and viewmodel) and fog are both working.

KadaverJack wrote:That might not be entirely your fault. The neon tubes on nexdm08 are broken with current cvs builds & r_glsl enabled.
However bloom (on health packs and viewmodel) and fog are both working.

Thanks a lot for telling me this

I thought bloom wasn't working, so I presumed fog also (don't know where fog is?) - the "Glow Arena" is what I used to double-check for bloom. (They look like plastic tubes without bloom on them lol.) Did you notice that if you increase the bloom intensity way too high, then you can see the bloom coming from the tubes in the glow arena (still not coloured correctly though)?

Updated the patch in the post above - removed the tempcolor vec and cleaned it up a bit, also added a force-check for the last block in the fp shadercode. Bloom was partly working before, but in this patch it should really be working.

KadaverJack do you know if nexdm03 (the map with the glowing green 'plasma' tubes) is supposed to work with cvs, because it doesn't for me with GLSL on. The effect is strange - entire bloom's are shifting like the psychadelic underwater effect from the original glquake 1 - causing the map to be unplayable. I usually don't have it in rotation, but I ran through all the maps once to look at things (lots of new stuff in release candidate!) and all the others are playable.

Fog isn't used in Nexuiz, it can be turned on with this console command:
fog 0.1 0.1 0.1 0.1

That makes a somewhat dark grey fog fill the level, pretty densely, the parameters are density red green blue.

Naturally the fog can be set in any map that wishes to do so, but it disables skybox (because it's global fog and the skybox is infinitely far away... so logic dictates it hides the skybox).

P.S. the bloom doesn't use a shader, it's stock OpenGL 1.1 blending and texture copying stuff, it could be sped up a bit with a shader (texture fetches are cheaper than pixel blends) but I haven't gotten around to it, it works on any card, even an old TNT, but then pretty much everything in DP works on even the oldest cards

LordHavoc wrote:P.S. the bloom doesn't use a shader

I associated "bloom" with "glow" in my mind, it's easy to confuse variables in an engine with so many effects

ds01 wrote:KadaverJack do you know if nexdm03 (the map with the glowing green 'plasma' tubes) is supposed to work with cvs, because it doesn't for me with GLSL on.

Yes, those don't work either.
This is how they look here:

KadaverJack wrote:

ds01 wrote:KadaverJack do you know if nexdm03 (the map with the glowing green 'plasma' tubes) is supposed to work with cvs, because it doesn't for me with GLSL on.

Yes, those don't work either.

That's insightful, but I'm still not sure how r_glsl 0 would be behaving any differently from my looks at the code.

KadaverJack, have use tried r_glsl_deluxemapping?

It looks like that here also, but there is a more prominent issue - it seems the effects are on a different "plane" as the textures/models? Using the latest rc pk3 data, this only happens with nexdm03. Using the data from cvs in addition, then nexdm10 also shows this "multi-planar" effect (lava level). The diff plane moves in a swaying motion like a boat effect. Also causes the text to go messy and the 'real' plane with the textures/models twigs out well shimmer would not be the correct term - sort of looks like it's not in tune only very very slightly (like out of sync HSync TV) and not swaying like the effect planar.

That only happens in certain areas of those maps, near green plasma in dm3 and red lava in dm10 - but thing is, if I turn away from those texture places and i.e. look at plain wall or go someplace else without those then everything (text/effects/sync) goes normal?!

Like I said before the lava map (dm10) does not do this with just rc data even with r_glsl=1, it does it with cvs data added. BUT you can make it do it with just rc data, by forcing r_glsl_deluxemapping=2. Could it be that those textures when deluxemapped make the engine freak out? (using the cvs + patch so I can use GLSL). I don't know how a deluxemap is generated, but if those textures are removed before the deluxemap is created then it would probably fix the problem right - does anyone else see the same thing here (please tell me it is not just me). I noticed all the maps are getting deluxedmap conversions into cvs, so this could be a big problem if more maps start using methods of that sort for textures.

This link following, the server has per-hour b/w limit, so be warned if you bang on refresh you hose it for everyone:
Link to screenshots of the issue in nexdm03

ds01 wrote:KadaverJack, have use tried r_glsl_deluxemapping?

Yes, it's on by default and the map was from svn, already recompiled with deluxemaps.
Of course it is only enabled on the r_glsl 1 shot...

Link to screenshots of the issue in nexdm03

Now that looks really weird...

ds01 wrote:Link to screenshots of the issue in nexdm03

Looks like an offset bloom texture, I wonder what would be messing with the bloom texturing? Does it scroll around? If so it's probably the driver ignoring one of the glLoadMatrixf calls on the GL_TEXTURE matrix, and thus leaving it in a bad state when it advances to rendering other multitexture stuff like applying bloom (r_textureunits 1 would probably hide this artifact by using a multipass method to blend the final bloom texture onto the screen).

Regarding the glow problems, I just realized why that happens on deluxemapping, but am not sure of a solution at the moment. The issue is that the q3map2 deluxemaps are modelspace (which I hate), so the rendering needs the deluxemapping to be disabled on the vertex lit stuff, or at the very least it needs a better direction vector than the non-existent one it's using on vertex-lit stuff.

Basically, deluxemapping doesn't affect vertex lit surfaces, and those are.

We might have an easy way out because those surfaces probably have the NOLIGHTMAP flag set on them, thus being batched in a separate texture chain, if so that could disable deluxemapping on the batch, but we don't know for sure if that is always the case without doing some tests or research on the data.