pixel shader computation: precision problem?

Hi,

I do the following computation in a pixel shader:

const float texel = 1.0 / 128.0;

result = floor(In.texcoord.w / 16.0) / 64.0 + texel

In.texcoord.w containes an integer value between 0 and 299, stored as a float, interpolated over the triangle, where each vertex stores the same value, so in practice there shouldn't be any interpolation happening.

When I output result as a color, I notice distortions over the triangle and a flickering when I move the mouse, for some values of In.texcoord.w. But actually the result should be constant for each of the triangles pixels because the computation uses the same values for each pixel - at least it should.

Any idea of what causes this Precision issues Is there any remedy

Nico

Answer this question

pixel shader computation: precision problem?

Jean-Francois Peyroux
Thanks. I just didn't know. That was probably to simple to pop into my mind... lol ;-)

By the way, do you know why the round function is so slow
shango
Why don't you use floor(In.texcoord.w + 0.5) That's the standard way to do rounding.
loukia
Well, yes, but I have to do some updates of my programming environment for that. I'm still using the VS2005 beta... lol

I plan to do this, but probably in a few weeks, so up to then I have to find another solution. I implemented a convenient CPU environment for executing hlsl pixel shaders. I could use that for debugging, but it will not reproduce the GPU precision issues.
OffThaTop
As you have this problem with RefRast too you can try to use the shader debugger.
Prajkta
I use a GeForce 7800 GTX and a Shader 3.0. There is no change when I use the RefRast. The mouse move thing is strange, but I'm not aware of anything that could change the rendering due to mouse movements. Had this phenomenon with other computations before though. I output the w value divided by 300 and it seems alright.
pramodh
Your debugging project sounds interesting!

I just found the cause to the distortion. As I suspected it was due to imprecision during interpolation. When I round the values of In.texcoord.w before I use them the problem disappears.

Unfortunately rounding is a costly operation. I do it using this code:

float tmp = frac( In.texcoord.w ) ;

if ( tmp > 0.0 )

{

if (tmp < 0.5) tmp = floor(In.texcoord.w);

else tmp = ceil(In.texcoord.w);

}

else tmp = In.texcoord.w;

Then I use tmp in my computation. When I use the hlsl round fuction my frame count drops even more...

Anyway, thanks for your ideas Ralf. ;-)
signy
Yes, equal inputs should get equal outputs but GPUs are nor CPU and care less about floating point accuracy.

Maybe some additional information about the shader profiles and hardware you are using could be helpful.

Have you try this with RefRast

You say that your mouse moves caused flickering. Do you change anything depending on the current mouse position

Maybe you should try to output your W coordinate to a F32 render target to check if the input is already wrong.
MADLAX
The debugger use Refrast and can’t show you GPU precision issues too because it runs on the CPU. I was playing around with a shader debugger that runs on the GPU. It splits the shader in parts and stores the intermediate results in FP32 render targets that are transferred to the CPU to show them for every pixel. But unfortunately this is only one of my playtime projects and not ready to use.
pixel shader computation: precision problem?

Answer this question