Vertex shader problem

Hey,

Ive been trying to add a vertex shader to my engine(right now its just running a pixel shader). Currentlly if I activate the vertex shader the world gets drawn distorted(a small random red box with a triangle going out to infinity).

Heres the simple vertex shader im using.

struct VS_INPUT
{
    float3 position : POSITION;
    float4 color0    : COLOR0;
    float2 texcoord0 : TEXCOORD0;
};

struct VS_OUTPUT
{
    float4 hposition : POSITION;
    float4 color0    : COLOR0;
    float2 texcoord0 : TEXCOORD0;
};

VS_OUTPUT main( VS_INPUT IN,
                uniform float4x4 worldViewProj : c0 )
{
    VS_OUTPUT OUT;

    OUT.hposition = mul( worldViewProj, IN.position );
    OUT.color0    = IN.color0;
    OUT.texcoord0 = IN.texcoord0;

    return OUT;
}

Too calculate the worldViewProjection matrix im getting matrix values from D3DXMatrixLookAtLH and D3DXMatrixRotationYawPitchRoll. Then multiplying the matrixs together and passing them to the shader as a size 4 float with SetVertexShaderConstantF.

If it helps any im rendering with Quake 3 BSP maps.

Any Ideas

Thanks,
-Justin

Answer this question

Vertex shader problem

Tommy8890
Not work even in 'fixed pipeline'
Following is each process of the matrix setting.
You should make sure if the problem depends on vertex shader only.

// by fixed pipeline
pD3DDev->SetTransform( D3DTS_WORLD, &matWorld );
pD3DDev->SetTransform( D3DTS_VIEW, &matView );
pD3DDev->SetTransform( D3DTS_PROJECTION, &matProjection );
pD3DDev->SetVertexShader( NULL );

// by vertex shader (no use ID3DXEffect)
D3DXMATRIX matWVP = matWorld * matView * matProjection;
D3DXMatrixTranspose( &matWVP, &matWVP );
pD3DDev->SetVertexShaderConstantF( 0, (float*)&matWVP, 4 );
pD3DDev->SetVertexShader( pVertexShader );
MicahN
I tried it still doesn't work.
jebrown
Hi, maybe...
wrong> float3 position : POSITION;
correct> float4 position : POSITION; (position.w = 1.0f at default)
If wrong case, _41,_42,_43,_44 of the matrix will be ignored by being three dimensions.
Addition:
wrong> OUT.hposition = mul( worldViewProj, IN.position );
correct> OUT.hposition = mul( IN.position, worldViewProj );
Robert Franks
Please replace like below in trial,

before:
VS_OUTPUT VS( VS_INPUT In, uniform float4x4 worldViewProj : c0 )
after:
float4x4 worldViewProj : register(c0);
VS_OUTPUT VS( VS_INPUT In )
drfoxs13
This is always fun - there's a whole bunch of ways to get this wrong. I generally don't attempt to remember the right way to do things, I remember how to try all the ways, so I'll eventually find the right one. Seriously - you'd never guess I'd written real game engines, would you :-)

Anyway, things to try:

-Careful about what you do and don't call D3DXMatrixInverse on - I seem to recall that D3DXMatrixLookAtLH generates a ready-inverted matrix (i.e. it generates a matrix that transforms world->object, whereas something like D3DXMatrixRotationYawPitchRoll generates a matrix that transforms object->world). So you might not need to invert it.

-Reverse the order of arguments to the matrix multiplies. I always get this wrong at least five times.

-Transpose the matrix before you feed it to the VS constants (VS "mul" might not do what you expect - check out stuff like "#pragma pack_matrix").
He is Cool
You also need to concatenate a projection matrix. Take a look at D3DXMatrixPerspectiveFovLH().
fisher_iso
I am sorry I forgot to include that.

Heres my code

    // Calculate the world matrix
    D3DXMATRIX m_mView;
    D3DXMatrixLookAtLH( &m_mView, &viewport.m_vEye, &viewport.m_vLook, &viewport.m_vUp );
    D3DXMatrixInverse( &sys_data.g_matWorld, NULL, &m_mView );

    // caluculate the worldProjMatrix
    sys_data.worldViewProjection = sys_data.g_matWorld * m_mView * sys_data.m_matProjection;

    // Setup the perspective matix. Quake's axis are different so rotate accordinglly.
    D3DXMatrixPerspectiveFovLH( &sys_data.m_matProjection, 70.0*(D3DX_PI/180.0), 1.0f, 1.0f, 100000.0f );

And im passing it to the shader by this which calls SetVertexShaderConstantF.

R_SetParam(0,sys_data.worldViewProjection,4,PROGRAM_VERTEX);

Four is the size of the float that is being passed and 0 is the register.
Vertex shader problem

Answer this question