Hi everyone!
I cant really wrap my head around this problem with so called depth-masking using shaders. My intention basically is to use it for whats called portal rendering, see image below:
http://answers.unity3d.com/storage/temp/21316-portal_effect.png
The idea is to have an arbitrary mesh serving as a window through which one can see objects from another scene, as seen from current camera viewpoint, while the rest of the original scene renders normally around this hole.
I have been using this successfully in Unity 3D, whre it can be achieved like this - you make 2 shaders, one for the objects being viewed through the portal:
Pass{
Stencil {
Ref 1
Comp Equal
...
and another fot the portal itself:
ZWrite Off
ColorMask 0
Pass {
Stencil {
Ref 1
Comp always
Pass replace
}
}
(that is Unity 3d kind of hybrid HLSL/ShaderLab shader syntax i guess)
I can avchieve the effect of seeing into another space in DX9 even without writing a shader, just by using ZWriteEnable node with its func set to ‘Greater’ (connected to the shader dodes of the objects behind the hole), but obviously the objects revealed through the mask in this way then cannot sort their Z order with respect to each other correctly anymore (they actually seem to be rendered in reversein this way).
So I tried to port the above code to build my own DX9 shader, like this…
the objects:
technique TPhongPoint
{
pass P0
{
//Wrap0 = U; // useful when mesh is round like a sphere
VertexShader = compile vs_3_0 VS();
PixelShader = compile ps_3_0 PS();
//ZWriteEnable = false;
//ZFunc = Equal;
StencilEnable = true;
StencilRef = 1;
StencilFunc = Equal;
}
}
and the mask:
technique TSimpleShader
{
pass P0
{
//Wrap0 = U; // useful when mesh is round like a sphere
VertexShader = compile vs_3_0 VS();
PixelShader = compile ps_3_0 PS();
ColorWriteEnable = 0;
ZWriteEnable = false;
StencilEnable = true;
StencilRef = 1;
StencilFunc = Always;
StencilPass = Replace;
}
}
But this neither gives the desired result, even though the code compiles sucessfully, so I suspect the stencil buffer functionality may not be supported for DX9 in 4v…?
Sooo I decided to move to DX11, after quickly checking that it supports some advanced stencil funcionality which was looking really promissing - first I tried to fiddle with the DepthStencil and DepthStencil (Advanced) nodes, with no luck, then finally when I tried to put my own DX11 shader together, I couldn’t even find a way to set the StencilRef value or its equivalent for the DepthStencilState in DX11:/
I believe there must be either another proper way to do this in DX11 which I just didn’t come around as my knowledge of shaders is pretty basic, or I am missing some essential shader concepts regarding 4v rendering…? Anyways for now I am out of ideas, I feel like I tried every possible way which makes sense to me so far.
Although I am curious if this could still be possible to do using DX9, as the project I am using it for is originally based on DX9 rendering and I dont want to mess up the two in one patch (as it is being discouraged on the forums generally), somehow I feel I should give the DX11 solution a higher priority for the sake of future development and performance optimizations.
Would anyone kindly push me in the right direction?
Cheers everyone, long live the community!:)
