[maxest]

I extract perspective-frustum planes by first extracting the corners, which are computed in this manner:

		viewFrustumCorners[0] = CVector3(-1.0f, 1.0f, 0.0f);
		viewFrustumCorners[1] = CVector3(1.0f, 1.0f, 0.0f);
		viewFrustumCorners[2] = CVector3(1.0f, -1.0f, 0.0f);
		viewFrustumCorners[3] = CVector3(-1.0f, -1.0f, 0.0f);
		viewFrustumCorners[4] = CVector3(-1.0f, 1.0f, 1.0f);
		viewFrustumCorners[5] = CVector3(1.0f, 1.0f, 1.0f);
		viewFrustumCorners[6] = CVector3(1.0f, -1.0f, 1.0f);
		viewFrustumCorners[7] = CVector3(-1.0f, -1.0f, 1.0f);

		for (int i = 0; i < 8; i++)
		{
			viewFrustumCorners[i] = (CVector4)viewFrustumCorners[i] * viewProjTransformInversed;
		}

So I start with clip-space and go back to world-space. Now I need to do the same thing but for frustum that is not perpsective, but orthographic. So I tried do it in the same way. Unfortunately I don't get proper results. Could you give me some clues on how to extract the ortho-frustum corners?

[Nick]

They are homogeneous coordinates, so they have 4 components. To go from 4D vectors to 3D vectors, just divide the first three component by the last one. Conversion from 3D to 4D is done by setting the last component to 1.

CVector4 viewFrustumCorners[8];

viewFrustumCorners[0] = CVector4(-1.0f, 1.0f, 0.0f, 1.0f);
viewFrustumCorners[1] = CVector4(1.0f, 1.0f, 0.0f, 1.0f);
viewFrustumCorners[2] = CVector4(1.0f, -1.0f, 0.0f, 1.0f);
viewFrustumCorners[3] = CVector4(-1.0f, -1.0f, 0.0f, 1.0f);
viewFrustumCorners[4] = CVector4(-1.0f, 1.0f, 1.0f, 1.0f);
viewFrustumCorners[5] = CVector4(1.0f, 1.0f, 1.0f, 1.0f);
viewFrustumCorners[6] = CVector4(1.0f, -1.0f, 1.0f, 1.0f);
viewFrustumCorners[7] = CVector4(-1.0f, -1.0f, 1.0f, 1.0f);

for (int i = 0; i < 8; i++)
{
	viewFrustumCorners[i] = viewFrustumCorners[i] * viewProjTransformInversed;

	viewFrustumCorners[i].x /= viewFrustumCorners[i].w;
	viewFrustumCorners[i].y /= viewFrustumCorners[i].w;
	viewFrustumCorners[i].z /= viewFrustumCorners[i].w;
}

That should do it.

[maxest]

Yes, I know that. And my overloaded * operator does that automatically. As I said, it works for perspective projections, but doesn't want for ortho

[Nick]

Then just set a simple orthographic projection matrix for which you know the frustum planes, and check the calculations by hand. Somewhere something is not working the way you expect it to work. So verify things step by step at a lower level.

[maxest]

I guess I've just found the problem. My function which calculates the inverse of a matrix returned the same matrix. It didn't compute the inverse because determinant of ortho matrix is near 0, and I have an "if" which says that in this case the computations are to be skipped, because it is not possible to compute inverse matrix from a singular input matrix. Yet ortho isn't singular (althought it's determinant is very small)