Let's talk a bit about the GS.
The weakness of the GS is that it can only read 1 texture and it has a very limited blend equation. So, you can only do match linearly in a equation like (((A op B) op C) op D). Those parens are cruel and unforgiving.
The strength of the GS is that it can switch settings in just a few cycles. That's way faster than pretty much any other hardware. Even modern "bindless" hardware is going to spend more than a few cycles if only for the cache miss. So, you can draw a dozen tris, switch blend mode and textures, push the already transformed tris again, switch textures, etc... and it's OK. It's a necessity on the GS because you are required to do multipass rendering to do any interesting shading and you don't want to transform & clip each triangle over and over for each pass.
There are a few basic GS "shaders". AKA: common multipass setups.
- Single pass - Just draw the texture. Maybe with some vertex lighting.
- Lightmapping - 2 sets of UVs. Unfortunately, the GS can only do monochrome multiplication.
- Blending - Environments geo needs to mix textures to hide tiling. Investigate combining vertex and texture alpha. Optionally needs a lightmap pass in addition.
- Masked Sphere Mapping - Surprisingly effective and worth a detailed explanation.
The artists I worked with used masked sphere mapping on way more than I expected. Characters, environments, pretty much everything. It feels dynamic and looks way more sophisticated than it is. We had magazine reviews praising our "normal mapped characters and environments" even though the PS2 does not support normal mapping.
The idea is simple: 1st pass do
spherical environment mapping. So, since we can't use cube maps, just use a 2D image of a lit sphere and convert the view-space normals to 2D UVs. For the 2nd pass, draw the vertex-lit diffuse texture blended over the environment map. For the blend equation you can either do mul-add or lerp. mul-add is a bit more flexible. Lerp is a bit easier for artists to understand and control. With the blend, the diffuse texture's alpha becomes a per-pixel specular mask The per-pixelness of it is unexpected on the PS2 and makes it feel special.
It is possible to do shadow mapping on the GS. The math is a bit of a mind-bend. You can render an 8-bit depth map easy enough. And, you can use classic projection math to project that map onto an mesh like everyone does for shadow mapping. The catch is that you can't do the normal "if the interpolated vertex distance from the light is > the sampled projected shadow map distance : do shadowing" math everyone uses. But, what you *can* do is: Calculate the vertex distance to the light on the VU and put that in the GS vertex alpha? specular? I forget. But, you can do an alpha test vs. "vertex alpha + shadow map value" And, you can set them up such that the map contains the (8 bit) distance *from* the light *to* the shadow caster. And, the vertex alpha contains the distance *from* the mesh *to* the light. So, if the two distances in their opposite directions cross each other A--<->--B, their sum will be >= 255. With that you can do an alpha test vs. 255 and that's where there is shadow. Unfortunately, you cannot do a multiple or lerp operation in this setup. Best you can do is subtract. But, subtracting colors looks really unnatural. Looks like burning the screen. So, usually you subtract 255 to slam the shadow to full black. :/
Use 8-bit palettized textures. Really. There is no excuse for 24/32 bit textures. 4 bit textures are more trouble than their worth though. Palettization algos are such old news now that the hard part will be finding a good one that hasn't been abandoned and deleted from the internet. Each texture can have it's own palette. No need to share. I'm a huge fan of palette swaps and
color cycling. But, it's rare to find an artist who understands the concept. The only good paint program I know of that supports the practice is
https://www.aseprite.org/ Watch this video
it's worth it.
Do not treat GS RAM as a fixed limit on texture memory. Stream textures from EE RAM into GS RAM during the frame. This isn't free. It requires some smart scheduling and space management. But it's faster than you'd expect.
Particles are usually either additive or lerping. I'm a big fan of "premultiplied alpha" which lets you blend between both effects per-pixel. Great for glowy fire and dark smoke in a single sprite. A bit hard for artists because they don't teach it in art school. Subtractive blending can also be used for evil, unnatural effects. You can get sub-pixel antialiasing for tiny particle by having the VU clamp the
minimum screen-space size of a sprite to be 1x1 screen pixel. That way it always touches 1 pixel and never gets lost between pixels. Instead of letting it get smaller, modulate (dim) the particle alpha/brightness by how much smaller than 1x1 it would have been. Boom. Manually calculated sub-pixel color contribution.
ms). It worked by doing 5 passes with UV offsets in horizontal line accumulating into a half-width destination buffer. Then it would read from there to do 5 passes with UVs offset to read from a vertical line into a half-height destination. By recursively writing half-width, then half-height, it would eventually reduce down to a single pixel. Along the way it would fill exactly 1 screen with the intermediate results. From there it would recurse in reverse. Starting from the final single-pixel it would stretch and blend the smaller, later results back up over the earlier, larger results in a single pass. By manually controlling the alpha value for each of these upwards blends, you could pick any combination of the intermediate results to contribute to the final result.
