3D engine as a college senior project

[Ahriman4891]

Dear all,

I have been reading the forums and the main site articles for about 5 years, and only now decided to register As the title says, I plan to write a 3d game engine for my senior project in college. I have programming experience in (best to worst) C/C++, Java, Haskell+WASH/CGI, Lingo. I know the theory behind 3D graphics reasonably well, but alas, I have very little practical experience with programming it (dismissing Lingo and a couple of little 3D flash games I took part in making). This semester, I am taking an OpenGL-based interactive 3D graphics course; I will ask that professor to become my project advisor, too.

I wonder if you guys could answer some of my questions:

1) Do you think it is a good idea in general? I am sure I will get the faculty permission, but maybe pursuing a different topic would be more fruitful in the long run? (Even though I would like to work in the game industry). Anyone having experience to share?

2) What would be the most reputable sources to get the basic idea of a 3D engine anatomy? (Books, websites, magazines, anything will do).

3) My working machine runs Linux. Given that I will be studying OGL in detail, and plan to utilize that knowledge, it does not seem to be a problem. Still, I have access to Windows as well. Any arguments or comments on Win/DX vs Linux/OGL?

4) I will be able to find someone interested in the topic who may want to join me. Only 1 or 2-ppl teams are allowed. Is such project doable by 1-2 ppl?

These are the most fundamental ones I am still not decided about the details, but I am inclined to a flightsim/rts engine, i.e. big open spaces. That's about it. Frankly, the lack of concreteness scares me quite a bit. I hope to get some helpful pointers/advice here. Thanks!

 

[Intel17]

Programming a game engine is definitely one of the more complex software engineering tasks. Tim Sweeney wrote the Unreal Engine 1 almost single handedly back in 1998, but it took him many years of complete devotion to the engine, and since then Epic has had people specialize for each subsystem (e.g. renderer, audio, physics etc.)

I think you could write a full game engine by yourself, if you spent a while hitting the books, and absorbing anything you can learn from that class, but it's quite questionable as to whether you will be able to do it on time, and/or whether you'll have the patience and tenacity to actually finish it (as I said, it's a HUGE software engineering project, especially for one person).

My advice would be, to learn a lot of OpenGL, and spend a LOT of time thinking about how to implement things in graphics, and while doing so, write a nice, simple 3D graphics demo. (I've written a few so far, but I'm a beginner still, so they basically consist of texture mapping, basic lighting, using the stencil buffer for planar shadows etc.)

 

[JHoxley]

It very much depends on what your ambitions are for this "3D engine". You could certainly write yourself a pre-shader level engine as such a project (assuming your lack of practical experience). Start venturing into a decent effect-based/programmable/shader-driven engine and I'd say you're probably pushing too high.

To even design the API and general outline of a shader-driven engine is a complex enough beast - implementing it will be just as fun.

1) Do you think it is a good idea in general? I am sure I will get the faculty permission, but maybe pursuing a different topic would be more fruitful in the long run? (Even though I would like to work in the game industry). Anyone having experience to share?

There's no doubt it'd be a good learning experience - not just in the applied/graphical sense but also in that it'll definitely flex your design pattern and software architecture muscles.

Although, a word of warning, if you're hoping this might be your ticket into the games industry then you may want to re-think things. I don't want to destroy your ambitions here )) but every wannabe coder (and their dog) is trying their hands at writing "engines" these days. A much better entry ticket would be a completed (albiet small) game thats actually playable.

2) What would be the most reputable sources to get the basic idea of a 3D engine anatomy? (Books, websites, magazines, anything will do).

There are quite a few game coding books around, but I haven't read them - I prefer the general software engineering texts myself

Some books that might interest you: http://www.gamedev.net/columns/books/books.asp?CategoryID=41.

3) My working machine runs Linux. Given that I will be studying OGL in detail, and plan to utilize that knowledge, it does not seem to be a problem. Still, I have access to Windows as well. Any arguments or comments on Win/DX vs Linux/OGL?

I'm a tiny bit biased on this one all things considered... but for what you've described, I don't think it'll matter. If I had to make this choice I'd base it on two things:

1. What are my target audience expecting/using?
2. Which do I have the most experience in using?

#2 is important as you're going to be learning/experimenting with a lot of new technologies with this project - starting from a solid foundation (platform/OS) will be a distinct benefit

4) I will be able to find someone interested in the topic who may want to join me. Only 1 or 2-ppl teams are allowed. Is such project doable by 1-2 ppl?

Again, difficult to comment on this one without knowing how big and how far you want to go. I'd say that, for definite, something is doable with 1-2 people - but whether that something is upto your "vision" I don't know

hth
Jack

 

[Ahriman4891]

Intel17, thanks!

Well, yes, of course I realize it's a huge project. Maybe calling it a "game engine" (as in something whole, including graphics, AI, physics, and network code) is too much I am only aiming at the renderer part of it, + MAYBE some very primitive physics. I will have a small demo by the end of the year which will most likely be scripted (or maybe I will just move the camera around in real time), just to show off the rendering capabilities. Hopefully, I will not be ashamed of showing them by that time.

 

[Ahriman4891]

JHoxley, thanks!

As I said in my previous post, I am only really aiming at the renderer, and probably it will not be shader-driven. My targets, really, are to get a nice occlusion-culling going on + some modest lighting and texturing.

P.S.
I will definitely think about making a small, but playable 2D game instead.

 

[ERP]

Intel17, thanks!

Well, yes, of course I realize it's a huge project. Maybe calling it a "game engine" (as in something whole, including graphics, AI, physics, and network code) is too much I am only aiming at the renderer part of it, + MAYBE some very primitive physics. I will have a small demo by the end of the year which will most likely be scripted (or maybe I will just move the camera around in real time), just to show off the rendering capabilities. Hopefully, I will not be ashamed of showing them by that time.

Just make sure you define what you're building.

To me "game engine" means something very different than say the original quake or unreal "engines".

What problems are you solving?
Are you providing tools?
You can knock out a quick rendering demo in a few days, what are you providing that is more than this?

 

[Killer-Kris]

4) I will be able to find someone interested in the topic who may want to join me. Only 1 or 2-ppl teams are allowed. Is such project doable by 1-2 ppl?

I'll add a quick word of warning on this part because that's what I have the most experience with out of all your questions.

Working in groups is an interesting adventure all on it's own. With more people you can possibly get more done quicker, but with out the proper combination of people, communication, and organization things can take a very quick turn for the worse.

Here are a few suggestions I have if you're going to be working in a group...
- make sure that the work is easily divided up so that it can be worked on and tested seperately

- plan ahead, having the interfaces between the sections defined before hand saves a ton of time later on when you attempt to integrate them. (ie. do a well thought out design before writing a single line of code)

- use CVS or something similar, trust me you'll thank yourself later.

Your mileage may of course vary, but these are just a few of the most important lessons I took away from my last software engineering class.

 

[Graham]

I'll thrown my coins in...

It is a big undertaking, thats blindingly obvious. It's also a very long term project. Don't be surprised if it takes 2+ years to build. And, most importantly, you absolutly *need* to stick with it. It's very easy to fall off the trail and get distracted/give up, especially when it's not paid work (heck, even if it is )

The big problem you will face, is that designing an engine is probably harder than actually making it. And the only thing that makes it easier is past experience of things going wrong, designs going bad. This, needless to say, could be an ideal focus for your study. Write about what goes wrong, and maybe look at how best to avoid it in the future.?

I have made a bunch of different 'engiens' in my time. All have just been for my own enjoyment mostly. My last engine I'd say was probably a failure. Sure it could do a bunch of stuff, but in reality it wasn't very good. A combination of poor early design, lack of experience, and trying to retrofit a 2 year old engine for shaders, modern tech, etc. Doesn't work.

Group work is a *very* tricky prospect. Especially in an educational setting. Unfortunatly, most universities, etc, don't really stress good work practises, group planning, and even basic maintainability. Sure someone might think they know how quick sort works, but that is utterly irrelivant if the largest class hirachy involved 3 classes.

Your trouble is going to be visualising how the system works. There are 10,000 ways to write an engine, from hard core procedural to modular task based OO systems. Personal preference plays a massive part in this, and can often be blinding too. I talked to someone at university a while ago, they wanted make a very simple app, to which I recommended C# as the language of choice. *Bang* anti-microsoft rubbish starts errupting from his ears. No, sorry, has to be done in C. *sigh*. (I'm using C# 2 as my language of choice for my current engine btw, it's very nice, if you tried to like java but just couldn't, you will love C#)

Ok.
I'll reiterate.

Design design design. But don't over design.
I know thats a contradiction, but hear me out - You need to design loosly, but deep. Not strict and shallow (a common trap people get into). Your design will need to be loose enough to accomodate *inevitable* changes (minor or major). It also needs to be loose enough that it can be extended. Don't underestimate an extensible design.
However, it needs to go deep. You need to think about the exact structure of how your code is going to run. For example, a very simple question: Do you plan to take advantage of multi-core processors?. Very simple question, most people would say 'of course', but the true answer is actually *much* deeper than that, and requires *very* exacting design. I underestimated this, and one of my main goals with my current engine was multi-threaded capability. I didn't say MT was hard, it's just different, and it's definitly not something you can just decide to add later on (as with pretty much any design change).

Start simple and extend.

This sounds obvious, but it's surprisng how often people don't do this.
A simple example is if your engine started off as a quake3 map renderer, how would you add in a user interface? chances are you would write it all from scratch including openGL/DirectX code. How about if the first thing you do is an abstracted system for drawing triangles as batches (ie, you do not render things straight away, you batch things up first)? Sound too simple? sound pointless? lets give an example where it might help:

a) you want to add support for direct3D when you finish your engine... How? Think about it.
b) you want to re-order rendering of transparent geometry. How? Think about it.
c) you want to minimize texture swaps. How?
d) You want to do dynamic lighting, where the light itself does the lighting. How?
e) You want to make a UI without adding any new low-level rendering code. How?
f) ...
z) ..
zz)...

etc. You really, really have to think how you will be using your code. And doing that is very hard. And that only covers the basics of rendering, not things like shaders, materials, lighting, scene graphing space partitioning, culling, collision detection, object logic, etc. etc etc. It's a big thing when you really want to start pushing the boundaries.


As for your actual questons,
I'll answer the one about linux.
Do not even think about uisng the likes of xemacs to write your code. Get a good IDE. Eclipse is OK, and is probably your only good option, but do *seriously* consider visual studio 2005, also do consider C#, it will save *a lot* of hassles early on, but OpenGL isn't really an option in C# as since TapenGL moved to the mono project it's become a tad unreliable (Tao is a .net wrapper for openGL). If you prefer OO, then you will enjoy managed DirectX (ie, DirectX for .net). It's a lot nicer than unmanaged DX, and it's faster too.

 

[Ahriman4891]

Thank you guys, especially Graham--that kind of detailed reply is what I was looking for. I am still evaluating all my options and reading on 3D engines, becoming more scared as time passes :uneasy_grin: I will continue posting here as the whole epopee continues (or should I say *if* it continues)--hopefully someone will find it helpful later on. I will have a concrete plan by next week.

 

[Graham]

you should do it. You would regret it otherwise. It is definitly a lot of fun. Just very hard.

So long as you prepare to make impossibly large mistakes you will be fine.

 

[Scott_Arm]

... If you prefer OO, then you will enjoy managed DirectX (ie, DirectX for .net). It's a lot nicer than unmanaged DX, and it's faster too.

I've been thinking of doing some DirectX programming with C#, mostly because I'm not thinking of game programming as a career, but more of a hobby that I'll have limited time to commit to. I would have thought managed would be quite a bit slower? Do you ever find C# to be inflexible when writing something complex like a game engine? I know it's great for regular apps.

 

[Graham]

Well I'm currently building an engine using C# 2 beta, and I have to say there really isn't anything you are limited to. You just need to think differently when writing it.


For example, you could use interop and unsafe code to get around problems that you may encounter, but in reality, this is a design issue not a limitation of the language / .net. I don't believe there is a case where unsafe code can not be done with managed code (ie, Buffer,BitConverter, whatnot). And often, these classes actually internally use unsafe code (but are guarenteed not to fall over or cause a security risk) so are better (and probably better optimised). As for interop, there are a few cases where I am using it, but I've put them into a platform module of it's own. Pretty much the only thing that *has* to be in there is changing the video mode, but directX can do this for you anyway, so even that could be removed if I wanted to (but I've left it in as a option if someone else ever made a openGL renderer, etc).

Flexibility, well, frankly I feel using a managed language frees you of lower level responcibility, giving you more freedom.
Initially I actually started my current engine as a C++ app. But after a week, I realised everything I'd written was already provided for me in .net, and provided better (reference counting for example, arrays, etc). Personally I cannot think of a case where I've been limited by .net. I can think of many cases where C++ has limited me in the past, and this is something I'll mention as well:

Do not underestimate reflection, meta data, events / delegates and *real* interfaces.

.net provides support for object/class/property/etc meta data. In the form of 'attributes' which are actually just regular classes (but special in some ways). What this means, is you can tag the declaration of members of a class with another class. The sheer power of this when it comes to games is very easy to underestimate. It's something that requires a different kind of thinking... For example, imagine in a multiplayer game you have the following data in an object:

private Matrix matrix;
private int state;
private byte health,armour,ammo;

now. imagine you wanted a client-server architecture, it could potentially be very hard to implement. Now, imagine this:

[ClientSidePredicted(0.5)]
private Matrix matrix;

[ServerSideUpdate(0.1,Prediction.ClientSideAssumed)]
private int state;

[ServerSideUpdate(1,Prediction.None)]
private byte health,armour,ammo;

You have just tagged that class's data with meta data which the client/server classes can then access themselves through the meta data. This sort of thing can be applied all over the place if you want to be creative. It's a very different way of programming to C++, but has potentially huge advantages. And that just scratches the surface of what .net can do if you push it.


As for speed, I personally believe that both managed and unmanaged languages are just as fast as each other, you just need to know how to write for them. Certain things run slow in a managed language (nth generation GC for example) while certain things run slow in unmanaged code (template bloat, etc). Often in C++ you end up doing a lot of redundant checking yourself anyway for things like array bounds and reference checks. These get optimised out in a managed language.
The advantage Managed DirectX has over normal DirectX is that COM can effectivly be removed from the equation. On top of that, because the code is effectivly inline (as it's compiled at runtime) redundant state changes can be removed with far greater efficiency (although this doesn't seem to happen all the time).

Don't forget also that a managed language will get faster on a newer architecture, umanaged won't. (intel is apparently going to start adding instruction sets dedicated to .net to it's processors soon).

In the directX sdk, there is a managed and unmanaged version of the skeletal animation app. On my old radeon 9500, the managed version runs twice as fast. On a 9800SE, I've seen it run 4x faster, yet they are basically exactly the same app. This isn't scientific I know, and there probably is a deeper reason, but one thing is definitly certain: expect a massive speed boost when vista comes along, more so in MDX than umDX I'd expect.


On top of that, MDX is far nicer to use than normal DX.
for example, it makes signigicant use of .net properties.

Enabling lighting, for example, would be:

device.RenderState.Lighting = true;

because the lighting state is wrapped up in a boolean property. Much nicer than device->SetRenderState(D3DRS_LIGHTING,D3D_TRUE); or whatever it is. And of course you get full integration of the api and documentation with intellisense (which is fantastic).


Think about it. I'm not saying use it, but keep it in mind. It's definitly the way of the future (managed vs unmanaged). The great thing is, is that .net will suceed unless something *significantly better again* comes along (which you can say about java vs .net, which I feel is no contest). It's great really.


I might be considered a 'fan boy' of the language, but I think my reasoning is pretty sound. I've been writing .net code for a good number of years now. When I first picked it up, I was a die hard C++ guy, and thought it would just be another java, ie, a distraction, but ultimatly too limiting and annoying to use productivly. I was quite wrong - I'm not saying it's for everyone, just it suits me perfectly.

 

[Scott_Arm]

Thanks for the input. I'm not a career programmer (yet). I just know what I've learned in school and a bit of toying around on my own. Definitely not a hardcore .NET or java developer though I've used both. Very informative little read you gave me.

Edit: Are there any significant differences between the new C# in VS2006 and C# in VS2003?

 

[Graham]

There are some very significant differences,

Visual Studio 2005 (or is it 2006 now?) is built on .net 2.0, visual studio 2003 is built on .net 1.1

There are many subtle changes here and there of course, performances improvments, ease of use tweaks, bug fixes, and a whole bunch of new classes/extensions to existing classes.

The biggest change however, is support for generics. Not java style questionable-usefulness generics, these are real generics much closer to C++ templates than java's.

For example, in 1.1, you used the ArrayList class for dynamic arrays, but this required casting the output, which was potentially a performance problem (and it was ugly). in 2.0, there is now List<type>, which is faster, especially for structure value types.

There are also new ones like Dictionary<key_type,type>, SortedDictionary, etc. Not the biggest selection, but they are all very good and work well.

Also new are in code defined iterators (which are *really* elegant), partially defined classes (define a class over multiple files, very useful when used with the form designer), inline delegates (think of them as annonymous inner member functions). And a bunch of other things too (read here)

And of course VS2005 is a massive improvment on VS2003 (which is already good)

 

[Simon F]

Hmm... I just "upgraded" from VS6 to .Net and in the first thing I tried to compile had

#if X
#define BLAH (1)
#else
#define BLAH (2)
#endif

The new system reported that I'd redefined "BLAH" and so aborted the compilation.

I think I'll stick with gcc for the time being.

 

[ERP]

Hmm... I just "upgraded" from VS6 to .Net and in the first thing I tried to compile had

#if X
#define BLAH (1)
#else
#define BLAH (2)
#endif

The new system reported that I'd redefined "BLAH" and so aborted the compilation.

I think I'll stick with gcc for the time being.

Bizarre I've never seen that issue in the .NET compiler.
This was the only file in the code?

The .net compiler is considerably better than the VS6 one both in terms of following the standard and generating code. That made it something of a pain when moving large code bases over from VS6. The IDE is a mixed bag.

Just tried this and I see ehat the issue is. You deleted the precompiled header include and didn't turn off precompiled headers, VS6 would have done the same thing, I get the following error with PCH disabled. Or you can just leave stdafx.h include at the top of the file and it will work too.

Linking...
LIBCMTD.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function _mainCRTStartup
C:\Documents and Settings\Rob\My Documents\Visual Studio Projects\wanktest\Debug\wanktest.exe : fatal error LNK1120: 1 unresolved externals


Which is what I'd expect from the fragment.

 

[darkblu]

Hmm... I just "upgraded" from VS6 to .Net and in the first thing I tried to compile had

#if X
#define BLAH (1)
#else
#define BLAH (2)
#endif

The new system reported that I'd redefined "BLAH" and so aborted the compilation.

I think I'll stick with gcc for the time being.

compliance-wise and output-quality-wise vc7 is better than vc6 hands down. but recently i spotted a compiler that really got me excited - how about using _one_ compiler to generate SIMD code for both altivec _and_ sse - yep, that's part of apple's gig with their new gcc port that comes with xcode 2.1

 

[Graham]

Well it's been positivly *ages* since I did any serious C++ programming (cursed linker errors, compile times, header files!! etc)

Are you defining that macro in a header file or a source file?

compliance-wise and output-quality-wise vc7 is better than vc6 hands down. but recently i spotted a compiler that really got me excited - how about using _one_ compiler to generate SIMD code for both altivec _and_ sse - yep, that's part of apple's gig with their new gcc port that comes with xcode 2.1

doesn't gcc 4 do that already?

 

[darkblu]

Well it's been positivly *ages* since I did any serious C++ programming (cursed linker errors, compile times, header files!! etc)

Are you defining that macro in a header file or a source file?

doesn't gcc 4 do that already?

yes, it actually does. i was being too excited after i learned about this feature from osx's later accelerated.framework, overlooking the fact that gcc4 is supposed to do that for every fp simd isa through the auto-vectorization :silly: moreover the accelerated.framework does not provide full platform-agnostic functional support, just a single set of simd data types and _some_ higher-level functions, but you still have to use platform-specific intrinsics for the basic stuff, i.e. simple arithmetics. ..although there's this possibility to mix the accelerated.framework universal simd types, the framework's higher-level functions _and_ gcc's auto-vectorization to achieve a practically platform-agnostic effect ; )

 

[Intel17]

Graham,

Do you think that C# and the .NET framework are going to eventually replace C++ (or at least be a solution that isn't scoffed at when mentioned for usage in a project)?

I started programming with C++ and spent the last few weeks learning the basics of D3D9 Native, but after fiddling around with C#, I'm quite starting to like it. Do you think that it's a wise decision to focus a little more on Managed DirectX and C#, and would it be suitable for writing an enigne that's fairly advanced (even maybe commercial level...for small developers)?

Intel17