Building our own 3D engine

I'd like to see our (by which I mean your ) hardwork used elsewhere. So I say that we build an engine from the ground up, and GPL it.


And, no offense anyone, but there's a LOT more to building a game then just saying... "Dude, let's make a game."


So my vote is from the engine

 

I think you guys should consider using SDL+OpenGL; it's pretty nice to have the same code compile on windows and most unices.

 

Do yourselves a favor and nominate someone to be in charge.

The Open Source projects that work all have strong leadership, if your going to rely on everyone agreeing not only will you fight about everything, but you'll build crap, because it will lack a unified vision.

Whoever it is should create a vision statement, then start to layout a top level plan.
Various people can then take pieces of that plan in a coordinated way.

The project leader should of course listen to everyone else, you just need a central point to make decisions in some unified fashion.

 

a piece of advice: for a workhorse langauge, pick one that has good compilers, good debugging and good profiling tools. which basically shrinks down your choice to C++ : ) you may want to choose some easy-going (read: light-weight scripting) language to bind things on top and what not. say, lua or python. /end of advice.

 

What's wrong with c# (mono), it's managed & fast. C++ is still faster but slower to develop and harder to debug.
I'd say use c# (mono), unless you want to do something incredible with the 30% performance gain of c++.

 

managed code may not be the best tool for game/engine development. not because of the magnitude of the performance difference with c++ but when/where you get that differece. and i'm not sure mono is available everywhere (it may be, i just have not checked on it lately). last but not least i still have not understood what's all the fuss about throwing in managed code - i personally like full control of my code (and obedience from the compiler). if i really want anything managed i can write the management code and it will be tailored to the specific needs at hand. of course that's just my opinion, and that's not even from the POV of a participant in this b3d project.

 

Definately agree with ERP, get a leader and a good decent place on the web with high visibility to everybody involved. Often things like these take a LONG time, so keeping it alive is often the hardest process...

Don't get too hung on languages, OpenGL/DirectX etc. Its a largely a choice of Windows or everthing else. Windows is easier because MS provide the best free bits and peices but if you want Mac/Linux etc. than you don't have a choice...

So if your happy with Windows have a look at XNA (C#) assuming you don't mind a little speed lose or D3D (and D3DX), else its time to hit sourceforge and see what you can grab for free...

Really can't recommand enough, that scaverging bits of tech is a really good choice if you want to finish stuff... My home stuff has colloda, flash, cairo, D3DX... pretty much everything I can get for free

 

My opinion is to design the game first, or the application. Then use that to design the engine.

IMHO, if you want speed of development, ease of understanding, and colloboration, use C# + Managed DirectX, or Java + OGL bindings, or Python+OGL. I don't think performance is going to matter much given what is being talked about, and for pedagogical purposes, it would help people alot more if the code was written to be extremely understandable rather than fast, so I think scripting is perfect.

In fact, I would advocate punting on designing your own renderer, and find a good scenegraph library. This can let you rapidly prototype your app, no need to write importers (they usually already exist). Then, after the app is buildable and usable by people, you can selectively go back and replace scenegraph specific rendering with your own custom engine.

I think the goal should be a flexible teaching platform for prototyping ideas, and not really as a mega game engine. Want to teach someone how say, ambient occusion lighting effects a scene? Build a small module that does it, publish it to the site, let people play around with editing it and rebuilding it. Much like Humus's demos, but probably much more oriented towards teaching than personal use.

I learn towards OGL, and Cg, plus some scenegraph that uses those. Then, Mono, Java, or Python bindings. Why? Cross platform and free development tools rather than having to use Visual Studio. I'm not familiar with Collada, maybe it's better.

Also, a project like this can't be collaborative from the start. It needs to be "seeded" first by a knowledgable coder. Starting from absolute scratch on open source collab projects don't work. The early stages of coding need a benevolent dictator.

 

This is a very interesting idea, and I'd like to weigh in with some brainstorming. In my firm, I'm the kind of coder that likes to get things done yesterday rather than tomorrow, and that's only possible if you make sure you have a flexible framework to build on. In my case that means using objects/libraries, making a good distinction between data, business logic and user interface logic, and using xml for all the data.

Programming Languages for Main Objects: OpenGL and C.

I certainly vote for using OpenGL and C, maybe even just plain C, for the final project code and definitely for the rendering part of the User Interface. This always leaves the best flexibility and power. And if you design a project well, it's easy. Also, since the crux of the matter here is making a 3D 'interface', a big part of the ease of higher level development tools is negated.

However, as long as we make sure that the different parts of our code can communicate in a transparent manner and we build everything that can be a standalone object as such, then everyone can contribute in any language on any platform they like. Once we have good and working design, translating that code to anything else is typically a fraction of the total workload.

Storing and communicating Data: XML

This should be a no-brainer, but as I've noticed still quite a lot of people haven't used it before, here's a summary of the reasons why.

- it makes debugging a snitch as you can view a snapshot of the data at any time
- you can create the data in lots of different ways, from creating your own xml strings using byte manipulation to using commercial and free objects that manipulate xml (I use .NET's XML library a lot myself, and have done so since the first XML DOM object library was released by Microsoft).
- you can read the data from anywhere without requiring platform specific libraries; if you can parse strings, you can parse xml data (nearly all platform has a library available). This means that people can write their own renderers optimised for their specific platforms very easily.
- you can create and handle hierarchical data very easily

I personally ignore stuff like transformations, namespaces and document type definitions, as they often just create a lot of overhead in your coding. They can be useful in a larger project like this, but shouldn't be used in code unless we have come across some real problems for which they are without a doubt the best solution.

What are we going to build?: 3D Forum Interface

This idea is definitely the best idea in this thread, by far, for the simple reason that it is something everyone on this forum, by definition, has a stake in. We all browse here, we are all interested in 3D, and so a 3D forum browser matches our target audience 100%, both on the development level and on the end product level. Everything we do will be immediately useable.

Also, we have a basic problem very clearly laid out, and the data, structure, content and players are all in place. We just have to imagine a good way to render the information in 3D. An important design goal should be to actually makes it superior to browsing in plain old 2D, as this will keep the drive to get this done and out there a lot higher.

At the same time, we can start with very simple rendering techniques, and slowly build this out to become really complex and test out every bit of modern technology that the likes of Nvidia and ATI can throw at us.

Also helpful will be if it can work with the data in the forums as is. I think this is very well possible, as the forum posts have plenty of structure in them as it is now. There are threads and replies, and after long consideration, really the only thing I would like to have in there which I think there isn't now, is a way of seeing to which post I made a reply.

What work do we need to get done? A Fair Bit.

Obviously, there are the basic three programming areas that we need to get work done:

1. The Data Layer

This is not so much the database design, as it is the interface that a) retrieves data from the forum software, and b) presents this in a generic form to the business logic, which will then in turn present it to the user interface logic (renderer).

For the collecting data part, I think we have two options of going about this

– using crawler routines to map the website’s structure. The advantage of this one is that we can eventually use our technology to go beyond just this type of forum, and we don’t have to worry nearly as much about security and such. The disadvantage is that we will have more difficulty identifying more detailed data such as user names and such.

- writing a routine that interfaces with the forum database directly somehow. This means that we will need a few server side routines that can return forum information in a data only fashion. That shouldn’t be too difficult either – although I don’t know much about forum software, I imagine that the data would be separated from the HTML layout definition. All we need is expose an interface through a web service or a PHP command. The advantage here is that it will be far easier and more consistent to identify the different parts of data, like a forum post’s user, title, etc.

For then exposing this data to the business logic, I think we need to model this data layer as much as possible like a query interface. E.g. supporting functions that can respond to queries like “give me all the new posts by user x on date y”, give me all threads in subforum x, and so on.

As a fail back so that we don’t get stuck, if neither option turns out to be feasible (too much work, or big technical hurdles to overcome), we can still set up our own new forum structure. In this case I would vote for XML documents describing the post and what they are linked to, that we store on various servers and contain all the data you need. This would then get rid of a lot of management stuff, database, bandwidth issues etc., and be terribly easy to implement. All we’d need is a good versioning system of the XML documents created.

2. The Business Objects

The Business Objects will consist of the main application routine, user settings object, etc. We should be able to keep it pretty simple initially, starting with the bare minimum of functions needed, but making sure that we can add as much as we want without running into design issues that force us to redesign our whole application from scratch later on or force us into patchwork programming. Basically this means a good way of interfacing with the User Interface, handling its events, collecting and preparing the data from the Data layer, etc. Those familiar with 3-tier design know the drill.

The best way to start here is list the forum functions that we want to have implemented in order of importance, and decide which ones we are going to do ourselves, and which ones we are going to leave to a default browser and forum page. For instance, we could have the 3d interface link back to the regular forum for reading posts and replying to them, saving us a lot of initial work that we can still add in later once we have the basic things going.

3. The User Interface Objects

This is where a lot of the work will happen, partly because as B3Ders we are most interested in this part. Of course this part contains the renderer, but just as important is generating the user input events. This part of the design is harder than you think, and requires good consideration right at the start of the rendering engine.

Also important to realise is that since we are focussing on a 3D renderer for the forum software, this is actually the most important part of our design. We already know our basic functionality and stuff we want to do, and the way we design the renderer and how we handle user input is going to determine how much of our wish-list we are going to be able to actually implement.

I have two ideas for this. They are fairly similar in nature. Both of them can have a very impressive impact if you can imagine them running in real-time – they don’t have to, but they could by design, and that could make them very impressive to just look at.

User Interface Ideas: Based on a Tree

The first is modelling forum thread structure in the form of trees. The stem would be the website hosting, and subforums would branch out of this in all directions evenly (or maybe for graphical effect slightly less evenly). Then in turn the subforums would get threads as new branches from them, and on these threads we would get the posts as leaves. The thickness and length of every branch depends on the number of sub-branches and leaves it contains and how long it exists could for instance affect its texture. Links could be, say, spider-silk like threads, or labelled flowers, or ivy that visibly connects to another tree when it supports the same interface. Attachments would be modelled after fruit but very clearly identifiable as the attachment type, and posts would be modelled after leaves, but very clearly identifiable by author name.

Travelling the information can be done from an overview mode (showing more or less detail depending on resolution and rendering power), and like a bug walking over the tree visiting leaf after leaf and consuming different fruit, or gliding to new threes over the spider-silk thread.

The interesting part here is that we could at once highlight all new posts in the tree in overview mode, by giving them all sorts of graphical effects that we can try out – HDR, different color, enlarge them, make the rest transparent, etc.

Also, if we want to focus on different information, we could build the tree up completely differently, just depending on the kind of information we select. For instance, we could build a tree based on a text query, or highlighting posts with certain keywords, or whatever.

Navigation could be done for instance by using the scroll wheel (or if not present, keyboard) to rotate the tree and then just mouse clicks. A click could then zoom in on the object being clicked and show it in more details with all sorts of options linked to it.

User Interface Ideas: Based on Neural Pathways

The Neurons would be the posts, and the pathways the threads, links and branches, in different shapes and forms, visibly reflecting what they are and what they connect. Every neuron would have a transparent neuron that you could select to add a post to the forum. Forum users reading a path of neurons would act like energy pulses that thicken the path and this would in turn dynamically suggest the most interesting path to read posts on.

Again this could be looked at from the outside at different levels of zoom (more parallel access if you will), or could be travelled one at a time (sequentially), depending on preference.

Most of the features detailed in the tree idea should be possible to apply here very similarly, but it would have the advantage of being more consistent for all the information and easily expandable to all directions and websites and whatever else we’d want to link.

You would be able to highlight a certain search result, paths to new posts from a certain location, and so on, or even create views showing only the information that resulted from a query or selection of a certain thread, for efficiency and less clutter on the screen.

Housekeeping

- Communications

I think special subforums and threads would be nice, but they shouldn't be private. Everything should be possible to do in public, and if possible, everyone should be able to weigh in on them too. If things get out of hand, mods can always delete posts to clean up. There's a lot of adult people on this board and I think the spirit of openness and exchange of ideas is key for our project and not forgetting that a main objective also is to allow all here to learn from each other.

- Coordination

There are really only a few essential things here: that we agree on the general architecture (basically, that we work with three layers, and how the three layers communicate), and that we have good disciplin in sharing our work, keeping each other up to date online and posting useable results. That way, everyone who wants to contribute can easily see where the input would be most valuable, and those who are responsible for the general architecture can monitor whether the work being done suits the architecture we agreed on. As long as we keep the layers well separated, we can do a lot of work without even having to integrate a lot of the code or worry about a unified setup, binaries, etc. Just make sure I can generate data at the business layer that the renderer can read, that's all that's needed for a good part of the project.

Even if we never get to a complete package, the learning goal will easily be met.

Conclusion

Long post already, so I’ll stop. This should be a good start, if people agree this could be a worthwhile project. People can sign up for different parts of the system. I for instance would be much better with the business logic or data parts of this project than with the 3D, but I’ll at the same time be very interested in following the developments in the 3D department to learn a few things about 3D.

I think the basic premise of finding a good way of representing information in 3D is a bit of a holy grail for many of us anyway (3D windows anyone?), and it could be a nice idea to see how far we can get, even if we start with more modest stuff first.

I myself have been working with hierachical workflows, and have been thinking about how I could design an interesting 3D interface for them at some point in time, and from that perspective it would be fairly easy for me to contribute to this project – a lot of overlap exists on the business and data level with the stuff I am working on professionally.

 

As far as tools and APIs go, free Express versions of the VS tools are available and there are .NET bindings for OpenGL.

So why not architect against the .NET runtime (giving a modicum of portability to non-Windows OSes, especially if an OpenGL renderer exists for the engine), giving us arguably the best toolset and the ability to use a bunch of different languages so that more people can chip in, using Python or LUA for scripting.

Mono means we get to go cross platform from day 0, which is important (helps people learn about portability).

Just my $0.02. And I'll be setting up a space on B3D in the near future for all of this (FTP and web spaces, wikis, code repos, email addresses, etc), and a dedicated subforum here so we can keep everything in one place, discussion wise.

 

I have no qualms with .NET, as this is what I work with 98% of my time and have been for years now. However, the few moments I've coded for the PSP, I realised that by far the most portable stuff is plain old C or C++ combined with OpenGL. For instance my test project in my sig I just coded for GCC and OpenGL using Visual Studio as an editor, and then I could recompile it for PSP (using Cygwin and PSPSDK) almost without any modifications.

 

Amen
And congrats for your project

 

Have you considered Humus framework?

It has support for Linux/Windows, OpenGL/Direct3D, has a very friendly license, and looks rather modular. (I haven't done a detailed analysis of it though.)

It's not a game engine (well actually, one of his demos is a game engine), but it should be a good start.

Humus what do you think?