Building our own 3D engine

[DemoCoder]

No need for crawling the forums. If B3D is helping to support this, implement a simple REST interface PHP script for obtaining forum data without all the HTML. It would be trivial for them to take the current set of index/forum/thread php scripts, strip out all the HTML, and dump XML of just the relevant data.

My own personal opinion is to use Google's GDATA atom format for the dump (and also maybe support RSS later). Then, the 3D Forum Reader will not only work on B3D, but can be used to read Google Groups (USENET) and others.

I would also push to make it a browser plugin instead of a separate standalone application, so that it can be integrated with HTML panels. For example, Rys could stick an <OBJECT> tag in the PHP to invoke the plugin (if set in preferences) but still maintain the HTML ad banners and other site navigation.

Or, some more enterprising individual might want to *script* the 3D Forum Reader with JavaScript, mashing up Google Maps or Calendar, or some other Web2.0 site with 3D Forum.

Thus, given that I know it is not an alternative Web interface, I would say

1) write base in C + OGL. Make this base as small as possible.

2) event handling code, "logic" of the reader, for the UI, all done with Rhino JavaScript engine (used by Firefox). Widgets, core animation, rendering all in C, but UI events like clicks, form field entry, etc all handled in JavaScript callbacks.


Yes, it's tempting to write the entire thing in Mono or Java if it is standalone, but I think if it is a Forum interface, it should sit well in the browser and use JavaScript, so that people can embed it themselves and use it for applications. Mono/Java are too bulky for this approach IMHO.

If someone wants to make a Mono/Java/Ruby/whatever binding for it later, they can.

BTW, didn't Alex St John produce a ActiveX 3D component engine?

I would love it if someone enhanced Mozilla's CANVAS object to return a 3D Context so that you could 3D programming in the web browser with HTML + JavaScript. See http://developer.mozilla.org/en/docs/Drawing_Graphics_with_Canvas for example.

I just don't think the majority of the logic fo needs to be done in anything but JavaScript. What should probably be C is the UI Widgets themselves: 3D button, 3D HTML panel, 3D textfield, etc

The other advantage of a C + JavaScript implementation is that other sites, besides Beyond3D can easily customize it without needing to be 3D experts or C programmers.

 

[Graham]

I'm willing to give out a preview version of the engine I've been working on in my spare time. Although I'm not entirly happy to, as it's still got little bugs all over and is feature incomplete.

It's C#/MDX currently, although the MDX part is not tied to the design.

 

[Frank]

Good ideas all around. I think we pretty much agree on the engine, language, interface and datastructures part. But I offer an alternative user interface.

Let's combine forums with chatrooms and personal messaging, adding a game-like aspect when we're at it. And allow us to browse the rest of the web seamlessly at the same time. Like this:

When you come to (the) B3D (forum), you're offered to download a browser plugin (Firefox only at first) for the 3D engine. And you choose a 3D avatar. Or upload your own one, and use that. And enter the site in 3D.

If you go to the forum, that's basically a recorded discussion over time. Like, a lot of chatrooms with a soapbox. You see a room with portals for each board, which all give access to a hallway with doors to all the threads in historical sequence. Enter the one you like (walk), and you can chat with everyone inside (select them when you come close to them), or take your place on the soapbox and have it recorded in the forum.

Now, half the fun of a real forum is, that you can read the history and respond in length when you feel like. We copy that. Read the history, write your lengthy speech, go stand on the soapbox and press the "submit" button.

Unless (and even when) we want to incorporate speech, we need an interface to make that process manageable. For the soapbox, we use a floating portrait of the speaker with textbox in the HUD, where you can set the speed and scroll through the message. And scroll back and forth through the tread history. And offer people the option to respond directly to it while you're still standing on the soapbox, which will be recorded as comments (quote). And throw him/her off, when someone else wants to orate.

And we'll give everyone a PDA, in which they can keep track of all that and their personal chat with others. Make it full-screen when you want to see the old forum, or when visiting webpages that don't offer a 3D view. Searching or going to another site will materialize a portal for you to step through.

And, if we do that, it's only a small step to record the movement of the avatars alongside it. So you can play it all back at your leisure. Shouting and emotes would be desirable here.

That is essentially what most people want, I think. Chatting, MSN, or even a large portion of the people who play on-line games.



If we want to do all that and make it persistent, we need to offer people the opportunity to design rooms and avatars themselves. And a context and additional rules, like you get a gun when you enter and can shoot at whatever you want, for the on-line FPS people. Which can all be included in or attached to html or css pages as (zlibbed) xml files. With an object repository. The main one would initially be at B3D, but everyone could add their own one by supplying a valid url.

Links open portals, and when you step through you either get a full-window PDA with the html page, or (if that site suppies the required xml file(s) that describes the 3D aspect of it) you simply enter the room at the other side.

Animations, materials, rules and scripts would be compressed xml files as well.


Edit: and we would have to supply a basic function that displays the html page as a texture on a surface as requested by the xml description. And supply hooks for the user interface.

 

[Frank]

If we want to make it mainstream, we should also come up with a good system where people can offer their goods (from helping make a script or texture, to buying some goods and have them delivered at your doorstep) for pay.

And see when Google starts to render a 3D display of the page when you search.

 

[Frank]

If we would build all that, you could ask around what's happening and hot, go there and join the fun. Be it a seminair of 3D engine techniques, a nice political discussion or a quick FPS romp. Or just take a walk around and go exploring all of it.

And everyone who wanted to could have their own office, with their personal 3D playground.

 

[Frank]

As for a starting point: how about the Irrlicht Engine? It seems to do all we would require.

Edit: demos are included with the download, in the ./bin directory.

 

[Frank]

Btw, I understand that we want to do it ourselves. But without an existing codebase, there is not enough of a starting point. We need results first. That's how it works.

I really suggest we start with the above engine, and add our own changes to get up to speed. We need to write lots of things, like our own xml bindings for a start. And we might end up replacing all of it. But we need a "live" starting point to be able to get something working fast. Which will be when this project starts to take off seriously.

 

[Arwin]

No need for crawling the forums. If B3D is helping to support this, implement a simple REST interface PHP script for obtaining forum data without all the HTML. It would be trivial for them to take the current set of index/forum/thread php scripts, strip out all the HTML, and dump XML of just the relevant data.

My own personal opinion is to use Google's GDATA atom format for the dump (and also maybe support RSS later). Then, the 3D Forum Reader will not only work on B3D, but can be used to read Google Groups (USENET) and others.

That sounds great, just what we need. What do you think, Rys, is that feasible?

I would also push to make it a browser plugin instead of a separate standalone application, so that it can be integrated with HTML panels. For example, Rys could stick an <OBJECT> tag in the PHP to invoke the plugin (if set in preferences) but still maintain the HTML ad banners and other site navigation.

Either way is pretty ok. As I mentioned, I also think somehow blending existing browser technology, certainly for reading the forum posts, would be good. This may actually be a better way to start than the other way around, but you can control a browser from the outside quite well also, with the added bonus that we might have more room for something that's universally applicable. I would, for instance, be interested in porting this to the PSP at some point, but it's not a necessity to begin with.

Or, some more enterprising individual might want to *script* the 3D Forum Reader with JavaScript, mashing up Google Maps or Calendar, or some other Web2.0 site with 3D Forum.

Thus, given that I know it is not an alternative Web interface, I would say

1) write base in C + OGL. Make this base as small as possible.

Yes, agreed.

2) event handling code, "logic" of the reader, for the UI, all done with Rhino JavaScript engine (used by Firefox). Widgets, core animation, rendering all in C, but UI events like clicks, form field entry, etc all handled in JavaScript callbacks.

Should be fine. And as long as we bother with proper object separation and three-tieredness, we can always go somewhere else.

I just don't think the majority of the logic fo needs to be done in anything but JavaScript. What should probably be C is the UI Widgets themselves: 3D button, 3D HTML panel, 3D textfield, etc

Yeah. But let's visualise the interface better first.

The other advantage of a C + JavaScript implementation is that other sites, besides Beyond3D can easily customize it without needing to be 3D experts or C programmers.

That's an advantage, sure.

 

[Frank]

I'm willing to give out a preview version of the engine I've been working on in my spare time. Although I'm not entirly happy to, as it's still got little bugs all over and is feature incomplete.

It's C#/MDX currently, although the MDX part is not tied to the design.

If you're willing to show it to us, we will gladly consider it. And little bugs are no problem.

 

[Frank]

Arwin, would you be interested in taking the lead in designing the xml data structure? That would probably be the major accomplishment in this project.

And do you agree that it would be best if that could be embedded into html pages? Would you prefer JavaScript functions to gather the data we would require, or only use those for the UI interaction, and use urls that point to xml pages for all the metadata we require? As Demo said, we should stick to interfaces that are used today if we can.

 

[Arwin]

Arwin, would you be interested in taking the lead in designing the xml data structure? That would probably be the major accomplishment in this project.

I'd be happy to do some work on that, but I do think that we have a bit of work to do before we get there.

And do you agree that it would be best if that could be embedded into html pages? Would you prefer JavaScript functions to gather the data we would require, or only use those for the UI interaction, and use urls that point to xml pages for all the metadata we require? As Demo said, we should stick to interfaces that are used today if we can.

90% of our work is a good design. If you do that part of the work right, the coding is a piece of cake, and the environment and languages you are going to code in will be almost self-apparent.

So this is what we need to brainstorm on a little more. What do we want to build exactly? What are our design goals? Once we figure those out, we can move on to working out how to meet them. I suggest we brainstorm a bit more on that part before we move on to the technical bits.

This is a lot like my day-job, in the sense that people often come with a vague conception on something that they want a computer to solve, and then I sit down with them and help them first focus on the larger view, maybe restructure some processes, and then figure out where software can really add something, save work, improve quality, etc.

In this case, the question I would put ourselves is "how can we use 3D to improve our forum experience." If the conclusion of that would be 'we can't', then we might shift our design goals and just try to recreate the same experience but use 3D graphics to make it look a lot hotter. But of course I'm hoping we can actually make something that adds some level of efficiency, by giving a structural view of the information that goes way beyond what we have now. Preferably something dynamic and (semi-) realtime.

 

[Graham]

If you're willing to show it to us, we will gladly consider it. And little bugs are no problem.

k.
I put up the current code base here.
There is a brief readme included.

It may not be the sort of thing you guys are after, but I thought I'd put it out there anyway. Could simply just be an opportunity to see my ideas for api design, etc.

It's mostly been a skills exercise. Designing any engine of even modest scale is a significant challenge.

Keep in mind that although I've been working away at this for 18 odd months, thats 18 months of rather limited spare time, so probably equates to 15 minutes a day or so.

[edit]
I'll also say it's 100% my own code. No 3rd party stuff at all.

 

[[maven]]

Could someone explain to me how that 3D browser is supposed to look like in a (few) mock-up(s)? I'm having a hard time picturing it.
Also, if we display plenty of text, the a large focus should be on high-quality text-rendering (and by that I don't mean a GDI render-to-texture). One could do some interesting things with pixel coverage for a ClearType a-like for text and lines.
As for the choice of language, I'd personally prefer C or C++ for multi-platform reasons (Mac here ), if you say C# is multi-platform, then Objective-C is multi-platform...
For the renderer, I think it's best to define your own API and then map that to OpenGL or Direct3D, so it'll work with any sort of back-end.

 

[Razor1]

when ya guys are ready and need artwork let me know I'll gladly help out!

 

[Rys]

That sounds great, just what we need. What do you think, Rys, is that feasible?

Sure, I can provide the forum hooks for the project without any real difficulty.

 

[Arwin]

when ya guys are ready and need artwork let me know I'll gladly help out!

Thanks for the offer!

Personally, there are two projects I would like to try out. There are just my personal preferences and I will go along gladly with anyone who has a better idea (or even an idea that gets more support than mine despite being obviously inferior )

Purely Aesthetical
First of all, I would like to render a tree based on posts on the idea I outlined earlier, but instead of making it useful, I would like to make it pretty. That means making a real, growing tree that reflects the status of the beyond 3d forum, or possibly a number of separate trees/plants, one for each subforum, so that the beyond 3d forum status is reflected by a garden, with some parts containing a lot of fresh shoots and flowers, and others withering. The tree is just one option, but I think it would be a good starting point.

Purely Functional
The second thing I would like to do is make a 3D representation of the forum, which looks a lot more like the forum already does now, but with as many of its features and items drawn in paralel as possible, which can be navigated in 3d. We would just use 3d representations of the icons and buttons we already have now in 2D to start with. From there on, we can freely start experimenting with the 3D representation, dynamic views and effects. If it turns out that the best and most efficient we can do is recreate the almost exact same forum layout as we have now, but using 3D objects, then so be it, but at least it will be 3D and we can play around with all sorts of lighting, shadow, shader effects and so on.

One thing that I have on the drawing board is starting with the Forum root, and then for example having each child item hang below the next, in a very simple umbrella skeleton like structure. We could allow a user to rotate the items to bring the one he's interested in into focus, and depending on the client machine's power render more or less sub items at once at various zoom levels.

@Rys: ok, that sounds great. Assuming we go ahead with this project, I will start with attempting to map out the most basic functions we need and see how we can hook that up to the forum and create the first bits of xml that can function as input for creating stuff that is to be rendered at 3d. The best thing would be to get something incredibly simple going very quickly, as this kind of prototyping tends to help stimulate new ideas and enthusiasm ...

@Maven: text rendering is always interesting and useful. I have been wondering about the demands of it though, meaning it can get very poly intensive. But that should be less and less of a problem, and the advantages of being able to have a 3D text renderer that you can scale, apply lighting and shadowing and sharder effects on and so on, is of course incredibly appealing. I'm just a tad worried that would be a project in and of itself. But maybe we can afford to have some people work on that.

I will also see if I can get some visuals ... have been experimenting with drawing one or two things, but I'm not really a graphics person, so that doesn't come as naturally to me as other stuff. I do like to visualise though, just clumsy at representing that mental visualisation on paper/screen.

 

[Kanyamagufa]

Ready and willing to contribute any artwork whenever it's needed. 2D, 3D, you name it.

I'm afraid I can't help much with the programming, but I'm incredibly interested in what goes into building an engine and how it all works. I think this is a great idea, and I'm looking forward to learning about it!

 

[Reverend]

To be completely honest, I doubt we'll ever see a game created out of this suggestion. Or at the very least, created in any kind of timeframe for it to be relevant to the focus of this site/forum (i.e. late-breaking, new technologies). The main thing is that I think it's going to take a tremendous amount of time, mostly due to the creation of art assets, also mostly due to the fact that this will be a largely Internet-based cooperative effort.

Because of the time this involves, the "game" will probably never get done due to revisions upon revisions due to suggestions upon suggestions.

I am fairly confident, however, that a game engine (much more than the thread title, i.e. just a 3D rendering engine) can be created by this community. I also see this happening over a long period of time but ut will be a considerably shorter one compared to an actual "game".

I'm game if most of you acknowledge the possibilities (of problems) I outlined above. And yes, I won't have a lot of time for this project (in case I hadn't hinted such) but for those involved that knows how I can help, there's the PM and my email address.

Just say the engine will not feature, gasp, ray-casting.

 

[Frank]

k.
I put up the current code base here.
There is a brief readme included.

It may not be the sort of thing you guys are after, but I thought I'd put it out there anyway. Could simply just be an opportunity to see my ideas for api design, etc.

It's mostly been a skills exercise. Designing any engine of even modest scale is a significant challenge.

Keep in mind that although I've been working away at this for 18 odd months, thats 18 months of rather limited spare time, so probably equates to 15 minutes a day or so.

[edit]
I'll also say it's 100% my own code. No 3rd party stuff at all.

I like it. Especially the bit about an intermediate shading language as part of it. Can you give us a short introduction with some code snippets and how we could make an executable with a basic scene (just some primitives) and move them around?

 

[Frank]

I think I know too little about building 3D engines to be of much help in coding one, I'm afraid. So, I would really like it if others would take a look at Grahams work as well and tell us what they think and how we can make it into a great engine.

What I really like about it is the shader parser, and I think it would be a very good idea to focus the design goal around creating a high level scripting language that would be the input for the engine. Together with (compressed) xml for the objects and scenes, that would make it very powerful and flexible.

That would also require lots of import functions (scripts) to convert 3D objects to xml.