Unreal Engine 5, [UE5 Developer Availability 2022-04-05]

[milk]

But my point was that having "more memory" to allow more things to be loaded in at a time doesn't actually help the case where you are traversing linearly through a world. It only potentially helps the case where you are wandering back and forth, and even then the scaling with area/volume is not good. If your streaming throughput is enough to have loaded X region as you approached it in the first place, it's also enough to stream in and out parts of it as you move around it.

That is indeed true if for instances of constant speed traversal, which are not entirely rare, but are also not the only times some games go through traversal stutter. Many moments of gameplay have characters slow down here and there (stoplights, obstacles, curvy roads, funneling terrain, enemy interactions, stopping to grab loot, etc) providing ample time for the "look ahead buffer" to catch up and delay the point of streaming bottleneck. So for those instances (which are typical) a larger buffer can mean less instances of stutter. And if the lower priority work scheduling is implemented correctly, it should not add any aditional stutter, the worst case scenario would only mean the buffer staying unnused.

 

[tuna]

Does it really matter what I do when virtually no console games are shipping with Rust code or the fact that game development related studios only hire experienced C/C++ programmers ? Also it's well known in the industry alone that the only compilers available on consoles are either Clang or MSVC (only for Xbox platforms) ...

Yeah, let's stop making progress in dev tooling for consoles!

 

[Lurkmass]

Yeah, let's stop making progress in dev tooling for consoles!

Console developers haven't had the freedom to pick whatever compiler they wanted for a while now and the contribution community/licensing model of Clang is attractive to the corporate world for the purposes of both selling their proprietary NDA'd or highly controlled (Apple) platforms as well as enabling cross-platform software development for them ...

 

[tuna]

Console developers haven't had the freedom to pick whatever compiler they wanted for a while now and the contribution community/licensing model of Clang is attractive to the corporate world for the purposes of both selling their proprietary NDA'd or highly controlled (Apple) platforms as well as enabling cross-platform software development for them ...

Are you actually limited in the compiler choice you have with consoles? As long as you can interface with the OS and the system libraries you should be good to go, or am I wrong?

 

[Lurkmass]

Are you actually limited in the compiler choice you have with consoles? As long as you can interface with the OS and the system libraries you should be good to go, or am I wrong?

Frankly, why would console vendors allow custom/alternative compilers when it both costs them more developer support resources (maintaining support for frameworks outside of their official toolchain) AND opens up another attack vector (yes I'm going to use the security argument here) on their platforms ?

 

[Dominik D]

It appears to be a remaster of the Ultimate Edition which used a heavily-modified version of Unreal Engine 3

The reason I said a week ago that UE3 games are still being made.

 

[Dominik D]

Frankly, why would console vendors allow custom/alternative compilers when it both costs them more developer support resources (maintaining support for frameworks outside of their official toolchain) AND opens up another attack vector (yes I'm going to use the security argument here) on their platforms ?

How would you stop developers from using other compilers? You're linking binary blobs, it shouldn't matter where they came from.

 

[Lurkmass]

How would you stop developers from using other compilers? You're linking binary blobs, it shouldn't matter where they came from.

With console toolchains, I would expect them to be apart of the "signing process" for compiled code/binaries. Either that or the console vendors will sign the code/binaries after certification ...

 

[Shifty Geezer]

With console toolchains, I would expect them to be apart of the "signing process" for compiled code/binaries. Either that or the console vendors will sign the code/binaries after certification ...

Is that supposed to be 'a part' or 'apart', as both work but mean the opposite! In context, pretty sure the former.

 

[Dominik D]

With console toolchains, I would expect them to be apart of the "signing process" for compiled code/binaries. Either that or the console vendors will sign the code/binaries after certification ...

That's not how certification works. Game calls into some APIs and you are expected to do it in a way that 1) doesn't make your game crash, 2) doesn't spam certain APIs (e.g. you're expected not to call some Xbox Live status APIs too often as that could DDoS the service). That's the level of "binary" verification that certification goes through. You game needs correct manifest as well which could be considered binary, but that's it. Most of the certification deals with user-facing stuff and game behaving in a predictable and stable manner.

What you're advocating for (or against, depends on the POV) would eliminate any assembly optimizations. It would make it hard if not impossible to develop in anything other than C++, yet there are games in C# (il2cpp, I know), Haxe and I've heard (unconfirmed) rumors about a game on PS3 written in Pascal. In reality it's pretty simple to link libraries built with Rust (especially nostd ones) with Xbox or Switch executables (haven't tried this on Sony hardware but I expect it to just work). There's absolutely nothing stopping you. Modern consoles are running sophisticated operating systems that prevent games from doing most of the funky stuff you could do on a bare metal so your security argument wasn't sound.

 

[Lurkmass]

What you're advocating for (or against, depends on the POV) would eliminate any assembly optimizations. It would make it hard if not impossible to develop in anything other than C++, yet there are games in C# (il2cpp, I know), Haxe and I've heard (unconfirmed) rumors about a game on PS3 written in Pascal.

Are you aware of the existence of programming language extensions and intrinsics for the express intent of bypassing the compiler for more optimal codegen ? If developers really wanted to speed up the execution times in their programs, they're not stuck with just writing only standard C++ as you would implicate ...

In reality it's pretty simple to link libraries built with Rust (especially nostd ones) with Xbox or Switch executables (haven't tried this on Sony hardware but I expect it to just work). There's absolutely nothing stopping you. Modern consoles are running sophisticated operating systems that prevent games from doing most of the funky stuff you could do on a bare metal so your security argument wasn't sound.

Do you even know what an operating system is supposed to achieve from a security standpoint ? It's principal security function is to prevent user space processes from escalating to higher privilege levels (ring 3 -> ring 0) to prevent them from hijacking kernel level (same level as the OS) processes ...

It's mininformation to insinuate that it's the job of an operating system to cover up/work around the security problems posed by potentially buggy compilers or other user space exploits ...

 

[Dominik D]

If developers really wanted to speed up the execution times in their programs, they're not stuck with just writing only standard C++ as you would implicate

You can optimize for other things than speed. But the gist of what I'm trying to convey is that hand written assembly and binary output of "unapproved" compiler is the same from the point of view of, well, everything. Including security.

Do you even know what an operating system is supposed to achieve from a security standpoint

I do. Your statements sound like something I would confidently (and incorrectly) write online some twenty years ago. So I get it, you think you know your stuff - we've all been there.

Originally console games were running pretty much bare metal. Even early consoles with "rich" experiences - like Xbox 360 - were essentially replacing OS with a game on launch. Game was linked to a stub of system services that were universal for all applications (things you could access from guide button).

Since 360 had access to Live, there was much more emphasis on binary analysis. It was a big no-no to have any sort of self-modifying code (you couldn't make writable memory page executable) so JITing for instance was not possible. Plus: it was much harder to publish on a console in 2005 than it is today.

This security by obscurity was pretty much phased out and consoles today are running full blown OSes with a hardware-anchored chain of trust that's used to validate binaries. You've got multitasking with all its benefits and drawbacks and all major consoles are running a cut down version of a generally available OS (Windows or FreeBSD).

Privilege escalation is not the only security feature of operating systems running on consoles and it's not even the most important one. Consoles are locked down and the most important thing OSes are doing is aforementioned binary validation. But we can agree to disagree on what's most important, it's not impacting reality what you or I think is most important.

It's mininformation to insinuate that it's the job of an operating system to cover up/work around the security problems posed by potentially buggy compilers or other user space exploits ...

All compilers are buggy, what does it have to do with anything discussed here? No sane developer with access to devkit is going to deliberately build a game that makes it easy to exploit console. Historically most of the vulnerabilities found in games that were used to run homebrew came from custom data parsing code (savegames typically). This buggy code would be buggy regardless of C/C++ compiler used. Preventing someone from using clang instead of msvc changes nothing. On the other hand letting developers write data parsers in safer languages would make life much easier for the platform holder.

 

[Charlietus]

For as much criticism that unreal gets (and it's valid criticism), it enables something like Clair Obscure to exist. I don't think that a game like that could exist on such a budget without Unreal.

 

[arandomguy]

I've been on the side of (if you can term it like that) engines like UE (and also Gamebyro and it's later Creation iteration and etc.) over others due to those reasons since way back. I know end users don't tend to look at this way and I can understand why, but I prefer to judge the tooling by what it enables the creators to do.

I do understand why end users though tend to look at it in terms of the game and development would essentially be the same but with the visual output/performance (well perceived ideal anyways) if it were using another engine even if that's way overly simplistic and not going to happen.

 

[Dominik D]

I prefer to judge the tooling by what it enables the creators to do

People really underestimate how powerful blueprints are. Yes, you can create massive headaches with them but I've seen enough code to know you can shoot yourself in the foot with code as well. More often than not you can build most of the game using visual scripting and deliver quality experiences faster and cheaper with it. There's really no productivity tool like it (other than Odin for Unity, maybe).

 

[Lurkmass]

You can optimize for other things than speed. But the gist of what I'm trying to convey is that hand written assembly and binary output of "unapproved" compiler is the same from the point of view of, well, everything. Including security.

I do. Your statements sound like something I would confidently (and incorrectly) write online some twenty years ago. So I get it, you think you know your stuff - we've all been there.

If you say so ...

Originally console games were running pretty much bare metal. Even early consoles with "rich" experiences - like Xbox 360 - were essentially replacing OS with a game on launch. Game was linked to a stub of system services that were universal for all applications (things you could access from guide button).

That ultimately doesn't change the fact that console vendors unanimously did away with the idea having games being able to run in kernel space for a while now ...

This security by obscurity was pretty much phased out and consoles today are running full blown OSes with a hardware-anchored chain of trust that's used to validate binaries. You've got multitasking with all its benefits and drawbacks and all major consoles are running a cut down version of a generally available OS (Windows or FreeBSD).

Privilege escalation is not the only security feature of operating systems running on consoles and it's not even the most important one. Consoles are locked down and the most important thing OSes are doing is aforementioned binary validation. But we can agree to disagree on what's most important, it's not impacting reality what you or I think is most important.

If operating systems by themselves can't provide a complete end-to-end security solution and you yourself allude that "hardware measures" (beyond/deeper level than the OS) are necessary then why do you keep selling the idea that they're the an omnipotent factor in the security architecture of a console ?

BTW, "binary validation" is mostly integrated at the hardware level within the SoC since data/code stored outside of the processor can't be trusted ...

All compilers are buggy, what does it have to do with anything discussed here? No sane developer with access to devkit is going to deliberately build a game that makes it easy to exploit console. Historically most of the vulnerabilities found in games that were used to run homebrew came from custom data parsing code (savegames typically). This buggy code would be buggy regardless of C/C++ compiler used. Preventing someone from using clang instead of msvc changes nothing. On the other hand letting developers write data parsers in safer languages would make life much easier for the platform holder.

If you look into the very link I posted at the start, there's an example of Webkit exploit on PS5 that was left behind when it was compiled with Clang and the source of the vulnerability is presumably due to the lack of coverage in one of Clang's anaylsis pass. A compiler project as big as Clang(!) with multiple corporate sponsors is enough proof that compilers can contain security vulnerabilities that can be used by attackers ...

The memory safety model of Rust does not implicate a 'secure' design and it certainly doesn't prevent sources of attack vectors such as compilers as I mentioned ...

 

[Dominik D]

If you say so ...

Tell me, was it compiled with vendor approved compiler or not?

why do you keep selling the idea that they're the an omnipotent factor in the security architecture of a console

Nothing is omnipotent, you're making stuff up. They are the primary reason consoles don't fall apart the day they release. Your insistence that compilers are somehow providing safety guarantees on the other hand is pretty bizarre. Recall that this whole discussion is about whether developers can or should be prevented from using 3rd party compilers. So don't move the goalpost, please.

BTW, "binary validation" is mostly integrated at the hardware level within the SoC since data/code stored outside of the processor can't be trusted

That's not how any of this works. SoC doesn't do any binary validation (with or without ""). All data and code run on the CPU or GPU comes from somewhere. It's lifted from some storage media and uploaded into RAM. What is or isn't code or data is driven by the layout of executable and managed by the OS. Whether you can map the same page as writable and executable is up to the OS and most force W^X (writable XOR executable) for majority of pages and applications. There may be exceptions for certain processes, and browsers are typically exempt to facilitate on the fly JS compilation. That's why browsers are inherently great vectors of attack. But before browsers were ubiquitous, savegames were much easier target for exploits.

If you look into the very link I posted at the start, there's an example of Webkit exploit on PS5 that was left behind when it was compiled with Clang and the source of the vulnerability is presumably due to the lack of coverage in one of Clang's anaylsis pass. A compiler project as big as Clang(!) with multiple corporate sponsors is enough proof that compilers can contain security vulnerabilities that can be used by attackers

Exploit that you linked has nothing to do with clang. Two CVEs referenced are use after free and buffer overflow and are bugs introduced by webkit authors. If anything, clang has more tools protecting from both vulnerabilities than MSVC does (with AdSan and UbSan). Which brigs us back to the original topic: why shouldn't developers use clang or anything LLVM-based on Xbox if they want to? Why should platform holder stop them from doing it? Rust, for instance, protects from use after free and buffer overflows.

Just FYI: I won't be responding to your posts from now on. It's painfully obvious you don't have first-hand experience with any of the topics discussed and you don't want to learn. So it's a waste of my time. Cheers!