HTML5 on consoles

[JPT]

How is this issue resolved?

What issue?

If my speculation using the Motorola cell phone showing a Firefox web browser, that the front end only was ported to Android, is incorrect then there would have to be two versions of the backend (Webkit and Firefox) as webkit is used to provide tools for other Android programs on Android platforms. Since Android generally has many apps pre-loaded in memory to eliminate slow startup times, this might impact free memory in such a system.

Download the source code for Firefox for Android and see if there is any webkit in there.

https://wiki.mozilla.org/Mobile/Fennec/Android

 

[makattack]

What issue?

Download the source code for Firefox for Android and see if there is any webkit in there.

https://wiki.mozilla.org/Mobile/Fennec/Android

Or, for those who aren't able to read source code, read a developer blog justifying porting gecko to android:

http://limpet.net/mbrubeck/2010/10/04/why-fennec-is-different.html

 

[jeff_rigby]

Or, for those who aren't able to read source code, read a developer blog justifying porting gecko to android:

http://limpet.net/mbrubeck/2010/10/04/why-fennec-is-different.html

Thanks for the link, it's just what I needed.

There are several other browsers for Android, but all of them use the built-in WebKit rendering engine (except Opera Mini, which uses a proxy server for rendering). The same is true for iOS, which uses WebKit too – as do the latest versions of BlackBerry, Symbian, and Palm webOS. [Update, November 2010: Opera Mobile for Android is now in beta and uses its own rendering engine too.]

You can understand how I could make an assumption.

OK, Opera and Firefox use their own engine. Does that mean webkit is still on the platform and may be needed by other applications?

 

[Shifty Geezer]

OK, Opera and Firefox use their own engine. Does that mean webkit is still on the platform and may be needed by other applications?

If webkit isn't supported in the platform OS, you can just include it in your application. Just as EA developed a PS3 Webkit for their purposes. The wonderful thing about software is you can write it to do whatever you want.

 

[jeff_rigby]

Needs to be deleted

 

[JPT]

There is webkit code in Firefox just snippets though.

But that is not what you claimed before, you claimed that Firefox for Android is just a front end for webkit.

 

[jeff_rigby]

But that is not what you claimed before, you claimed that Firefox for Android is just a front end for webkit.

Yes I did and I was wrong, I made an assumption, a speculation based on multiple previous facts that did not apply to the New Broadcom SOC, the 2 month old Opera or new version of Firefox. I was correct about all other Web browser versions on Android using webkit as the back end.

It's still not clear if parts of webkit still have to reside in memory even if Firefox is being used as the web browser. Not for Firefox but for Android to use. Does Firefox have it's own SQLite or does it use Android's to support HTML5 database features?

There is such an effort to free as much memory as possible on Android that I can't imagine a process where duplication is allowed to exist. The same applies to the PS3 with it's limited memory.

 

[JPT]

I'm sorry I just get so annoyed by the sloppy work that is being put into all of this research/science, I'll stay out of from now with the help of my ignore list, once again sorry for the noise.

 

[jeff_rigby]

This needs to be deleted

 

[jonabbey]

Is this thread topic still useful for any purpose?

 

[AzBat]

Is this thread topic still useful for any purpose?

Whew, I thought I was the only one. Not really the purpose I originally intended. I'm sure Jeff will spend 10,000 words explaining that it does though. I highly recommend Jeff take this talk outside the Console Forum. If future talk isn't about PS3, Xbox or Nintendo using HTML5, then I say lock it.

Tommy McClain

 

[patsu]

Jeff Rigsby confirmed that JavaScriptCore runs on POSIX on the PS3. He also speculated correctly that the video apps run on HTML5, at least partially. Makattack pointed us to EA version of WebKit on PS3. Sony licenses Opera for Blu-ray players and TVs. 3DS can switch to a web browser in-game. That's all we know so far.

If we don't spam the thread with out-of-thin-air speculations, we may get to see HTML5 stuff from MS.

 

[jeff_rigby]

Whew, I thought I was the only one. Not really the purpose I originally intended. I'm sure Jeff will spend 10,000 words explaining that it does though. I highly recommend Jeff take this talk outside the Console Forum. If future talk isn't about PS3, Xbox or Nintendo using HTML5, then I say lock it.

Tommy McClain

I kinda missed you in defending HTML5 in consoles...you started the thread and did not choose to defend, provide information or help prove. What was your original intention?

 

[jeff_rigby]

Jeff Rigsby confirmed that JavaScriptCore runs on POSIX on the PS3. He also speculated correctly that the video apps run on HTML5, at least partially. Makattack pointed us to EA version of WebKit on PS3. Sony licenses Opera for Blu-ray players and TVs. 3DS can switch to a web browser in-game. That's all we know so far.

If we don't spam the thread with out-of-thin-air speculations, we may get to see HTML5 stuff from MS.

Yeah the PS Suite thread is using information developed/discovered in this thread. One of the items mentioned in PS Suite is converting from "C" to HTML5 as well as tools to convert Wii "C" to WP7 for the Xbox 360 - WP7 ecosystem using MS Silverlight 5 (HTML5) which is similar to the Sony PS Suite.

The PS3 is in the process of getting a HTML5 browser and now has a HTML5 Webkit javascript engine, the Xbox 360 has at least a javascript engine and will be getting Silverlight 5. It's still not known how the PS3 supports Adobe Media server 3.5 streams.

Multiple people discovered and posted information useful to this thread.

PS Suite also gives us a better understanding of where Sony is heading and firms up speculation so it's not so wild.....Cross platform, Open Source and Industry STANDARDS. HTML5, POSIX, OpenGL and I think Android open source libraries for the "windows" OS support to be needed by Webkit on the PS3.

http://forum.beyond3d.com/showthread.php?t=59421&page=4 Message 89 on

Thanks (Shifty) for confirming the logic on this one point OpenGL on the PS3.

With Cross platform, OpenGL is, I think, a given. With Open GL support on the PS3 it would be easily possible to support remote play, a touted feature of the PSP but never fully developed. This is so obvious that the author of the quote thought it implied in Sony press releases.

Beyond remote play, the same application could support a remote XMB screen and access to all PS3 features like a remote desktop application does for PCs,

IF the XMB is rewritten with OpenGL calls, a rewrite which I think is coming for the Webkit port and possibly to allow PSS access/ports and HTML5 widgets . Many of the Open source libraries for the NGP will probably be used and they probably take advantage of OpenGL. It's a feature I think would help sell the NGP.

The Logic of this starts with the Email from the Sony employee in charge of the Webkit port, "It's a Cairo POSIX port of Webkit and more is coming". Cairo can use OpenGL as the back end for the Graphics. Webkit requires windows support. PS Suite uses OpenGL to easily allow cross platform porting and windows as well as surface management are probably a requirement for PS Suite porting. What is being taken as a leak by Sony is that PS Suite will be used to port games and applications to the PS3. That also supports OpenGL and windows management routines on the PS3 from two directions, PS Suite and the Webkit port. There is also that to display 3-D from the browser requires both HTML5 and WebGL (OpenGL)

What we can take from this: Webkit is definitely coming....Sony won't stop as MS did with just the javascript engine. A full implementation which requires changes to the XMB for HTML5 Widgets is also more likely. Open Source OS support libraries for this from the NGP will probably be used for the PS3 making PS Suite porting to the PS3 easier. The XMB will be rewritten to support OpenGL and HTML5 UI.

The PS3, according to posts in this forum, had a poor OpenGL implementation (shaders were slow and buggy) and developer engines used Sony provided lower level calls which are not designed to support cross platform. With cross platform PS Suite and the coming webkit port, OpenGL must have been rewritten and might now be usable for AAA games on the PS3.

 

[deathindustrial]

Not sure how this affects things but the Netflix client on the PS3 has a sort of debug menu that can be brought up via the D-Pad sequence:

Up, Up, Down, Down, Left, Right, Left, Right, Up, Up, Up, Up

For the GUI that I have it mentions using QT Version 4.6.2 which makes it sound more like a native client than a generic HTML based one. The UI version is "release-105-high" in case anyone is keeping track.

Cheers

 

[jeff_rigby]

Not sure how this affects things but the Netflix client on the PS3 has a sort of debug menu that can be brought up via the D-Pad sequence:

Up, Up, Down, Down, Left, Right, Left, Right, Up, Up, Up, Up

For the GUI that I have it mentions using QT Version 4.6.2 which makes it sound more like a native client than a generic HTML based one. The UI version is "release-105-high" in case anyone is keeping track.

Cheers

Recent interviews have more information about Netflix. It's in an article from a QT employee.

http://www.fosdem.org/2011/interview...e-christiansen

"Qt WebKit is a very powerful engine, and it is quite easy to embed web usage in your apps. It even spots a hybrid approach where you can extend the engine from C++, so that you can access your C++ methods from the web view runtime. Quite powerful and used a.o. by Netflix on the Roku and the PS3 with a custom port."

QT webkit is a cross platform Hybrid "C" and webkit development platform. The includes in the Netflix 21 meg application are a webkit toolkit (not the whole webkit) and "C" library extensions to the basic POSIX PS3 OS.

QTWebkit can also use Gstreamer plugins which may have provided a trick/adaptive streaming H.264 player (Cell optimized codec plugin for Gstreamer probably provided by Sony).

Sony uses a Gstreamer core (200k) but the Open source player (Playbin2 higher level built on Gstreamer) does not include hooks for DRM.

How Sony would provide DRM was the subject of much speculation. As a closed console they may have decided they did not need to support DRM. All my speculation about Flash being needed or Air 2.5 would have been in error. It is possible that Sony may not include any Flash support beyond a player. IF Google releases a DRM to include with HTML5 video then even a Flash player is not needed as the web might switch nearly overnight to HTML5 video. (Applications outside the browser could/would still support H.264)

In the QT Webkit site a FULL QT webkit library for an Embedded Linux platform is said to be 21 megs. This is exactly the size of the Netflix downloadable application before the first connect. After that first connection it grows to about 28 megs. I'd guess that the first 21 megs is QT Webkit and the additional 7 megs or so difference is the actual Netflix QT application that runs using the 21 meg QT library.

Netflix did a custom port of QT webkit to the PS3. This is what they are claiming as a full webkit port to the PS3. The timing of PS3 Firmware 3.5 including a partial webkit and Netfix using HTML5 turned out to be coincidence.

Hulu, MLB and other applications which are significantly smaller (7-12 megs) most likely do use the HTML5 javascript engine and other resources provided by/in the PS3 included in PS3 firmware 3.5.

 

[jeff_rigby]

More info here: http://community.us.playstation.com/thread/3415233?tstart=0

NEW INFORMATION:

In the above site is a summary of the proof for a webkit coming to the PS3 plus some new information I haven't seen available on B3D. The key is; Gstreamer, it's being used for the multi-media in the PS3 and most likely the NGP. It has features that lends it'self to supporting media and a Sony ecosystem. It is being used by Opera and Firefox to support HTML5 <video> and Opera has implemented SVG support for Gstreamer.

cairographics with Open GL or PSGL support will be used in the PS3 and NGP for the XMB and Webkit. Between the two, Gstreamer and Cairo, remote desktop and remote play should be easy to implement.

Gstreamer and Cairo support is a possible reason to not use a Chrome flavor webkit in the PS3 (Chrome does not support Gstreamer or Cairo). This might also impact the browser choice in Sony Android platforms. Possibly Opera will be used which supports Gstreamer and Gstreamer integrated with SVG graphics. Or Sony might include Gstreamer in the Android platform and include their own webkit.

The player in the PS3 is most likely written by Sony using Gstreamer and was updated sometime around firmware 3.5 to support adaptive and trick playing. Sony would have written the H.264 codec optimized to use the Cell SPU as a plugin for Gstreamer. Any new applications or updates using Gstreamer would automatically use the proper plugin. For instance "Home" and any IPTV applications also using Gstreamer calls can access the new upper level Gstreamer player application. Netfront may have and probably still is benefiting from Gstreamer updates.

Collabora released an open source trick/adaptive streaming player based on Gstreamer at about the same time the PS3 player was updated. Collabora also has an open source video editor based on Gstreamer. If you look at the first graph in the Gstreamer link it appear as if PS3 release and PS3 adaptive streaming release dates have peak commits/work on Gstreamer.

For Cairo and the PS3/NGP: http://snap.sonydeveloper.com/about/

SNAP has a re-architected display model and backend based on Cairo evolving toward COLLADA over time. Our look and feel also includes navigation metaphors derived from standard Sony UX constructs such as our XrossMediaBar.

Sony is expecting to have Cairo evolve to use game assets (COLLADA). That to me means that Cairo will be used as the basic building block for (applications, desktop and possibly on-line games) the NGP and PS3. The front end (desktop) will be based on Cairo as will the webkit ports. An upper level Cairo library would be more efficient for remote desktop. PSGL would be used for games (NGP and PS3) and could allow select games to be played on the NGP and displayed on the PS3, it would help if assets, (collada library) would exist on each platform.

Edit: there is a HTML5 canvas vs. SVG for web pages and Raster vs vector graphics for game assets discussion in several articles. Google search now indexes crawls svg files . Google now has a category to search for SVG web pages. Will there be SVG only browsers or is SVG such a desirable feature for some platforms that users may decide to use only SVG? The prevailing wisdom is that both pixel graphics and vector graphics have their place in the web and in games.

Doing a Google search for (webkit, Cairo, Gstreamer) brings up a GTK+ webkit version by Collabora. Collabora may have also provided the PS3 Gstreamer based player as well as much of the code for the video editor.

The GTK port of the WebKit HTML rendering engine has gained support for the HTML5 video element. The media backend, which uses GStreamer, was implemented by Pierre-Luc Beaudoin of Collabora. Developer Alp Toker integrated the backend with GTk/WebKit's Cairo graphics pipeline, making it possible for the video content to be embedded in SVG and manipulated with CSS and JavaScript.

Plans for GStreamer-based HTML5 video support in GTK/WebKit were initially revealed by Toker in September, when he published slides from his presentation at LinuxConf Europe.

As some of you may recall, support for the HTML5 video element was implemented experimentally for Firefox back in August.

The GTK/WebKit port appears to be maturing rapidly and offers some unique advantages over Firefox's Gecko rendering engine in certain contexts. GTK/WebKit is lightweight and less resource intensive than Gecko, which makes it a particularly good choice for mobile and embedded environments. GTK/WebKit will also eventually be a very good solution for GTK and GNOME applications that want lightweight embedded HTML rendering

Opera & Gstreamer HTML5 video. http://www.osnews.com/story/22677/_Re-_Introducing_HTML5_Video

"You have to understand that this goes far beyond just replacing Flash video players. That is the first step, but with video being a first class citizen in the DOM and not hidden in a sandbox, developers can style and play with the video data however they can imagine. You can spin, skew, colourise and even map it onto a 3D cube. Anything else on the page can change or interact with changes in the video. We’ll be able to invent new ways of annotating and commenting on videos, all without the use of Flash, and therefore inherently compatible with any OS and any device, including mobile phones."

Firefox implemented the first use of Gstreamer for HTML5 video (2008). Collabora (open source) and Opera now use Gstreamer integrated with the Cairo or SVG graphics pipeline for the HTML5 <video> tag. Sony and others will implement this. This may be one of the reason for Sony's choice to use Opera on their 2011 TV's and blu-ray players, Opera uses Gstreamer for HTML5 video support and integrates the video with 2D SVG.

The quotes above; " integrated the backend with GTk/WebKit's Cairo graphics pipeline, making it possible for the video content to be embedded in SVG and manipulated with CSS and JavaScript." and " You can spin, skew, colourise and even map it onto a 3D cube." describe something like the above video clip. It's just a series of pictures being manipulated, with Gstreamer, video streams can be manipulated the same way. It will be possible inside a webkit browser and on the PS3 XMB. Edit: The two finger iOS ability to pinch and expand pictures really impressed me and I think Sony. With Cairo and Gstreamer this and more is possible. I expect this to be a part of updates and new platforms.

One of the ports below, in my opinion, is coming to the PS3.

So possibly a GTK/Cairo/Gstreamer/Soup/Pongo port. (guess)
There is also now a GTK/Cairo/Gstreamer/Clutter webkit port by Collabora. Which might lend it'self to a game platform more than any other Webkit port. http://blog.kov.eti.br/?p=139 (may use Pongo also but no information on that yet)

Both above would use the generic webkit javascript engine now in the PS3.

This still requires some form of X windows support or Direct FB (Direct Frame Buffer); so libraries for that? Big job!

New information Cairo used exclusively

GDK no longer wraps the antiquated X11 drawing API; we’ve made a clean break and exclusively rely on cairo for all our drawing needs now. This has also enabled us to remove several other X11-centric concepts such as GCs, colormaps and pixmaps. The hero who did most of this work is Benjamin Otte (Red Hat).

Advantages: more portable, cleaner code, faster - can optimise around Cairo.

There is a Cairo OpenGL version so with custom PSGL support for Cairo on the PS3 and NGP a smaller fully supported webkit is possible and at the same time an OpenGL (PSGL) Cairo library would then exist on the PS3 and could be used for multiple applications. If the Cairo library is active and being used for graphics in the XMB screen, it should be easy to open application windows including Webkit on the XMB.

Back to what is the NGP using?
Recently confirmed at GDC 2011 was that NGP and PS3 were going to use the same or similar PSGL programmable GPU and shader C libraries, the same toolkit chain and the same or similar Open Source OS libraries. This makes it easier for "C" portability between platforms. Since NGP appears to now have the libraries required for a webkit port, those libraries in NGP will most likely be ported to the PS3.


http://en.wikipedia.org/wiki/Clutter_(toolkit)

Clutter allows implicit animations of every item on the canvas using special objects called behaviours: each behaviour can be applied to multiple actors, and multiple behaviours can be composed on the same actor. Behaviours handle animations implicitly: the developer specifies the initial and final states, the time (or number of frames) needed to complete the animation, the function of time to be used (linear, sine wave, exponential, etc.), and the behaviour will take care of the tweening. Clutter provides a generic base class for developers to implement custom behaviours, and various simple classes handling simple properties, like opacity, position on the Z axis (depth), position along a path, rotation, etc.

Gstreamer integrated with Cairo's SVG both in and outside the browser on the PS3 and NGP! This could make for some interesting and unique features on the PS3 and NGP desktop using the strengths of both. We might see front ends blowing away the very attractive iOS and Android front ends.

Collabora is a consulting (by contract) firm that helps companies integrate open source libraries into their hardware and software products.

http://www.collabora.co.uk/ Look at the open source projects at the bottom of the page, coming to the Sony ecosystem?

 

[jeff_rigby]

SVG graphics (vector) have advantages in a smaller browser and webpage memory footprint. Vector graphics and text are also scalable to fit multiple resolutions. The handheld and embedded industry is apparently moving to SVG. (Some Articles deny a savings for complex screens, this might be pre-cairo and pre rewrite of javascript. Not sure here, see quote below.)

Android currently has a limited SVG ability that is apparently not hardware accelerated and third parties are attempting to integrate Cairo into Android for full SVG support as a temporary measure. And Gstreamer into Android also. I would guess that more powerful Playstation certified hardware with Android 2.3+ being used for Games and multi-media can support more powerful vector graphics and muti-media libraries. Android officially evolving toward the Sony PS3 library choices: Gstreamer and Cairo?

Firefox uses both Cairo and Gstreamer and Opera uses Gstreamer but I can't find out what Opera uses for SVG support.

https://github.com/anoek/android-cairo
http://openvirtualworld.blogspot.com/

I discovered another argument for cairo compared to WebKit. In a blog post (which I do not find anymore) a Google Chrome developer discusses the choice of skia. Skia is not hardware accelerated, which makes skia look like a bad choice. But, the Chrome developer argues, that most of the time the browser spends in HTML code, arranging DOM elements and applying hierarchies of CSS styles. Once the rendering code knows where to paint something, then only a small fraction of the time is spent for the actual drawing. This is a hint, that using WebKit is much worse performance-wise than using plain cairo. When we use cairo directly, then we get rid of all the time that WebKit spends dealing with HTML features. The trade off is, that there is no CSS, no JavaScript, and no iframe in the avatar display. CSS makes applying different styles easy. Styles and themes need much more effort in plain cairo drawing code. But it is should be much faster. And speed counts, because we do not want to slow down your PC.

And the above perhaps explains why Cairo and Clutter support might be attractive to Sony. Why use HTML5 for a UI, custom routines using Cairo.

So Cairo will be used to support a custom PS3 desktop with windows like support. I guess this fills out the requirements for a webkit port. Desktop animations are also possible with Cairo, Cairo may already be used for the PS3 XMB.

How The Google Android OS does without Xwindows PS3 - NGP - & iOS the same?

Now, why then does Google not use an X windowing system in Chrome OS? The technical explanation for this is “they simply don’t need it for what they’re doing.”

Another acquisition Google made in 2005 was a company called Skia Inc. that marketed a 2D graphics library also called Skia. The Skia library, written in C, does vector graphics, text, and images. It was open-sourced in 2008. Same features as Cairo

In writing Android, Google used Skia as the graphics backend for its windowing system and GUI toolkit. They also wrote a Skia backend for WebKit, which is used in the Android web browser. This same Skia Webkit backend is also used by Chrome on all of the platforms it supports. This is what I think Sony will do with Cairo

What Google has done in Android, and will do in Chrome OS, is write a simplistic windowing system using Skia, and let that output directly to the device framebuffer, while I’m no expert on Skia, I assume it also has a backend implementing its painting operations via OpenGL, which would then make it easily hardware-acceleratable on a variety of different hardware offerings.

So to sum it up:

Google already has Skia-on-kernel-without-X working in Android, and they already have Chrome/WebKit using Skia. Chrome OS is unlikely to have the sort of drag-windows-around-on-a-desktop user experience that OS X or KDE offer (I expect it will be something more iPhone/Android-like), so while they could certainly use X, the small subset of its functionality they actually require is easily written anew using Skia, and doing so means they don’t have to import a significant outside codebase and get involved with its development.

I believe the above also applies to the PS3 and NGP but using a much more powerful Cairo.

Again, much of this will be shared between PS3 and NGP.

 

[jeff_rigby]

How Sony would provide DRM was the subject of much speculation. As a closed console they may have decided they did not need to support DRM. All my speculation about Flash being needed or Air 2.5 would have been in error. It is possible that Sony may not include any Flash support beyond a player.

Best source for DRM information

IF Google releases a DRM (they purchased Widevine) to include with HTML5 video then even a Flash player is not needed as the web might switch nearly overnight to HTML5 video. (Applications outside the browser could/would still support H.264) Gstreamer supports HTML5 WebM or any codec used for HTML5 <video> provided the plugin is on the platform, it's automatic.

Web browsers would have to be modified to include this Google open source DRM support if coming. Possibly the Gstreamer plugins will be modified. This would quickly provide support for DRM in Opera and all browsers using the Gstreamer support on the platform (PS3 and NGP). DRM support would then be in the PS3 and some form of home brew might be allowed. Gstreamer being ported to Android might then be doubly important.

Not including the H.264 codec in Opera would not affect Opera using H.264 if it's included on the platform as Opera will use the Gstreamer library in the platform if it exists (Opera is then not required to pay the $6.5 million or less (cap) per year). That this is a feature built into Opera tends to make me believe many platforms will be supporting Gstreamer.

No Flash support of any type in the PS3 is also a possible.

In the discussions where I incorrectly assumed (maybe) that Sony or others were providing DRM inside the PS3 to protect IP (movies and music), pointing out the PS3 was a closed platform and does not need DRM would have immediately changed the nature of the discussion.

If this is true (No DRM protection in the PS3) then there will never be any homebrew software allowed on the PS3. That has major implications.

My not seeing this might be due to a feeling that a platform must have some form of customization and easy configuration for the various different hardware configurations in Home theater setups. Same for plug and play devices. This is something that will still have to be addressed by Sony.

Open source DRM provided by Google could be an answer to all my speculation. Is it needed, will it increase web usage and fix a problem we currently have on the internet? Why did Google buy Widevine?

Sony apparently has advance knowledge about what's coming and when, be it from the open source Linux community or Google. In some cases I believe they fund open source software to support future plans. I believe they have a timetable they are following and our feeling that they are slow to release new features or a stumbling behemoth is probably in error.

If Collabora is being contracted by Sony, my impression is they generally publish new open source in Sept -Oct (PS3 firmware 3.5 was released September 29th) at the EU linux conference but the company that contracts with them to develop new Open Source applications gets advance access. This might, if true, only support the timetable we are used to; E3 announce and E3 or October release for new features. (Chicken and egg, the Christmas selling season is probably driving the timetable and thus software release dates as is the choice of the E3 and Tokyo Game Show dates.)

All the pieces appear to be in place except DRM. It's possible that a WOW front end XMB redesign for the PS3 might hold up the PS3 Webkit release. E3 announce and October release with the Sony S1?

 

[jeff_rigby]

DRM and Gstreamer (2005)

Remember this is a 2005 article:

GStreamer is an extensive support library for the creation of multimedia applications. Audio and video applications can be constructed as a series of pipelines;

According to GStreamer hacker Christian Schaller, future releases of GStreamer may contain a feature which is less welcome to many: digital restrictions management (DRM) support. There are, says Mr. Schaller, clear reasons why one might want to support DRM-enabled GStreamer modules:


Any DRM module which is to gain the trust of the entertainment industry (much less avoid DMCA suits) will have to prevent the user from capturing an unencrypted stream. To that end, GStreamer will have to be able to create "secure pipelines"; DRM modules will then refuse to connect to modules which cannot be "trusted" with protected content. If GStreamer is to retain its current power and flexibility, many of its standard modules - and certainly those concerned with the actual playing and display of media - will have to be reworked to participate in secure pipelines. Either that, or significant parts of the GStreamer will have to be duplicated in a "secure" mode. It is hard to see how the entire GStreamer pipeline could be made to be secure without affecting people who have no interest in DRM-enabled content.

There is also the obvious question of how DRM can be done securely in an environment where source is available. Mr. Schaller points at Sun's "Opera" project as a possible example of how things could be done, and notes:

There might be some ramifications of being free software which will make the resulting system have conditions for use that makes it painful, like a requirement for being online when playing back as an example, but its definitely not impossible. (Ultraviolet)

Still, anybody who can hack on the source can obtain an unencrypted stream from a GStreamer DRM module. So it seems clear that such modules are expected to be shipped in a binary-only mode. Even then, though, one should remember that the Linux kernel is free software too. So even if the GStreamer pipeline is entirely secure and uncrackable, a quick kernel hack will still make the capturing of unrestricted streams easy. Another reason for removing Linux from the PS3.

That suggests, in turn, that the people looking to put DRM code into GStreamer envision operating in environments where users cannot install their own kernels. The TPM chips being put into an increasing number of computers may make that kind of restriction possible, but the real target is probably elsewhere: embedded systems.

This in the last year:

This topic was just recently discussed again. The answer is that there are
various ways to use gstreamer also for DRM protected content, although the
gstreamer project is not offering a ready made solution.

DRM support has a lot of platform security aspects that impact the design of you media solution.

The whole path has to be secure from source to sink (display, HDCP is also needed and has to report to the DRM that it's enabled). All GStreamer modules have to be rewritten to be DRM aware. This is a big job!!! If gstreamer is secure and supports a DRM scheme like ultraviolet then any application on the PS3 can automatically take advantage of DRM.


But we have this: http://mail.gnome.org/archives/tracker-list/2011-February/msg00051.html

Supporting DRM formats usually involves proprietary multimedia framework extension (such as the DRM-aware GStreamer plug-in) and low-level proprietary HW/SW support. It seems better to enhance GStreamer extractor than writing a new DCF extractor, because:
- If it depends on GStreamer, it cannot extract meta data faster than the GStreamer extractor.
- If it does not depend on GStreamer, it has to use proprietary code to do decryption and cannot be open source.
- OMA DRM is an open DRM scheme for mobile market and the underlying multimedia frameworks like GStreamer can support it by installing extensions. GStreamer extractor can try using GStreamer framework to get metadata at first; if nothing can be got because no GStreamer DRM extension is installed, the extractor can do something by itself.

There are some indications that DRM in some form is in the PS3 but NO information on an ultraviolet DRM with Gstreamer. Still big ?

Possible for Google to release a DRM scheme and Collabora to implement a Gstreamer version. Open source then available to platform developers to implement. Standards for DRM and video then available to the entire internet. All gstreamer plugins would have to be replaced with DRM aware versions, large job for Collabora and hardware platforms especially for the PS3 with the Cell SPUs being used to hardware accelerate Gstreamer based video codec plugins.